fovea is retired — the review service, its LiteLLM key and lanes, its Keycloak client and its webhook host leave the cluster
Date: 2026-09-28 Status: Accepted — supersedes hl-0ah0, hl-93me, hl-jecc and hl-buts Deciders: Sean Brandt
Context
Section titled “Context”fovea is a GitHub App that reviews pull requests. It ran in the fovea namespace with a dedicated Qdrant, and it called the LiteLLM Service through its own virtual key and three lanes (fovea-scout, fovea-deepdive and fovea-embed). Work on fovea is paused.
The fovea LiteLLM key reached its max_budget of $100 on 2026-09-25 at 04:52 UTC. Since then LiteLLM refused every fovea call, so fovea did no reviews. fovea kept retrying, and LiteLLM logged thousands of refusals each hour. Issue #2468 asked for the removal. It supersedes #2271 (the fovea budget decision) and #2208 (the fovea engram learning loop).
Decision
Section titled “Decision”Remove fovea from the cluster:
- The
foveaArgoCD Application,argocd/app-configs/fovea/andargocd/app-configs/fovea-chart/. ArgoCD prunes the namespace, the Deployments, the ExternalSecrets and the three PVCs (fovea-audit-js,fovea-indexandqdrant-fovea-data). - The
foveaentry intf/litellm/main.tf. Terraform deletes the virtual key. - The three fovea lanes in
argocd/app-configs/litellm-chart/values.yaml. Thezdr-probe-embedcontrol lane now mirrorsoctopus-embed(openrouter/baai/bge-m3), because it must share a model with a live lane. - The
fovearules in thelitellm-proxy-ingressand clickstack CiliumNetworkPolicies. - The
foveawebhook host intf/cloudflare/variables.tf. Terraform deletes thefovea-wh.fzymgc.netrecord and the tunnel ingress rule. - The
fovea-engramKeycloak client. The client-credentials loop into engram was never turned on (#2208), so nothing consumes it. - The
fovearow inscripts/seed-litellm-vault.sh.
Terraform forgets the Vault record secret/fzymgc-house/cluster/fovea/oidc instead of deleting it, because the terraform-keycloak-admin policy grants no delete on that path. The Vault policy grant for the path stays until the keycloak apply is complete, and a later pull request removes it. The removed block in tf/keycloak/fovea_engram.tf goes with it.
Consequences
Section titled “Consequences”- No pull request review from fovea on any repository. octopus stays.
- Two Deployments, one namespace, three PVCs, one LiteLLM key, three lanes, one Keycloak client and one public hostname fewer to run.
- Three Vault paths keep their data:
cluster/fovea(the GitHub App private key, webhook secret and the Anthropic and Qdrant keys),cluster/fovea/oidcandcluster/litellm-keys/fovea. The operator deletes them by hand when fovea is not coming back. - The
fovea-reviewGitHub App and its installations stay in GitHub. GitHub keeps sending webhook deliveries tofovea-wh.fzymgc.netuntil the operator suspends or uninstalls the app, and each delivery fails. The app settings page ishttps://github.com/settings/apps/fovea-review. - Velero holds the last daily backups of the
foveanamespace for the retention period of the schedule. - To roll back, revert this pull request. ArgoCD recreates the workloads with empty PVCs, Terraform re-registers the key with the value in
cluster/litellm-keys/foveaand re-creates the Keycloak client, and the Vault grant for theoidcpath must exist before the keycloak apply.