Skip to content

fovea is retired — the review service, its LiteLLM key and lanes, its Keycloak client and its webhook host leave the cluster

Date: 2026-09-28 Status: Accepted — supersedes hl-0ah0, hl-93me, hl-jecc and hl-buts Deciders: Sean Brandt

fovea is a GitHub App that reviews pull requests. It ran in the fovea namespace with a dedicated Qdrant, and it called the LiteLLM Service through its own virtual key and three lanes (fovea-scout, fovea-deepdive and fovea-embed). Work on fovea is paused.

The fovea LiteLLM key reached its max_budget of $100 on 2026-09-25 at 04:52 UTC. Since then LiteLLM refused every fovea call, so fovea did no reviews. fovea kept retrying, and LiteLLM logged thousands of refusals each hour. Issue #2468 asked for the removal. It supersedes #2271 (the fovea budget decision) and #2208 (the fovea engram learning loop).

Remove fovea from the cluster:

  • The fovea ArgoCD Application, argocd/app-configs/fovea/ and argocd/app-configs/fovea-chart/. ArgoCD prunes the namespace, the Deployments, the ExternalSecrets and the three PVCs (fovea-audit-js, fovea-index and qdrant-fovea-data).
  • The fovea entry in tf/litellm/main.tf. Terraform deletes the virtual key.
  • The three fovea lanes in argocd/app-configs/litellm-chart/values.yaml. The zdr-probe-embed control lane now mirrors octopus-embed (openrouter/baai/bge-m3), because it must share a model with a live lane.
  • The fovea rules in the litellm-proxy-ingress and clickstack CiliumNetworkPolicies.
  • The fovea webhook host in tf/cloudflare/variables.tf. Terraform deletes the fovea-wh.fzymgc.net record and the tunnel ingress rule.
  • The fovea-engram Keycloak client. The client-credentials loop into engram was never turned on (#2208), so nothing consumes it.
  • The fovea row in scripts/seed-litellm-vault.sh.

Terraform forgets the Vault record secret/fzymgc-house/cluster/fovea/oidc instead of deleting it, because the terraform-keycloak-admin policy grants no delete on that path. The Vault policy grant for the path stays until the keycloak apply is complete, and a later pull request removes it. The removed block in tf/keycloak/fovea_engram.tf goes with it.

  • No pull request review from fovea on any repository. octopus stays.
  • Two Deployments, one namespace, three PVCs, one LiteLLM key, three lanes, one Keycloak client and one public hostname fewer to run.
  • Three Vault paths keep their data: cluster/fovea (the GitHub App private key, webhook secret and the Anthropic and Qdrant keys), cluster/fovea/oidc and cluster/litellm-keys/fovea. The operator deletes them by hand when fovea is not coming back.
  • The fovea-review GitHub App and its installations stay in GitHub. GitHub keeps sending webhook deliveries to fovea-wh.fzymgc.net until the operator suspends or uninstalls the app, and each delivery fails. The app settings page is https://github.com/settings/apps/fovea-review.
  • Velero holds the last daily backups of the fovea namespace for the retention period of the schedule.
  • To roll back, revert this pull request. ArgoCD recreates the workloads with empty PVCs, Terraform re-registers the key with the value in cluster/litellm-keys/fovea and re-creates the Keycloak client, and the Vault grant for the oidc path must exist before the keycloak apply.