Deploy fovea (PR-review engine) via Argo CD

Design bead: hl-uh07 Source issue: #1239 Status: Design Date: 2026-06-09

Summary

fovea is the Go successor to octopus — a self-hosted GitHub PR-review engine (scout → gate → adversarial deep-dive). It ships as a generic Helm chart published to GHCR. This spec covers the cluster wiring needed to run it: a multi-source ArgoCD Application (OCI chart + git-sourced cluster wiring), a Vault-backed ExternalSecret, a Cloudflare Tunnel webhook route, and a manually-created GitHub App.

fovea is stateless (single replica, no database, no PVC) and ships in shadow mode (publish_enabled: false) so it observes and emits metrics without posting PR comments until explicitly enabled.

This spec validates and corrects issue #1239. The issue is fundamentally sound and well-grounded in repo precedent, but its example manifests omit two conventions the established agent-memory precedent requires (a finalizer and an ExternalSecret ignoreDifferences block). Those corrections are folded in below.

Grounding & validation of issue #1239

Validated against the live repo (workspace off trunk()) and the authoritative charts/fovea/values.yaml in seanb4t/fovea.

Confirmed correct

Claim	Evidence
Chart valuesObject keys	Match charts/fovea/values.yaml: config.github.app_id/installation_id, config.flags.publish_enabled, privateKeySecret/webhookSecret/anthropicKey, observability.otlpEndpoint
otlpEndpoint: http://…:4317	Chart encodes transport in the URL scheme: http:// = plaintext in-cluster collector. Correct as written (differs from engram, which omits the scheme and uses a separate otlpInsecure flag)
Multi-source OCI(A) + git(B)	Matches argocd/cluster-app/templates/agent-memory.yaml
ExternalSecret shape (external-secrets.io/v1)	Matches octopus/secrets.yaml + agent-memory/external-secret.yaml
ghcr-pull-secret	Matches agent-memory/ghcr-pull-secret.yaml; Vault fzymgc-house/cluster/ghcr/pull-secret
Cloudflare webhook_services map shape	-wh suffix + fzymgc.net → fovea-wh.fzymgc.net; tunnel ingress loop + DNS resources need no change
core-services project	Exists (shared-resources/argocd-projects.yaml)
GitHub App is manual	Correct — there is no Terraform/API path to create an App
OCI Helm repo registration	ghcr.io/seanb4t/charts is already registered as an ArgoCD helm repo (shared-resources/ghcr-seanb4t-charts.yaml) — no new repo credential needed
OTLP auth + network gates	agent-memory/otlp-headers-secret.yaml (raw token, Authorization=<token>, Vault …/clickstack otel_ingest_api_key) + clickstack/networkpolicy.yaml per-namespace fromEndpoints on :4317. Both required or telemetry silently fails (see surfaces A + E)

Corrections folded into this spec

1. Missing finalizers: [resources-finalizer.argocd.argoproj.io] — agent-memory carries it; a stored memory note records orphaned-resource incidents when an Application that prunes resources lacks it.
2. Missing ignoreDifferences for ExternalSecret — agent-memory carries a block for conversionStrategy/decodingStrategy/metadataPolicy; without it the Application sits perpetually OutOfSync as ESO mutates those defaults.
3. targetRevision: main → HEAD for the git source; <this-repo> → https://github.com/fzymgc-house/selfhosted-cluster (repo convention).
4. ExternalSecret adds refreshPolicy: Periodic + deletionPolicy: Delete to match house style.
5. Vault policy — no change needed. ESO authenticates with fzymgc-cluster-secret-reader, which wildcards secret/data/*. (CLAUDE.md’s “update Vault policies for new secret paths” rule fires only for enumerated per-app policies, not this wildcard reader.)
6. OTLP export needs two extra gates the issue’s valuesObject omits — the clickstack collector rejects unauthenticated OTLP and whitelists senders by namespace. Setting otlpEndpoint alone (as #1239 does) yields a silently broken telemetry path. Added: a fovea-otlp-headers ExternalSecret + otlpHeadersSecret wiring (surface A/B) and a clickstack NetworkPolicy fromEndpoints entry (surface E). Found by the design-reviewer; confirmed against the agent-memory precedent.

Decisions

#	Decision	Rationale
D1	Store all 5 keys in Vault fzymgc-house/cluster/fovea	Canonical single record of fovea’s GitHub App + Anthropic credentials. See D1-note.
D2	Rollout is infra-first: tunnel/DNS → GitHub App → ArgoCD deploy	The App’s webhook URL must resolve before the App can deliver events; shadow mode means early events are harmless
D3	No NetworkPolicy at launch; L7/egress security comes via the mesh (D4), optional Cilium L3/L4 CNP as later defense-in-depth	Cluster has no global default-deny (octopus/agent-memory run policy-free); Cilium L7/toFQDNs is broken cluster-wide (hl-bdr), and the strategic L7 layer is Istio (D4), so a Cilium L7 CNP would be throwaway work
D4	Restrict fovea egress to an allowlist (api.anthropic.com + api.github.com) — delivered via the Istio mesh (epic hl-uljo), with fovea as the pilot workload (hl-z5fc)	fovea holds a GitHub App private key + Anthropic key and ingests untrusted PR content — a high-value exfil target. Cilium toFQDNs can’t express the allowlist (hl-bdr). Rather than build a throwaway standalone proxy, egress is delivered by the planned Istio mesh (Istio sidecar interception bypasses the broken eBPF/TPROXY path entirely; ServiceEntry + REGISTRY_ONLY + SNI gives the allowlist + mTLS identity). Not a launch blocker — fovea ships with default-allow egress now and is onboarded to the mesh first.

D1-note (intentional redundancy): the fovea chart reads app_id / installation_id only from config.* (rendered to FOVEA_GITHUB__* env), not from the mounted Secret — only the private key, webhook secret, and Anthropic key have secret→env wiring in the chart. Therefore the two numeric IDs still appear in the Application valuesObject (plaintext in git) for the chart to function, while Vault holds the complete 5-key record as the canonical reference. This is deliberate, not a contradiction — a future reader should not “fix” it by deleting the plaintext IDs. The IDs are non-secret (visible to anyone with the App).

Architecture

GitHub ── PR event ──▶ Cloudflare Tunnel ──▶ fovea-wh.fzymgc.net
                                                    │
                                                    ▼
                                   fovea.fovea.svc.cluster.local:8080  (webhook)
                                                    │
                          ┌─────────────────────────┼─────────────────────────┐
                          ▼                          ▼                         ▼
                   :8081 /healthz            :9090 /metrics          OTLP → clickstack
                                                                  cs-otel-collector:4317
                                                    │
                                          Anthropic API (deep-dive LLM)
                                          GitHub API   (post review comment)

Single stateless replica. Secrets injected by reference from the fovea-app Secret (ESO-managed). Image pulled from private GHCR via ghcr-pull-secret.

Change surfaces

A. argocd/app-configs/fovea/ (Source B — git)

• kustomization.yaml — namespace: fovea; resources: external-secret.yaml, ghcr-pull-secret.yaml, otlp-headers-secret.yaml. No namespace.yaml — the namespace is created by the Application’s CreateNamespace=true syncOption, matching the agent-memory app-config (which ships none). This deviates from issue #1239, which lists a namespace.yaml; the syncOption makes it redundant.
• ghcr-pull-secret.yaml — copy of agent-memory/ghcr-pull-secret.yaml (refreshPolicy: Periodic, refreshInterval: 1h, dockerconfigjson template, Vault fzymgc-house/cluster/ghcr/pull-secret), namespace fovea.
• external-secret.yaml — name: fovea-app, namespace fovea, refreshPolicy: Periodic, refreshInterval: 5m, deletionPolicy: Delete. Templates the 3 chart-consumed keys from fzymgc-house/cluster/fovea:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: fovea-app
  namespace: fovea
spec:
  refreshPolicy: Periodic
  refreshInterval: 5m
  secretStoreRef: { name: vault, kind: ClusterSecretStore }
  target:
    name: fovea-app
    creationPolicy: Owner
    deletionPolicy: Delete
    template:
      type: Opaque
      data:
        github_app_private_key: "{{ .github_app_private_key }}"
        github_webhook_secret: "{{ .github_webhook_secret }}"
        anthropic_api_key: "{{ .anthropic_api_key }}"
  data:
    - { secretKey: github_app_private_key, remoteRef: { key: fzymgc-house/cluster/fovea, property: github_app_private_key } }
    - { secretKey: github_webhook_secret,  remoteRef: { key: fzymgc-house/cluster/fovea, property: github_webhook_secret } }
    - { secretKey: anthropic_api_key,       remoteRef: { key: fzymgc-house/cluster/fovea, property: anthropic_api_key } }

• otlp-headers-secret.yaml — copy of agent-memory/otlp-headers-secret.yaml (name: fovea-otlp-headers, namespace fovea). The clickstack OTel collector rejects unauthenticated OTLP — every in-cluster sender must present the HyperDX ingest token. The chart injects this Secret verbatim as OTEL_EXPORTER_OTLP_HEADERS, so the template renders the full header string. The token is the RAW value with NO Bearer scheme (Authorization=<token>) — Bearer <token> is rejected and the error echoes the token into the sender’s logs (treat a mismatch as exposure). Sourced from the existing clickstack Vault path (no new secret to populate):

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: fovea-otlp-headers
  namespace: fovea
spec:
  refreshPolicy: Periodic
  refreshInterval: 5m
  secretStoreRef: { name: vault, kind: ClusterSecretStore }
  target:
    name: fovea-otlp-headers
    creationPolicy: Owner
    deletionPolicy: Delete
    template:
      data:
        headers: "Authorization={{ .token }}"
  data:
    - { secretKey: token, remoteRef: { key: fzymgc-house/cluster/clickstack, property: otel_ingest_api_key } }

B. argocd/cluster-app/templates/fovea.yaml (the Application)

cluster-app is a plain app-of-apps (empty values.yaml, no Helm gating) — a new app is a literal Application manifest dropped into templates/.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: fovea
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: core-services
  sources:
    # Source A — workloads from the fovea OCI Helm chart.
    - repoURL: ghcr.io/seanb4t/charts
      chart: fovea
      targetRevision: 0.1.2   # bump in lockstep with image tag as releases ship
      helm:
        releaseName: fovea
        valuesObject:
          image: { repository: ghcr.io/seanb4t/fovea, tag: "0.1.2" }
          imagePullSecrets: [{ name: ghcr-pull-secret }]
          config:
            github: { app_id: <APP_ID>, installation_id: <INSTALL_ID> }  # plaintext; chart reads from config (see D1-note)
            flags: { publish_enabled: false }                            # shadow mode first
          privateKeySecret: { name: fovea-app, key: github_app_private_key }
          webhookSecret:    { name: fovea-app, key: github_webhook_secret }
          anthropicKey:     { name: fovea-app, key: anthropic_api_key }
          observability:
            otlpEndpoint: "http://cs-otel-collector.clickstack.svc.cluster.local:4317"
            otlpHeadersSecret: { name: fovea-otlp-headers, key: headers }   # collector rejects unauthenticated OTLP
    # Source B — cluster wiring (ExternalSecrets + pull secret).
    - repoURL: https://github.com/fzymgc-house/selfhosted-cluster
      targetRevision: HEAD
      path: argocd/app-configs/fovea
  destination:
    server: https://kubernetes.default.svc
    namespace: fovea
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [ServerSideApply=true, CreateNamespace=true]
  ignoreDifferences:
    - kind: ExternalSecret
      group: external-secrets.io
      jqPathExpressions:
        - .spec.data[].remoteRef.conversionStrategy
        - .spec.data[].remoteRef.decodingStrategy
        - .spec.data[].remoteRef.metadataPolicy

C. tf/cloudflare/variables.tf (manual apply)

Add to the webhook_services map default:

fovea = { service_url = "http://fovea.fovea.svc.cluster.local:8080" }

This yields fovea-wh.fzymgc.net → fovea :8080 (the tunnel.tf ingress loop and the cloudflare_dns_record.webhook_services for_each need no change). Requires manual terraform apply in tf/cloudflare (per MANUAL_APPLY.md; not HCP-automated).

D. Vault fzymgc-house/cluster/fovea (manual)

Populated by hand after the GitHub App is created (D2):

key	value
github_app_id	numeric App ID (canonical; also copied to valuesObject)
github_installation_id	numeric Installation ID (canonical; also copied to valuesObject)
github_app_private_key	the .pem contents
github_webhook_secret	the webhook secret
anthropic_api_key	Anthropic API key

No Vault policy change (ESO reader wildcards secret/data/*). The OTLP token (surface A’s fovea-otlp-headers) reads the existing fzymgc-house/cluster/clickstack path — nothing new to populate there.

E. argocd/app-configs/clickstack/networkpolicy.yaml (network gate)

The clickstack-default CiliumNetworkPolicy whitelists OTLP ingress per source namespace. A new sender whose namespace is not enumerated has its exporter dials silently time out (dial tcp …:4317: i/o timeout in fovea’s logs) — the traffic is blackholed, never refused. Add a fromEndpoints entry for the fovea namespace scoped to 4317/TCP, mirroring the existing agent-memory entry:

    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: fovea
      toPorts:
        - ports:
            - port: "4317"
              protocol: TCP

(This is the one cross-namespace edit in the fovea PR. It is not a fovea-owned resource — it lives in the clickstack app-config and is synced by the clickstack Application.)

Rollout runbook (infra-first)

1. Merge repo PR with changes A + B + C + E. Operator runs terraform apply in tf/cloudflare → fovea-wh.fzymgc.net DNS + tunnel route live. The Application syncs immediately with <APP_ID>/<INSTALL_ID> still as placeholders — expect fovea to be transiently unhealthy (invalid GitHub config) between this step and step 3. Shadow mode + statelessness make this window harmless; it resolves when step 3 fills the real IDs.
2. Create GitHub App (manual): permissions Pull requests: read & write, Contents: read-only; subscribe to Pull request events; webhook URL https://fovea-wh.fzymgc.net/webhook + secret; generate .pem; note App ID + Installation ID; install on target repo(s).
3. Populate Vault fzymgc-house/cluster/fovea (5 keys), then fill <APP_ID> / <INSTALL_ID> in cluster-app/templates/fovea.yaml and merge.
4. ArgoCD syncs → confirm pod healthy on :8081/healthz.

Verification, then go-live

1. Sync; confirm pods healthy (/healthz on :8081).
2. Confirm OTLP export works — fovea logs show no :4317 i/o timeout or 401, and the funnel/telemetry lands in clickstack. (Both gates from surfaces A/B/E must be in place; a half-wired otlpEndpoint fails silently.)
3. Open a PR with a reachable nil-deref on a repo the App is installed on. In shadow mode no comment posts — confirm the funnel on :9090/metrics (fovea_findings_funnel{stage="scouted|gated|verified"}).
4. Flip config.flags.publish_enabled: true in the Application valuesObject, sync, re-trigger → expect exactly one inline comment (guarded variant → none). See fovea docs/RUNBOOK.md §4 for the full §15 acceptance walkthrough.

Error handling & risk notes

• Anthropic, not OpenAI — fovea uses anthropic_api_key (octopus used openai_api_key); the LLM provider is hardcoded to Anthropic in the chart.
• OTLP scheme trap — otlpEndpoint must keep the http:// scheme for the plaintext in-cluster collector; do not copy engram’s scheme-less form.
• OTLP silently fails without both gates — the collector rejects unauthenticated OTLP (needs fovea-otlp-headers, raw token no Bearer) and blackholes senders whose namespace is not whitelisted in clickstack’s NetworkPolicy (:4317). Both are wired (surfaces A/B/E). Symptom if either is missing: dial …:4317 i/o timeout (network) or 401 / scheme or token does not match (auth) in fovea’s logs — and a 401 echoes the token, so treat it as a token-exposure event and rotate.
• Image pull — relies on ghcr-pull-secret in the fovea namespace; the account-scoped PAT at fzymgc-house/cluster/ghcr/pull-secret already covers seanb4t packages.
• NetworkPolicy deferred — fovea launches policy-free (consistent with octopus/agent-memory). Egress to Anthropic + GitHub works via cluster default-allow. Hardening tracked in hl-z5fc; proper API-scoped toFQDNs egress is blocked on hl-bdr (broken L7 DNS proxy).
• Egress destination restriction deferred (D4) — fovea is a high-value exfil target (GitHub App private key + Anthropic key + untrusted PR input). The desired control is an allowlist limiting outbound to api.anthropic.com + api.github.com. Because Cilium toFQDNs is unusable (hl-bdr), this is delivered via the Istio mesh (epic hl-uljo) rather than a CNP, a standalone proxy, or Cilium Egress Gateway (which controls source IP, not destination). Istio’s pod-netns iptables interception bypasses the broken eBPF/TPROXY path entirely. fovea is the pilot workload (hl-z5fc); until the mesh lands it has default-allow egress like every comparable app. Not a launch blocker.

Out of scope

• The fovea application code / chart itself (owned by seanb4t/fovea).
• Egress destination restriction + mTLS — delivered via the Istio mesh (epic hl-uljo), fovea as pilot (hl-z5fc).
• The Istio mesh adoption itself (Cilium+Istio coexistence, sidecar choice, per-app migration) — designed under epic hl-uljo.
• Multi-repo App installation strategy beyond the initial target repo(s).

Follow-up beads

• hl-uljo (epic) — Adopt Istio service mesh (L7 + mTLS + egress); demote Cilium to CNI + L3/L4. Driven by the unreliability of Cilium L7 enforcement here (hl-bdr); Istio’s pod-netns interception sidesteps it. Sidecar mode is the low-friction path on this datapath (ambient conflicts with required bpf.masquerade=true).
• hl-z5fc (child of hl-uljo) — fovea mesh onboarding (Istio pilot): mTLS + egress allowlist (api.anthropic.com + api.github.com). fovea is the pilot workload — narrow egress, sensitive creds, no internal-service deps.