fovea Argo CD deployment — implementation plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Deploy the fovea Go PR-review engine into the cluster via a multi-source ArgoCD Application (OCI Helm chart + git-sourced cluster wiring), in shadow mode, with authenticated OTLP telemetry.

Architecture: Source A renders the ghcr.io/seanb4t/charts/fovea chart; Source B (this repo, argocd/app-configs/fovea) supplies the ExternalSecrets + pull secret. GitHub reaches fovea through a Cloudflare Tunnel webhook (fovea-wh.fzymgc.net → fovea.fovea.svc:8080). The GitHub App and Vault secret are created manually (no Terraform/API path to create a GitHub App). fovea ships with publish_enabled: false (shadow mode) and is flipped live after verification.

Tech Stack: ArgoCD (app-of-apps Helm chart), Kustomize, External Secrets Operator (Vault ClusterSecretStore), Cilium NetworkPolicy, Terraform (Cloudflare provider), Vault KV v2, jj (colocated jj+git VCS).

Spec: docs/engineering/specs/2026-06-09-fovea-deploy-design.md (design bead hl-uh07)

VCS note: This is a colocated jj+git repo. Use jj for all VCS ops; mutating git is blocked. Pass --no-pager on every jj command, and --git when reading diffs. Branch protection on main requires a PR.

Validation note: These are declarative manifests, not application code — there are no unit tests. Each task’s “test” is: the manifest renders (kustomize build / helm template), lints clean (yamllint), or formats clean (terraform fmt). The real integration test is the post-merge ArgoCD sync (Task 9).

File structure

Path	Surface	Responsibility
`argocd/app-configs/fovea/kustomization.yaml`	A	Source B kustomization; lists the 3 ExternalSecrets
`argocd/app-configs/fovea/ghcr-pull-secret.yaml`	A	GHCR image-pull creds (ESO)
`argocd/app-configs/fovea/external-secret.yaml`	A	`fovea-app` Secret — 3 chart-consumed app secrets (ESO)
`argocd/app-configs/fovea/otlp-headers-secret.yaml`	A	`fovea-otlp-headers` — HyperDX OTLP bearer (ESO)
`argocd/app-configs/clickstack/networkpolicy.yaml`	E	Modify — whitelist the `fovea` namespace for OTLP `:4317`
`argocd/cluster-app/templates/fovea.yaml`	B	The multi-source ArgoCD Application
`tf/cloudflare/variables.tf`	C	Modify — add `fovea` to `webhook_services`

Manual (no repo file): GitHub App creation, terraform apply, Vault fzymgc-house/cluster/fovea population, filling <APP_ID>/<INSTALL_ID>.

Task 1: Create the `fovea` app-config ExternalSecrets (Surface A)

Files:

Create: argocd/app-configs/fovea/ghcr-pull-secret.yaml
Create: argocd/app-configs/fovea/external-secret.yaml
Create: argocd/app-configs/fovea/otlp-headers-secret.yaml
Create: argocd/app-configs/fovea/kustomization.yaml
Step 1: Create the GHCR pull-secret ExternalSecret

Create argocd/app-configs/fovea/ghcr-pull-secret.yaml:

# GHCR image-pull credentials for the private fovea image.
# Vault path: secret/fzymgc-house/cluster/ghcr/pull-secret (username + read:packages PAT).
# The PAT is account-scoped to seanb4t and covers seanb4t packages.
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: ghcr-pull-secret
  namespace: fovea
spec:
  refreshPolicy: Periodic
  refreshInterval: 1h
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: ghcr-pull-secret
    creationPolicy: Owner
    deletionPolicy: Delete
    template:
      type: kubernetes.io/dockerconfigjson
      data:
        .dockerconfigjson: |
          {
            "auths": {
              "ghcr.io": {
                "username": "{{ .username }}",
                "password": "{{ .password }}",
                "auth": "{{ printf "%s:%s" .username .password | b64enc }}"
              }
            }
          }
  data:
    - secretKey: username
      remoteRef:
        key: fzymgc-house/cluster/ghcr/pull-secret
        property: username
    - secretKey: password
      remoteRef:
        key: fzymgc-house/cluster/ghcr/pull-secret
        property: password

Step 2: Create the app ExternalSecret

Create argocd/app-configs/fovea/external-secret.yaml:

---
# fovea GitHub App + Anthropic credentials. Only the 3 keys the chart consumes
# are templated; github_app_id / github_installation_id are non-secret and live
# in the Application valuesObject (the chart reads them from config, not a Secret).
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: fovea-app
  namespace: fovea
spec:
  refreshPolicy: Periodic
  refreshInterval: 5m
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: fovea-app
    creationPolicy: Owner
    deletionPolicy: Delete
    template:
      type: Opaque
      data:
        github_app_private_key: "{{ .github_app_private_key }}"
        github_webhook_secret: "{{ .github_webhook_secret }}"
        anthropic_api_key: "{{ .anthropic_api_key }}"
  data:
    - secretKey: github_app_private_key
      remoteRef: { key: fzymgc-house/cluster/fovea, property: github_app_private_key }
    - secretKey: github_webhook_secret
      remoteRef: { key: fzymgc-house/cluster/fovea, property: github_webhook_secret }
    - secretKey: anthropic_api_key
      remoteRef: { key: fzymgc-house/cluster/fovea, property: anthropic_api_key }

Step 3: Create the OTLP headers ExternalSecret

Create argocd/app-configs/fovea/otlp-headers-secret.yaml:

---
# HyperDX ingest token for fovea's OTLP export. cs-otel-collector rejects
# unauthenticated OTLP; the chart injects this Secret VERBATIM as
# OTEL_EXPORTER_OTLP_HEADERS. The collector expects the RAW token as the
# authorization value — NO "Bearer " scheme. "Bearer <token>" is rejected
# with "scheme or token does not match" AND echoes the token into the
# sender's logs (treat a mismatch as token exposure).
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: fovea-otlp-headers
  namespace: fovea
spec:
  refreshPolicy: Periodic
  refreshInterval: 5m
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: fovea-otlp-headers
    creationPolicy: Owner
    deletionPolicy: Delete
    template:
      data:
        headers: "Authorization={{ .token }}"
  data:
    - secretKey: token
      remoteRef:
        key: fzymgc-house/cluster/clickstack
        property: otel_ingest_api_key

Step 4: Create the kustomization

Create argocd/app-configs/fovea/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: fovea

# Source B of the multi-source fovea Application. Workloads are rendered by the
# fovea OCI chart (Source A). This path keeps ONLY the cluster wiring the chart
# can't own: the ExternalSecrets (app creds, GHCR pull, OTLP auth header).
# The namespace itself is created by the Application's CreateNamespace=true
# syncOption (matching agent-memory) — no namespace.yaml here.
resources:
  - external-secret.yaml      # fovea-app: GitHub App private key + webhook secret + anthropic key
  - ghcr-pull-secret.yaml     # pull creds for the private fovea image
  - otlp-headers-secret.yaml  # HyperDX ingest bearer for fovea OTLP export

Step 5: Validate the kustomization renders

Run: kubectl kustomize argocd/app-configs/fovea Expected: PASS — emits the 3 ExternalSecret manifests, all with namespace: fovea, no errors. (kustomize is built into kubectl; offline render, no cluster needed.)

Step 6: Lint the YAML

Run: yamllint argocd/app-configs/fovea/ Expected: PASS — no errors (warnings about line length acceptable if consistent with repo norms).

Step 7: Commit

Run: jj --no-pager commit -m "feat(fovea): app-config ExternalSecrets (app creds, GHCR pull, OTLP auth) [hl-uh07]"

Task 2: Whitelist the `fovea` namespace for OTLP in clickstack (Surface E)

Files:

Modify: argocd/app-configs/clickstack/networkpolicy.yaml (insert after the agent-memory OTLP ingress entry, currently around line 63)
Step 1: Add the fovea OTLP ingress rule

In argocd/app-configs/clickstack/networkpolicy.yaml, find the agent-memory ingress entry:

    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: agent-memory
      toPorts:
        - ports:
            - port: "4317"
              protocol: TCP

Insert this new entry immediately after it (before the prometheus/:8888 entry):

    # Allow fovea (fovea ns) to push OTLP telemetry into cs-otel-collector :4317.
    # Same missed-whitelist trap as the otel-scraper / agent-memory entries above —
    # without this, fovea's exporter dials time out (1/s "dial tcp …:4317: i/o
    # timeout" in fovea's logs) and ALL telemetry is silently dropped. (hl-uh07)
    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: fovea
      toPorts:
        - ports:
            - port: "4317"
              protocol: TCP

Step 2: Validate the clickstack kustomization still renders

Run: kubectl kustomize argocd/app-configs/clickstack >/dev/null && echo OK Expected: OK (no render error from the edit).

Step 3: Confirm the fovea rule is present in the rendered output

Run: kubectl kustomize argocd/app-configs/clickstack | rg -A2 "namespace: fovea" Expected: shows the io.kubernetes.pod.namespace: fovea matchLabels under the CiliumNetworkPolicy.

Step 4: Lint the YAML

Run: yamllint argocd/app-configs/clickstack/networkpolicy.yaml Expected: PASS.

Step 5: Commit

Run: jj --no-pager commit -m "feat(clickstack): whitelist fovea ns for OTLP :4317 ingress [hl-uh07]"

Task 3: Create the fovea ArgoCD Application (Surface B)

Files:

Create: argocd/cluster-app/templates/fovea.yaml
Step 1: Create the Application manifest

Create argocd/cluster-app/templates/fovea.yaml (leave <APP_ID> / <INSTALL_ID> as placeholders — Task 8 fills them):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: fovea
  namespace: argocd
  # fovea PR-review engine (Go successor to octopus). Multi-source: Source A =
  # the fovea OCI chart (workloads), Source B = the app-config ExternalSecrets.
  # finalizer ensures a clean prune cascade (agent-memory precedent, hl-uh07).
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: core-services
  sources:
    # Source A — workloads from the fovea OCI Helm chart.
    - repoURL: ghcr.io/seanb4t/charts
      chart: fovea
      targetRevision: 0.1.2 # bump in lockstep with the image tag as releases ship
      helm:
        releaseName: fovea
        valuesObject:
          image:
            repository: ghcr.io/seanb4t/fovea
            tag: "0.1.2"
          imagePullSecrets:
            - name: ghcr-pull-secret
          config:
            # app_id/installation_id are non-secret; the chart reads them from
            # config (FOVEA_GITHUB__*), not a Secret — so they live here in git.
            github:
              app_id: <APP_ID>
              installation_id: <INSTALL_ID>
            flags:
              publish_enabled: false # shadow mode — never posts until flipped
          privateKeySecret: { name: fovea-app, key: github_app_private_key }
          webhookSecret: { name: fovea-app, key: github_webhook_secret }
          anthropicKey: { name: fovea-app, key: anthropic_api_key }
          observability:
            # http:// = plaintext in-cluster collector (transport is in the scheme).
            otlpEndpoint: "http://cs-otel-collector.clickstack.svc.cluster.local:4317"
            # collector rejects unauthenticated OTLP — raw token, no Bearer.
            otlpHeadersSecret: { name: fovea-otlp-headers, key: headers }
    # Source B — cluster wiring (ExternalSecrets) that stays in this repo.
    - repoURL: https://github.com/fzymgc-house/selfhosted-cluster
      targetRevision: HEAD
      path: argocd/app-configs/fovea
  destination:
    server: https://kubernetes.default.svc
    namespace: fovea
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
      - CreateNamespace=true
  ignoreDifferences:
    - kind: ExternalSecret
      group: external-secrets.io
      jqPathExpressions:
        - .spec.data[].remoteRef.conversionStrategy
        - .spec.data[].remoteRef.decodingStrategy
        - .spec.data[].remoteRef.metadataPolicy

Step 2: Validate the cluster-app chart still templates

Run: helm template cluster-app argocd/cluster-app --show-only templates/fovea.yaml Expected: PASS — renders the fovea Application unchanged (cluster-app has no Helm templating; the file passes through verbatim). Confirms no YAML syntax error breaks the chart render.

Step 3: Lint the YAML

Run: yamllint argocd/cluster-app/templates/fovea.yaml Expected: PASS.

Step 4: Commit

Run: jj --no-pager commit -m "feat(fovea): multi-source ArgoCD Application (shadow mode) [hl-uh07]"

Task 4: Add fovea to the Cloudflare webhook services (Surface C)

Files:

Modify: tf/cloudflare/variables.tf:83-87 (the webhook_services default, after the octopus entry)
Step 1: Add the fovea webhook entry

In tf/cloudflare/variables.tf, inside variable "webhook_services" → default, add after the octopus block (before the closing } of default):

    fovea = {
      # fovea self-hosted PR review — GitHub App webhook.
      # GitHub POSTs to /webhook, forwarded as-is to the fovea Service :8080.
      service_url = "http://fovea.fovea.svc.cluster.local:8080"
    }

This yields fovea-wh.fzymgc.net → fovea :8080. The tunnel.tf ingress loop and the cloudflare_dns_record.webhook_services for_each consume the map automatically — no other edit needed.

Step 2: Format check

Run: terraform -chdir=tf/cloudflare fmt -check Expected: PASS (no diff). If it reports the file, run terraform -chdir=tf/cloudflare fmt and re-check.

Step 3: Validate (if providers are initialized)

Run: terraform -chdir=tf/cloudflare validate Expected: Success! The configuration is valid. — IF .terraform/ is initialized. If it errors with “provider not installed” / “module not installed”, skip validate (this workspace uses HCP Terraform; fmt is sufficient locally and the real plan runs in HCP / via MANUAL_APPLY.md).

Step 4: Commit

Run: jj --no-pager commit -m "feat(cloudflare): add fovea webhook service (fovea-wh.fzymgc.net) [hl-uh07]"

Task 5: Open PR1 (repo wiring) and merge

Files: none (VCS/PR actions)

Step 1: Fetch and rebase onto latest main

Run:

jj --no-pager git fetch
jj --no-pager rebase -s "$(jj --no-pager log -r 'roots(trunk()..@)' --no-graph -T 'change_id.short(12)')" -o main@origin --skip-emptied

Expected: the fovea commits (spec, plan, Tasks 1–4) rebase cleanly onto main@origin.

Step 2: Create a bookmark and push

Run:

jj --no-pager bookmark create feat/fovea-deploy -r @
jj --no-pager git push -b feat/fovea-deploy

Expected: bookmark pushed to origin.

Step 3: Open the PR

Run:

gh pr create --head feat/fovea-deploy --base main \
  --title "feat(fovea): deploy PR-review engine via Argo CD (shadow mode) [hl-uh07]" \
  --body "Deploys fovea per docs/engineering/specs/2026-06-09-fovea-deploy-design.md. Surfaces A (app-config ExternalSecrets), B (multi-source Application, placeholder App IDs), C (cloudflare webhook), E (clickstack OTLP whitelist). Ships in shadow mode (publish_enabled=false). Manual follow-up: terraform apply, GitHub App creation, Vault population, fill App IDs (PR2). 🤖 Generated with [Claude Code](https://claude.com/claude-code)"

Expected: PR URL printed.

Step 4: Merge after checks pass

Wait for CI/lint checks and review, then merge via the GitHub UI or gh pr merge. After merge, fetch: Run: jj --no-pager git fetch Expected: main@origin advanced with the merge.

NOTE: After merge, ArgoCD will sync the fovea Application with placeholder <APP_ID>/<INSTALL_ID>. fovea will be transiently unhealthy (invalid GitHub config) until Task 8. Shadow mode + statelessness make this harmless.

Task 6: Apply Cloudflare Terraform (manual)

Files: none (operator action)

Step 1: Apply the tunnel/DNS change

Per tf/cloudflare/MANUAL_APPLY.md, apply the tf/cloudflare workspace (HCP Terraform run or local terraform -chdir=tf/cloudflare apply). This creates the fovea-wh.fzymgc.net DNS record and tunnel ingress route.

Step 2: Verify the webhook host resolves

Run: dig +short fovea-wh.fzymgc.net Expected: a CNAME/A answer pointing at <tunnel-id>.cfargotunnel.com (proxied via Cloudflare).

The origin (fovea.fovea.svc:8080) need not be healthy yet — the GitHub App (Task 7) only needs a resolvable webhook URL to be created. Shadow mode means early webhook deliveries are harmless.

Task 7: Create the GitHub App + populate Vault (manual)

Files: none (operator action)

Step 1: Create the GitHub App

In GitHub (UI or App-manifest flow), create an App with:

Permissions: Pull requests: read & write, Contents: read-only
Subscribe to: Pull request events
Webhook URL: https://fovea-wh.fzymgc.net/webhook
Set a webhook secret (generate a strong random value; keep it)
Generate a private key (.pem); note the App ID and (after install) the Installation ID
Install the App on the target repo(s)
Step 2: Populate Vault

Run (substitute real values; anthropic_api_key is an existing Anthropic key):

vault kv put secret/fzymgc-house/cluster/fovea \
  github_app_id="<APP_ID>" \
  github_installation_id="<INSTALL_ID>" \
  github_app_private_key=@app.pem \
  github_webhook_secret="<WEBHOOK_SECRET>" \
  anthropic_api_key="<ANTHROPIC_KEY>"

Expected: Success! Data written to: secret/fzymgc-house/cluster/fovea.

Step 3: Confirm ESO can read it (no policy change needed)

Run: vault kv get -format=json secret/fzymgc-house/cluster/fovea | jq '.data.data | keys' Expected: ["anthropic_api_key","github_app_id","github_app_private_key","github_installation_id","github_webhook_secret"]. (ESO authenticates with fzymgc-cluster-secret-reader, which wildcards secret/data/* — no policy edit.)

Task 8: Fill App IDs and open PR2

Files:

Modify: argocd/cluster-app/templates/fovea.yaml (replace <APP_ID> and <INSTALL_ID>)
Step 1: Start a fresh change off latest main

Run:

jj --no-pager git fetch
jj --no-pager new main@origin

Step 2: Replace the placeholders

In argocd/cluster-app/templates/fovea.yaml, set the real numeric values:

            github:
              app_id: 123456
              installation_id: 78901234

(Use the actual App ID / Installation ID from Task 7. They are non-secret.)

Step 3: Validate render + lint

Run: helm template cluster-app argocd/cluster-app --show-only templates/fovea.yaml | rg "app_id|installation_id" Expected: shows the real numeric IDs (no <...> placeholders remain). Run: yamllint argocd/cluster-app/templates/fovea.yaml Expected: PASS.

Step 4: Commit, push, PR

Run:

jj --no-pager describe -m "feat(fovea): set GitHub App ID + installation ID [hl-uh07]"
jj --no-pager bookmark create feat/fovea-app-ids -r @
jj --no-pager git push -b feat/fovea-app-ids
gh pr create --head feat/fovea-app-ids --base main \
  --title "feat(fovea): set GitHub App + installation IDs [hl-uh07]" \
  --body "Fills the placeholder App IDs in the fovea Application now that the GitHub App exists (Task 7). 🤖 Generated with [Claude Code](https://claude.com/claude-code)"

Expected: PR URL printed. Merge after checks.

Task 9: Verify deployment, then go live

Files: none (verification)

Step 1: Confirm the Application is synced and healthy

Run: kubectl -n argocd get application fovea -o jsonpath='{.status.sync.status} {.status.health.status}{"\n"}' Expected: Synced Healthy.

Step 2: Confirm the pod is running and healthy

Run: kubectl -n fovea get pods Expected: one fovea-* pod Running, 1/1 ready. (Liveness/readiness hit /healthz on :8081.)

Step 3: Confirm the ExternalSecrets synced

Run: kubectl -n fovea get externalsecret Expected: fovea-app, ghcr-pull-secret, fovea-otlp-headers all SecretSynced / Ready=True.

Step 4: Confirm OTLP export works (both gates)

Step 5: Shadow-mode funnel check

Open a PR with a reachable nil-deref on a repo the App is installed on. In shadow mode no comment posts. Confirm the funnel advances: Run: kubectl -n fovea exec deploy/fovea -- wget -qO- localhost:9090/metrics | rg 'fovea_findings_funnel' Expected: fovea_findings_funnel{stage="scouted"} / gated / verified counters > 0.

Step 6: Flip to live (separate small PR)

Once shadow verification passes, set config.flags.publish_enabled: true in argocd/cluster-app/templates/fovea.yaml via a new change off main@origin (same push/PR cadence as Task 8). After sync, re-trigger → expect exactly one inline comment (guarded variant → none). See fovea docs/RUNBOOK.md §4 for the full §15 acceptance walkthrough.

Done-when

fovea Application is Synced/Healthy; pod Running; all 3 ExternalSecrets synced.
No OTLP i/o timeout or 401 in fovea logs (telemetry reaches clickstack).
Shadow funnel metrics advance on a test PR with no comment posted.
publish_enabled: true produces exactly one inline comment on re-trigger.
Follow-ups remain tracked: hl-uljo (Istio mesh epic) with hl-z5fc (fovea mesh-pilot egress allowlist + mTLS).

fovea Argo CD deployment — implementation plan

File structure

Task 1: Create the fovea app-config ExternalSecrets (Surface A)

Task 2: Whitelist the fovea namespace for OTLP in clickstack (Surface E)

Task 3: Create the fovea ArgoCD Application (Surface B)

Task 4: Add fovea to the Cloudflare webhook services (Surface C)

Task 5: Open PR1 (repo wiring) and merge

Task 6: Apply Cloudflare Terraform (manual)

Task 7: Create the GitHub App + populate Vault (manual)

Task 8: Fill App IDs and open PR2

Task 9: Verify deployment, then go live

Done-when

Task 1: Create the `fovea` app-config ExternalSecrets (Surface A)

Task 2: Whitelist the `fovea` namespace for OTLP in clickstack (Surface E)