fovea Audit Enablement (Phase 1) Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Enable fovea 0.5.1+‘s prompt/response audit facility against a dedicated in-namespace Qdrant, with embeddings routed through a new agentgateway ZDR lane, protected by a scoped CiliumNetworkPolicy — keyless in Phase 1.

Architecture: Stand up qdrant-fovea (copy of octopus/qdrant.yaml) in the fovea namespace; add a CiliumNetworkPolicy locking ingress to the fovea pod on :6334 and the node kubelet on :6333; add an llm-fovea-embed HTTPRoute (→ existing openrouter-embed-qwen3, qwen3-embedding-8b 4096-dim ZDR) plus its policy gating; set config.audit in the fovea Application’s valuesObject. ArgoCD auto-sync applies everything; no manual kubectl.

Tech Stack: ArgoCD GitOps (multi-source Application), Helm OCI chart (ghcr.io/seanb4t/charts/fovea 0.5.2), CiliumNetworkPolicy (cilium.io/v2), agentgateway (Gateway API HTTPRoute + AgentgatewayPolicy), Qdrant (qdrant/qdrant:v1.17.0), Longhorn-encrypted PVC.

Spec: docs/engineering/specs/2026-06-28-fovea-audit-enablement-design.md (design-reviewer READY, round 3). Design bead: hl-uh07.10. Phase 2 (API-key auth): hl-uh07.13, blocked on seanb4t/fovea#69.

VCS: This is a colocated jj+git repo. Use jj for all VCS operations (git mutations are blocked by a guard hook). Pass --no-pager. Commit per task with jj commit -m "<conventional msg>"; the working copy is a commit (@), so jj commit captures the current change and starts a fresh @. Push the branch with jj git push -b feat/fovea-audit-spec. One PR at the end.

Testing note: This plan is manifest-only (YAML). There are no unit tests — the “test” for each task is a render/lint gate (helm template, kubectl kustomize, yamllint, rumdl) plus post-deploy verification (Task 5). This is the documented TDD exception for config/manifest work.

File Structure

File	Action	Responsibility
`argocd/app-configs/fovea/qdrant.yaml`	Create	`qdrant-fovea` PVC + Deployment + Service (keyless)
`argocd/app-configs/fovea/qdrant-netpol.yaml`	Create	CiliumNetworkPolicy: `:6334` fovea-pod-only, `:6333` `fromEntities: host`
`argocd/app-configs/fovea/kustomization.yaml`	Modify	Add the two new resources
`argocd/app-configs/agentgateway/llm-routes.yaml`	Modify	Add `llm-fovea-embed` HTTPRoute
`argocd/app-configs/agentgateway/llm-policies.yaml`	Modify	Add `llm-fovea-embed` to `llm-apikey` + `llm-embeddings-retry` targetRefs
`argocd/cluster-app/templates/fovea.yaml`	Modify	Add `config.audit` block to `valuesObject`

All new files are in directories that already exist (argocd/app-configs/fovea/, argocd/app-configs/agentgateway/). The fovea namespace is created by the fovea Application’s CreateNamespace=true syncOption.

Task 1: Stand up `qdrant-fovea` (PVC + Deployment + Service)

Files:

Create: argocd/app-configs/fovea/qdrant.yaml
Modify: argocd/app-configs/fovea/kustomization.yaml
Step 1: Create argocd/app-configs/fovea/qdrant.yaml

This is a copy of argocd/app-configs/octopus/qdrant.yaml with namespaced names changed to qdrant-fovea and namespace: fovea. The image qdrant/qdrant:v1.17.0 matches the octopus precedent (consistency); verify latest in Step 3.

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: qdrant-fovea-data
  namespace: fovea
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: longhorn-encrypted
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: qdrant-fovea
  namespace: fovea
  labels:
    app.kubernetes.io/name: qdrant-fovea
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: qdrant-fovea
  template:
    metadata:
      labels:
        app.kubernetes.io/name: qdrant-fovea
    spec:
      securityContext:
        fsGroup: 1000
      containers:
        - name: qdrant
          image: qdrant/qdrant:v1.17.0
          ports:
            - containerPort: 6333
              name: http
            - containerPort: 6334
              name: grpc
          readinessProbe:
            httpGet:
              path: /readyz
              port: 6333
            initialDelaySeconds: 5
            periodSeconds: 10
          volumeMounts:
            - name: data
              mountPath: /qdrant/storage
          resources:
            requests:
              cpu: 100m
              memory: 512Mi
            limits:
              memory: 2Gi
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: qdrant-fovea-data
---
apiVersion: v1
kind: Service
metadata:
  name: qdrant-fovea
  namespace: fovea
spec:
  selector:
    app.kubernetes.io/name: qdrant-fovea
  ports:
    - name: http
      port: 6333
      targetPort: 6333
    - name: grpc
      port: 6334
      targetPort: 6334

Step 2: Add it to the fovea kustomization

In argocd/app-configs/fovea/kustomization.yaml, append to the resources: list (after otlp-headers-secret.yaml):

  - qdrant.yaml             # fovea audit vector store (fovea-aaf, hl-uh07.10); keyless Phase 1, netpol-protected

Step 3: Verify Qdrant image is current enough (argocd CLAUDE.md version rule)

Run: gh api repos/qdrant/qdrant/releases/latest --jq .tag_name Expected: a tag >= v1.17.0. If a newer stable is available, prefer matching octopus/qdrant.yaml’s pinned v1.17.0 for consistency unless the newer release carries a security fix — note the decision in the commit message if you bump.

Step 4: Render + lint

Run: kubectl kustomize argocd/app-configs/fovea | grep -E 'kind: (PersistentVolumeClaim|Deployment|Service)' Expected: all three kinds present, each with qdrant-fovea/qdrant-fovea-data names and namespace: fovea.

Run: yamllint -c .yamllint.yaml argocd/app-configs/fovea/qdrant.yaml Expected: clean (no errors).

Step 5: Commit

jj commit -m "feat(fovea): add qdrant-fovea vector store for audit [hl-uh07.10]"

Task 2: CiliumNetworkPolicy protecting `qdrant-fovea`

Files:

Create: argocd/app-configs/fovea/qdrant-netpol.yaml
Modify: argocd/app-configs/fovea/kustomization.yaml
Step 1: Create argocd/app-configs/fovea/qdrant-netpol.yaml

kind: CiliumNetworkPolicy matches the cluster convention (dolt/network-policies.yaml, clickstack/networkpolicy.yaml are both cilium.io/v2; only renovate uses native NetworkPolicy). fromEndpoints selects the fovea pod by namespace+label; fromEntities: host admits the local node’s kubelet for the readiness probe. The k8s:app.kubernetes.io/name: fovea label is verified: chart 0.5.2 templates/deployment.yaml:18 sets it on the pod template, and the live fovea pod (fovea-59d5799bb5-*) carries it — so the :6334 ingress rule matches fovea’s audit writes (no silent drop).

---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: qdrant-fovea-ingress
  namespace: fovea
  labels:
    app.kubernetes.io/component: network-policy
    app.kubernetes.io/part-of: fovea
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: qdrant-fovea
  ingress:
    # gRPC data path (:6334) — fovea audit client only.
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: fovea
            k8s:app.kubernetes.io/name: fovea
      toPorts:
        - ports:
            - port: "6334"
              protocol: TCP
    # HTTP (:6333) — kubelet readiness probe only (host entity).
    # fovea's qdrant client uses gRPC :6334 and never touches :6333.
    - fromEntities:
        - host
      toPorts:
        - ports:
            - port: "6333"
              protocol: TCP

Step 2: Add it to the fovea kustomization

In argocd/app-configs/fovea/kustomization.yaml, append after the qdrant.yaml line added in Task 1:

  - qdrant-netpol.yaml      # CiliumNetworkPolicy: :6334 fovea pod only, :6333 kubelet probe (fovea-fovea audit, Phase 1 keyless)

Step 3: Render + lint

Run: kubectl kustomize argocd/app-configs/fovea | grep -A2 'kind: CiliumNetworkPolicy' Expected: the CNP renders with endpointSelector app.kubernetes.io/name: qdrant-fovea and both ingress rules.

Run: yamllint -c .yamllint.yaml argocd/app-configs/fovea/qdrant-netpol.yaml Expected: clean.

Step 4: Commit

jj commit -m "feat(fovea): CiliumNetworkPolicy for qdrant-fovea ingress [hl-uh07.10]"

Task 3: `llm-fovea-embed` route + policy gating

Files:

Modify: argocd/app-configs/agentgateway/llm-routes.yaml
Modify: argocd/app-configs/agentgateway/llm-policies.yaml
Step 1: Add the llm-fovea-embed HTTPRoute

In argocd/app-configs/agentgateway/llm-routes.yaml, append a new HTTPRoute mirroring llm-embeddings-engram (lines 46-60 of that file). The path /fovea-scout/v1/embeddings is longest-prefix-distinct from the existing llm-fovea-scout route (/fovea-scout/v1/chat/completions) — no collision. Backend reuses the existing openrouter-embed-qwen3 (qwen3-embedding-8b, 4096-dim, ZDR) — no new backend.

---
# fovea audit embedder → openrouter-embed-qwen3 (qwen3-embedding-8b, 4096-dim, ZDR).
# fovea's audit embedder reuses gateway.openai.base_url (fovea-scout lane), so it
# calls /fovea-scout/v1/embeddings; this route longest-prefix-matches it away from
# the fovea-scout chat route. Mirrors llm-embeddings-engram. No new backend. [hl-uh07.10]
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm-fovea-embed
  namespace: agentgateway
spec:
  parentRefs:
    - { group: gateway.networking.k8s.io, kind: Gateway, name: agentgateway, sectionName: llm-gw }
  hostnames: ["llm-gw.fzymgc.house"]
  rules:
    - matches:
        - { path: { type: PathPrefix, value: /fovea-scout/v1/embeddings } }
      timeouts: { request: 60s, backendRequest: 45s }
      backendRefs:
        - { group: agentgateway.dev, kind: AgentgatewayBackend, name: openrouter-embed-qwen3, weight: 1 }

Step 2: Add llm-fovea-embed to llm-apikey targetRefs

In argocd/app-configs/agentgateway/llm-policies.yaml, in the llm-apikey policy’s targetRefs: list (after the llm-fovea-deepdive line, ~line 17), add:

    - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: llm-fovea-embed }

Step 3: Add llm-fovea-embed to llm-embeddings-retry targetRefs

In the llm-embeddings-retry policy’s targetRefs: list (after the llm-embeddings-engram-v2 line, ~line 141), add:

    # fovea audit embed lane — same OpenRouter 503 retry parity as the other embed routes. [hl-uh07.10]
    - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: llm-fovea-embed }

Step 4: Render + lint

Run: kubectl kustomize argocd/app-configs/agentgateway | grep 'name: llm-fovea-embed' Expected: 3 matches (the HTTPRoute metadata.name + 2 targetRef entries).

Run: yamllint -c .yamllint.yaml argocd/app-configs/agentgateway/llm-routes.yaml argocd/app-configs/agentgateway/llm-policies.yaml Expected: clean.

Step 5: Commit

jj commit -m "feat(agentgateway): llm-fovea-embed route for fovea audit embeddings [hl-uh07.10]"

Task 4: Enable `config.audit` in the fovea Application

Files:

Modify: argocd/cluster-app/templates/fovea.yaml (insert audit: block under config:, after the guard: block, before privateKeySecret:)
Step 1: Add the config.audit block

In argocd/cluster-app/templates/fovea.yaml, inside spec.sources[0].helm.valuesObject.config: (the config: key is at 10 spaces indent; its children at 12), insert the following block immediately after the guard: block’s closing line (threshold: 0.85) and before the privateKeySecret: line (which is at 10 spaces — a sibling of config:). Indentation: audit: at 12 spaces, fields at 14, qdrant/scrub at 16, their fields at 18.

            # 0.5.1 audit facility (fovea-aaf, #63). Phase 1: keyless —
            # audit.qdrant.api_key omitted (chart can't plumb it yet; Phase 2
            # hl-uh07.13 blocked on seanb4t/fovea#69). Qdrant is netpol-protected
            # (qdrant-netpol.yaml). Embedder reuses gateway.openai.base_url
            # (fovea-scout lane) → /fovea-scout/v1/embeddings → llm-fovea-embed
            # route → openrouter-embed-qwen3 (qwen3-embedding-8b, 4096-dim, ZDR).
            # vector_size MUST match the embedder output dim (4096). Raw
            # prompt+diff lives in the JetStream PVC (fovea-audit-js, chart-rendered);
            # Qdrant stores bounded response embeddings + metadata only. [hl-uh07.10]
            audit:
              enabled: true
              retention: 720h            # ~30d; JetStream MaxAge + Qdrant prune
              store_dir: /var/lib/fovea/audit-js
              embedding_model: "qwen3-embedding-8b"   # openrouter-embed-qwen3 pins qwen/qwen3-embedding-8b; backend overrides the model so this value is for validation/dim — confirmed vs llm-policies.yaml:164
              qdrant:
                endpoint: "qdrant-fovea.fovea.svc.cluster.local:6334"
                use_tls: false
                collection: "fovea-audit"
                vector_size: 4096
                # api_key OMITTED in Phase 1 (keyless; chart gap — fovea#69 / hl-uh07.13)
              scrub:
                max_embed_chars: 24000

Step 2: Extract the valuesObject and render the chart with it

The fovea OCI chart is private — authenticate to GHCR first (one-time per shell), pull the chart into a tmp dir, extract our valuesObject, then render:

gh auth token | helm registry login ghcr.io -u seanb4t --password-stdin
cd /tmp && rm -rf fovea-audit-render && mkdir fovea-audit-render && cd fovea-audit-render
helm pull oci://ghcr.io/seanb4t/charts/fovea --version 0.5.2
yq eval '.spec.sources[0].helm.valuesObject' "$OLDPWD/argocd/cluster-app/templates/fovea.yaml" > fovea-audit-values.yaml
helm template fovea ./fovea-0.5.2.tgz -f fovea-audit-values.yaml > rendered.yaml

($OLDPWD is the repo root you came from; if your shell lacks it, substitute the absolute repo path.)

Step 3: Verify the render gates

Run: grep -c '^kind:' /tmp/fovea-audit-render/rendered.yaml Expected: 4 (ConfigMap, Service, Deployment, and the fovea-audit-js PVC — the chart’s audit-pvc.yaml renders only when config.audit.enabled: true).

Run: grep 'fovea-audit-js' /tmp/fovea-audit-render/rendered.yaml Expected: the PVC fovea-audit-js (5Gi) AND a volumeMount / volume entry for audit-js in the Deployment.

Run: grep -A2 'audit:' /tmp/fovea-audit-render/rendered.yaml | head Expected: the audit: block present in the fovea-config ConfigMap with enabled: true, vector_size: 4096, endpoint: qdrant-fovea.fovea.svc.cluster.local:6334.

Run: grep 'service.version=0.5.2' /tmp/fovea-audit-render/rendered.yaml Expected: a match (the existing OTEL resource attribute — confirms the render used the merged values, not a stale 0.5.0).

Step 4: Lint

Run: yamllint -c .yamllint.yaml argocd/cluster-app/templates/fovea.yaml Expected: clean.

Step 5: Commit

jj commit -m "feat(fovea): enable prompt/response audit (Phase 1, keyless) [hl-uh07.10]"

Task 5: Open the PR

Step 1: Push the branch

jj git push -b feat/fovea-audit-spec

Step 2: Open the PR

gh pr create --head feat/fovea-audit-spec --base main \
  --title "feat(fovea): enable prompt/response audit (Phase 1, keyless) [hl-uh07.10]" \
  --body "$(cat <<'EOF'
## Summary

Enable fovea 0.5.1+'s prompt/response audit facility (fovea-aaf, #63) against a dedicated in-namespace Qdrant. Phase 1 ships **keyless** with a scoped CiliumNetworkPolicy as the access control. Spec: \`docs/engineering/specs/2026-06-28-fovea-audit-enablement-design.md\` (design-reviewer READY, round 3).

## Changes

- **\`qdrant-fovea\`** — PVC + Deployment + Service (copy of \`octopus/qdrant.yaml\`, \`qdrant/qdrant:v1.17.0\`, 20Gi \`longhorn-encrypted\`, keyless). \`argocd/app-configs/fovea/qdrant.yaml\`.
- **CiliumNetworkPolicy** — \`:6334\` (gRPC data path) from the \`fovea\` pod only; \`:6333\` (HTTP readiness probe) from \`fromEntities: host\` (kubelet). First Qdrant netpol in the cluster. \`argocd/app-configs/fovea/qdrant-netpol.yaml\`.
- **\`llm-fovea-embed\` HTTPRoute** — \`/fovea-scout/v1/embeddings\` → existing \`openrouter-embed-qwen3\` (qwen3-embedding-8b, 4096-dim, ZDR). No new backend. Added to \`llm-apikey\` + \`llm-embeddings-retry\` \`targetRefs\` (mirrors \`llm-embeddings-engram\`). \`argocd/app-configs/agentgateway/llm-routes.yaml\` + \`llm-policies.yaml\`.
- **\`config.audit\`** — \`enabled: true\`, \`vector_size: 4096\`, \`endpoint: qdrant-fovea.fovea.svc:6334\`, \`embedding_model: qwen3-embedding-8b\`, \`retention: 720h\`, \`api_key\` omitted (keyless). Chart renders the \`fovea-audit-js\` PVC + volumeMount on \`audit.enabled\`. \`argocd/cluster-app/templates/fovea.yaml\`.

## Why keyless (Phase 1)

The fovea chart (0.5.2) has no \`auditQdrantKey\` secret-ref field and no \`extraEnv\` — it can't plumb \`audit.qdrant.api_key\`. Enabling Qdrant auth without the key would 401 every audit write. Netpol closes the lateral-movement hole now; API-key defense-in-depth is Phase 2.

## Verification

Pre-merge: \`helm template\` chart 0.5.2 with the new \`valuesObject\` → 4 resources incl. \`fovea-audit-js\` PVC; \`yamllint\`/\`rumdl\` clean; \`kubectl kustomize\` renders \`qdrant-fovea\` + CNP + route. See plan Task 4.

Post-merge (plan Task 6): \`qdrant-fovea\` 1/1; fovea pod 0 restarts (\`config.Validate\` passed); both PVCs Bound; netpol isolates other namespaces; audit records land on live PR reviews.

## Follow-ups

- **hl-uh07.13** — Phase 2: Qdrant API-key auth (blocked on **seanb4t/fovea#69**, the chart feature request).
- **hl-z5fc** — fovea mesh onboarding (Istio mTLS + egress allowlist), separate.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

Task 6: Post-merge verification (after ArgoCD auto-syncs)

This task runs after the PR merges and ArgoCD syncs. Use the Kubernetes MCP (readonly) for live state and ClickStack/ClickHouse MCP for telemetry. No kubectl applies — ArgoCD manages deployments.

Step 1: qdrant-fovea is healthy

Get the Deployment (apps/v1 Deployment qdrant-fovea in fovea ns) via Kubernetes MCP. Expected: replicas: 1, readyReplicas: 1, availableReplicas: 1, condition Available=True. Pod 1/1 Running, 0 restarts. (If 0/1 with the readiness probe never passing, the CiliumNetworkPolicy :6333 rule is blocking the kubelet probe — re-check fromEntities: host.)

Step 2: fovea pod boots clean (config.Validate() passed)

Get the fovea Deployment / pods in fovea ns. Expected: image ghcr.io/seanb4t/fovea:0.5.2, pod 1/1 Running, 0 restarts. Zero restarts is the key signal — config.Validate() runs at boot and would CrashLoopBackOff if audit.qdrant.endpoint/embedding_model/vector_size/base_url/bus.impl were wrong.

Step 3: fovea-audit-js PVC + qdrant-fovea-data PVC are Bound

List PVCs in fovea ns (Kubernetes MCP resources_list or pods_list-equivalent). Expected: both PVCs Bound.

Step 4: llm-fovea-embed route + policies synced

Get the HTTPRoute llm-fovea-embed and the AgentgatewayPolicy objects in agentgateway ns. Expected: llm-fovea-embed present; llm-apikey and llm-embeddings-retry targetRefs include llm-fovea-embed.

Step 5: Netpol isolation (negative test)

kubectl exec into an existing pod in another namespace (e.g. a clickstack or argocd pod — do not create a debug pod; ArgoCD manages workloads) and attempt to reach qdrant-fovea.fovea.svc.cluster.local:6334. Expected: timeout/refused (Cilium drops it). The fovea pod itself can reach :6334 (audit writes succeed — confirmed by Step 6). If the only available pods lack a shell/curl, defer this to a manual check and note it.

Step 6: Audit records land (after a live PR review)

After fovea processes a real PR review (fovea is live, publish_enabled: true), query qdrant-fovea for the fovea-audit collection: expect llm_call points (4096-dim vectors) and guard_decision points (metadata-only). Via ClickStack: fovea audit activity (the durable consumer embedding/upserting) visible in fovea logs/traces. If no review has fired, this step waits — it’s the end-to-end signal, not a blocker for merge.

Step 7: Record outcome

bd note hl-uh07.10 "Phase 1 deployed: qdrant-fovea 1/1, fovea 0 restarts, PVCs Bound, netpol isolates, audit records landing on live reviews."

Leave hl-uh07.10 in_progress until Phase 1 is confirmed stable in prod; close after a soak period. Phase 2 (hl-uh07.13) remains open, blocked on seanb4t/fovea#69.

fovea Audit Enablement (Phase 1) Implementation Plan

File Structure

Task 1: Stand up qdrant-fovea (PVC + Deployment + Service)

Task 2: CiliumNetworkPolicy protecting qdrant-fovea

Task 3: llm-fovea-embed route + policy gating

Task 4: Enable config.audit in the fovea Application

Task 5: Open the PR

Task 6: Post-merge verification (after ArgoCD auto-syncs)

Task 1: Stand up `qdrant-fovea` (PVC + Deployment + Service)

Task 2: CiliumNetworkPolicy protecting `qdrant-fovea`

Task 3: `llm-fovea-embed` route + policy gating

Task 4: Enable `config.audit` in the fovea Application