Skip to content
fzymgc-house Cluster Docs

Enable fovea prompt/response audit (Phase 1: netpol-protected, keyless)

Design bead: hl-uh07.10 Related: hl-uh07 (fovea deploy epic), hl-uh07.12 (0.5.0→0.5.2 upgrade), hl-uh07.13 (Phase 2: Qdrant API-key auth), hl-z5fc (fovea mesh onboarding) External: seanb4t/fovea#69 (chart: plumb audit.qdrant.api_key) Status: Design Date: 2026-06-28

Summary

Section titled “Summary”

fovea 0.5.1 (fovea-aaf, #63) ships an opt-in prompt/response audit facility. When enabled, each fovea LLM call (llm_call) and guard decision (guard_decision) is captured to an embedded JetStream store (PVC) and a bounded embedding of the response is upserted into Qdrant for semantic retrieval. The facility is gated by config.audit.enabled; config.Validate() requires, when enabled, a reachable Qdrant endpoint, an embedding model, bus.impl=embedded-nats, and gateway.openai.base_url.

This spec enables the facility against a dedicated in-namespace Qdrant (qdrant-fovea), with embeddings routed through a new agentgateway lane (llm-fovea-embed → openrouter-embed-qwen3, qwen3-embedding-8b, 4096-dim, ZDR). The primary drivers are LLM observability (cost/quality/latency of scout+deepdive+guard calls) and, secondarily, injection forensics (reconstruct what an attacker-controlled PR diff did to the model) — the latter aligns with the 0.5.2 scout prompt-injection-hole fix (fovea-4cq/#67).

Phase 1 (this spec) ships keyless with a scoped ingress NetworkPolicy as the access control — the first Qdrant NetworkPolicy in the cluster. Phase 2 (deferred to hl-uh07.13, blocked on seanb4t/fovea#69) adds Qdrant API-key auth as defense-in-depth once the fovea chart can plumb audit.qdrant.api_key.

Goals

Section titled “Goals”
  • Enable fovea’s audit facility with zero behavioural change to PR reviews.
  • Stand up a dedicated qdrant-fovea (no cross-service coupling; matches the cluster’s one-qdrant-per-consumer convention: qdrant-octopus, engram’s qdrant).
  • Route audit embeddings through agentgateway on a ZDR lane (no new provider).
  • Protect qdrant-fovea from lateral movement with a scoped NetworkPolicy.
  • No secrets in git; no fovea-chart change required for Phase 1.

Non-goals

Section titled “Non-goals”
  • Qdrant API-key auth (Phase 2, hl-uh07.13, blocked on fovea chart change).
  • Istio mTLS / egress allowlist for the fovea namespace (hl-z5fc).
  • Native Qdrant snapshot CronJob (audit is 30d observability data; velero PVC backup suffices, unlike agent-memory’s durable memory store).
  • A query/UI surface for audit data — operators query Qdrant/JetStream directly.

Background

Section titled “Background”

fovea’s own audit design (seanb4t/fovea docs/superpowers/specs/2026-06-25-fovea-prompt-response-audit-design.md) established:

  • The embedder reuses the existing gatewayEmbedder (OpenAI-compatible /v1/embeddings, no new secret) — it calls {gateway.openai.base_url}/embeddings with the openai key. The vector is a bounded embedding of the response (diffs routinely exceed ~8k embedder tokens, so the full prompt is not embedded). The raw prompt+diff lives in the JetStream PVC.
  • Qdrant stores llm_call points (real vector) + guard_decision points (metadata-only, placeholder vector). Two query modes: payload filter (no vector) and semantic vector search.
  • Retention = audit.retention enforced as JetStream MaxAge + Qdrant prune age.

agentgateway routing is path-prefix-per-consumer on llm-gw.fzymgc.house: /v1/embeddings→openrouter-embed-bge (bge-m3 1024, octopus), /engram-embed/v1/embeddings→openrouter-embed-qwen3 (qwen3-embedding-8b 4096-dim ZDR, engram). fovea chat lives on /fovea-scout/v1/...→openrouter-zdr.

The fovea ns has no NetworkPolicy today (no default-deny egress); the only netpol referencing fovea is clickstack’s ingress-allow for OTLP. No Qdrant in the cluster has a NetworkPolicy (octopus + engram qdrants are unprotected).

Design

Section titled “Design”

Architecture

Section titled “Architecture”
PR review ── fovea pod (fovea ns) ──┬─ chat/guard: /fovea-scout/v1/... → openrouter-zdr       (existing)
                                   ├─ OTLP: cs-otel-collector:4317                            (existing)
                                   ├─ embed: /fovea-scout/v1/embeddings → llm-fovea-embed    (NEW route)
                                   │         → openrouter-embed-qwen3 (qwen3-emb-8b, 4096-dim, ZDR)
                                   │         (reuses gateway.openai.base_url; no fovea base_url change)
                                   ├─ raw log: PVC fovea-audit-js (5Gi, chart audit-pvc.yaml)  (NEW, chart renders)
                                   │         → JetStream @ /var/lib/fovea/audit-js
                                   └─ vectors: qdrant-fovea.fovea.svc:6334 (gRPC, keyless)     (NEW deployment)
                                             → collection fovea-audit (4096-dim)
                                             [CiliumNetworkPolicy: :6334 fovea pod only, :6333 fromEntities host]

Components

Section titled “Components”

  1. argocd/app-configs/fovea/qdrant.yaml (Source B, NEW) — copy of octopus/qdrant.yaml: PVC qdrant-fovea-data (20Gi, longhorn-encrypted), Deployment qdrant-fovea (qdrant/qdrant:v1.17.0, Recreate, fsGroup: 1000, requests ~100m/512Mi, limits 2Gi), Service qdrant-fovea (http:6333, grpc:6334). No auth (no QDRANT__SERVICE__API_KEY in Phase 1). Added to argocd/app-configs/fovea/kustomization.yaml.

  2. argocd/app-configs/fovea/qdrant-netpol.yaml (Source B, NEW) — CiliumNetworkPolicy (apiVersion: cilium.io/v2, matching the cluster convention — every netpol here except renovate’s is a CNP, e.g. clickstack/networkpolicy.yaml, dolt/network-policies.yaml) selecting qdrant-fovea (endpointSelector: app.kubernetes.io/name: qdrant-fovea), no egress policy (does not impose a namespace default-deny, so fovea’s existing egress to llm-gw and cs-otel-collector is untouched). Ingress rules split by port role:

    • 6334 (gRPC, the fovea data path): fromEndpoints pods k8s:io.kubernetes.pod.namespace: fovea, k8s:app.kubernetes.io/name: fovea only.
    • 6333 (HTTP, readiness probe): fromEntities: [host] so the local node’s kubelet httpGet /readyz probe is not blocked. fovea’s qdrant client uses gRPC :6334, so it never needs :6333. Note: fromEntities: [host] admits the local node entity; whether kubectl port-forward reaches :6333 is Cilium-version-dependent (its final hop is kubelet-sourced, which Cilium may classify as host and thus allow) — verify against the cluster’s Cilium build. This is non-load-bearing: :6333 is HTTP readiness only and exposes no audit vectors on a keyless Qdrant; the audit data path is :6334, locked to app=fovea pods.

    Kubelet-probe gotcha: an ingress rule allowing only app=fovea would drop the kubelet’s readiness probe (sourced from the node, not a pod) → qdrant-fovea never Ready. Cilium’s fromEntities: host is the idiomatic fix (same primitive the dolt-ingress-sql precedent uses for node-sourced traffic — that rule is broader, [host, remote-node, world], because dolt’s LoadBalancer SNATs external SQL; qdrant-fovea is ClusterIP so [host] alone suffices). This is the first Qdrant NetworkPolicy in the cluster.

  3. argocd/app-configs/agentgateway/llm-routes.yaml (NEW HTTPRoute llm-fovea-embed) — matches PathPrefix /fovea-scout/v1/embeddings → openrouter-embed-qwen3 backend (longest-prefix match, mirrors llm-embeddings-engram). timeouts: { request: 60s, backendRequest: 45s }. No new backend; reuses the existing engram ZDR embed backend.

    argocd/app-configs/agentgateway/llm-policies.yaml (MODIFY) — add llm-fovea-embed to the targetRefs of both policies that gate llm-embeddings-engram (verified: llm-policies.yaml lines 9-22 and 137-144), so the new route is not ungated/non-retried:

    • llm-apikey (mode: Strict virtual-key gate) — every fovea chat route (llm-fovea-scout, llm-fovea-deepdive) and every embed route is listed here; llm-fovea-embed joins them.
    • llm-embeddings-retry (attempts: 4, transient OpenRouter 503 retry) — parity with llm-embeddings-engram; an embed route to OpenRouter needs the same retry ride-through the other embed routes get.

  4. argocd/cluster-app/templates/fovea.yaml valuesObject.config.audit (NEW) — added under config::

    config:
      audit:
        enabled: true
        retention: 720h            # ~30d; JetStream MaxAge + Qdrant prune
        store_dir: /var/lib/fovea/audit-js
        embedding_model: "qwen3-embedding-8b"   # openrouter-embed-qwen3 pins qwen/qwen3-embedding-8b; backend overrides the model so this value is for validation/dim — plan confirms whether fovea wants the OpenRouter id (qwen/qwen3-embedding-8b) or a logical name
        qdrant:
          endpoint: "qdrant-fovea.fovea.svc.cluster.local:6334"
          use_tls: false
          collection: "fovea-audit"
          vector_size: 4096
          # api_key OMITTED in Phase 1 (keyless; chart can't plumb it yet — fovea#69)
        scrub:
          max_embed_chars: 24000

    With audit.enabled: true, the fovea chart’s audit-pvc.yaml renders the fovea-audit-js PVC (5Gi) and the Deployment volumeMount automatically — no Source B action for the JetStream PVC. bus.impl defaults to embedded-nats (chart default) — no change.

Data flow

Section titled “Data flow”

Each fovea LLM call and guard decision → captured to JetStream (audit.> subjects, PVC fovea-audit-js) → a durable consumer embeds the bounded response via llm-fovea-embed → upserts llm_call points (real 4096-dim vector) + guard_decision points (metadata-only) into Qdrant fovea-audit. Transient Qdrant/embedder errors → durable consumer redelivers; idempotent upsert; poison messages handled (per fovea’s audit design). Queries: payload filter for time/model/PR slicing; semantic vector search for “similar prompts/injection attempts.”

Access control (Phase 1)

Section titled “Access control (Phase 1)”

qdrant-fovea runs keyless. The scoped CiliumNetworkPolicy is the sole access control: only the fovea pod (same namespace) can reach :6334 (the gRPC data path); :6333 (HTTP) is limited to fromEntities: host for the kubelet readiness probe only. This closes the lateral-movement hole (a compromised pod in another namespace cannot read/write the audit store despite the Service being cluster-DNS). Defense-in-depth via Qdrant API-key auth is Phase 2 (hl-uh07.13).

Failure mode (audit path down)

Section titled “Failure mode (audit path down)”

Audit capture is decoupled from review publishing: fovea emits audit.> records to the in-process embedded NATS JetStream (bus.impl=embedded-nats, co-located with fovea — not a network dependency), and a durable consumer asynchronously embeds + upserts to Qdrant. A Qdrant or embedder-lane outage does not block PR reviews — reviews still publish (publish_enabled: true is unaffected); records redeliver on recovery (idempotent upsert, poison-message handling per fovea’s audit design). If the in-process JetStream itself is exhausted, fovea’s bus errors surface via existing OTLP telemetry. The exact emit/blocking semantics (sync vs. async to the review path) are confirmed against fovea source in the plan.

PII / secrets / backup

Section titled “PII / secrets / backup”
  • Qdrant vectors carry bounded response embeddings + payload metadata — no raw diff (diffs exceed the embedder token limit, deliberately not embedded).
  • Raw prompt+diff lives in the fovea-audit-js JetStream PVC — sensitive (PR diffs may contain leaked secrets), retained 30d. Access is namespace-scoped with no external exposure.
  • Backup: the fovea namespace is stateful and therefore velero-backed by default (exclude-only strategy). The fovea-audit-js and qdrant-fovea-data PVCs — which hold raw diffs and audit vectors — land in velero backups. Decision: accept (encrypted backups; forensics-retention value outweighs the exposure, which is already namespace-internal). Documented as a sensitive- data surface. Revisit if the cluster’s backup-target posture changes.

Alternatives considered

Section titled “Alternatives considered”
  • Reuse qdrant-octopus (new collection) instead of a dedicated Qdrant. Rejected: couples fovea audit to octopus’s Qdrant lifecycle; an octopus decommission would break fovea audit. Dedicated matches the one-qdrant-per- consumer convention; cost is ~40 lines + 20Gi.
  • bge-m3 (1024-dim) or a new 1536-dim lane instead of qwen3 (4096). bge-m3 was the recommended default (reuse existing openrouter-embed-bge, cheapest, proven for code text). Operator chose qwen3-embedding-8b (4096-dim, ZDR) for higher-quality embeddings; reuses the existing engram ZDR backend, so still no new backend. The 1536 default in fovea config matches no available lane and was rejected.
  • Block Phase 1 on the chart change (both controls day-one). Rejected: couples audit enablement + forensics value to fovea’s release cadence. Netpol already closes the lateral-movement hole; Phase 2 follows the chart change.
  • Native Qdrant snapshot CronJob (like agent-memory). Rejected: audit is 30d observability data, not durable memory; velero PVC backup suffices.

Phase 2 (deferred — hl-uh07.13, blocked on seanb4t/fovea#69)

Section titled “Phase 2 (deferred — hl-uh07.13, blocked on seanb4t/fovea#69)”

Once the fovea chart ships an auditQdrantKey secret ref → FOVEA_AUDIT__QDRANT__API_KEY env (feature request: seanb4t/fovea#69):

  1. Generate a Qdrant API key; store in Vault at fzymgc-house/cluster/fovea/qdrant-api-key.
  2. ExternalSecret fovea-qdrant-key in the fovea namespace.
  3. Set QDRANT__SERVICE__API_KEY env on the qdrant-fovea Deployment from that secret (enables Qdrant auth — all requests require the api-key header, or Authorization: Bearer; grounded via context7 /qdrant/qdrant).
  4. Set fovea config.audit.qdrant.api_key via the new auditQdrantKey ref so fovea’s qdrant client sends the api-key header.
  5. Bump chart targetRevision to the release with the field; verify no 401s.

The Phase 1 NetworkPolicy stays unchanged.

Testing / verification

Section titled “Testing / verification”
  • Pre-merge: helm pull chart 0.5.2 and helm template with the new valuesObject (audit.enabled: true) → confirm fovea-audit-js PVC + audit block in the ConfigMap render; yamllint clean; kubectl kustomize argocd/app-configs/fovea renders qdrant-fovea + netpol.
  • Post-merge: qdrant-fovea 1/1 Running; fovea pod 1/1 Running 0 restarts (config.Validate() passed — endpoint + embedding_model + vector_size + base_url + bus.impl all satisfied); collection fovea-audit (4096-dim) created on first review; a payload-filter query against qdrant-fovea returns records after a live PR review; ClickStack shows fovea audit activity.
  • Netpol: a pod in another namespace cannot reach qdrant-fovea.fovea.svc:6334 (connection refused/timeout).

Rollout

Section titled “Rollout”

ArgoCD auto-sync (selfHeal) applies the multi-source fovea Application: Source A (chart) re-renders with the new config.audit → renders fovea-audit-js PVC + audit ConfigMap + Deployment update; Source B adds qdrant-fovea + netpol; llm-fovea-embed route syncs via the agentgateway app. No manual kubectl.

Rollback = revert the commit. The fovea Application carries resources-finalizer.argocd.argoproj.io, so a revert that prunes qdrant-fovea and disables audit cascade-deletes the qdrant-fovea-data and fovea-audit-js PVCs. Whether the underlying Longhorn volumes are then deleted depends on the longhorn-encrypted StorageClass’s reclaimPolicy, which is not defined in this repo (only longhorn-retain is, with reclaimPolicy: Retain; longhorn-encrypted is created out-of-band via Longhorn UI/helm). The plan must confirm longhorn-encrypted’s reclaimPolicy before relying on this:

  • Delete → volumes are destroyed on prune (destructive revert).
  • Retain → volumes persist as orphaned after PVC prune and need manual cleanup (with a rebind nuance if audit is re-enabled).

Either way the audit data (raw diffs + vectors) is at risk on rollback; recovery relies on the velero backup cited in PII/backup. This is not a no-cost toggle — operators should drain/extract audit data first if forensics retention matters.