Cilium PR5a: Crown-jewel CNPs for vault — Implementation Plan

For agentic workers: Use superpowers:subagent-driven-development or superpowers:executing-plans.

Goal: Crown-jewel CiliumNetworkPolicy coverage for the vault namespace. Highest-risk PR of the initiative: vault is the cluster’s identity-plane root, and any breakage cascades through ESO, cert-manager, vault-agent-injected workloads, and Sean’s bd/vault operator workflow.

Splitting PR5 → PR5a (vault) / PR5b (authentik) / PR5c (cnpg). Per the spec’s “least-coupled first” ordering, vault is technically the most coupled. But Sean chose to do vault first; doing it solo gives full attention to raft heartbeats (the past-incident class per memory cilium-pr3-complete).

Architecture: One new ArgoCD-managed directory argocd/app-configs/vault-network-policies/. Two distinct workload identities to police:

• vault (the server StatefulSet pods) — selector app.kubernetes.io/name: vault, component: server
• vault-agent-injector (the mutating admission webhook deployment) — selector app.kubernetes.io/name: vault-agent-injector, component: webhook

Lessons applied:

• Raft heartbeat (port 8201) intra-namespace allow — explicitly between vault server pods. Memory cilium-pr3-complete documents past raft breakage from missed intra-NP rules. This is the single most important rule in the PR.
• No baseline default-deny CNP — direction-specific CNPs engage default-deny implicitly.
• k3s admission-webhook entities — vault-agent-injector ingress uses kube-apiserver + host + remote-node (memory: PR4b #1045 fix).
• DNS L7 rule co-located with any toFQDNs — N/A for vault (no external egress needed; vault is the internal CA root).
• host source observed empirically hitting vault:8200 — allow ingress from host and remote-node entities on 8200. Likely vault-agent sidecars on hostNetwork pods, or Sean’s workstation via in-cluster kubectl-port-forward routing through a node.

Traffic Audit (empirical, captured 2026-05-12)

Direction	Peer	Port	Notes
Ingress	vault server pods (intra-namespace, raft)	8201	CRITICAL — past-incident class
Ingress	vault server pods (intra-namespace, API forwarding)	8200	Cross-pod API forwarding from standbys to leader
Ingress	traefik pods	8200	UI/API via Traefik IngressRoute (route lives elsewhere)
Ingress	cert-manager controller	8200	Vault PKI issuance (vault-internal headless or vault-active)
Ingress	external-secrets operator	8200	Every ExternalSecret refresh
Ingress	vault-agent-injector pod	8200	Injector talks to vault to issue tokens for sidecars
Ingress	prometheus pods	8200 (metrics path)	ServiceMonitor scrape via /v1/sys/metrics
Ingress	host + remote-node entities	8200	Mystery source 10.245.2.25 (host) hits vault:8200 frequently per Hubble — likely vault-agent sidecars in hostNetwork pods. Allow to avoid regression.
Egress	vault server pods (intra-namespace, raft)	8201	Bidirectional with ingress
Egress	vault server pods (intra-namespace, API forwarding)	8200	Bidirectional
Egress	kube-dns	53 + DNS L7	Service-name lookups
Egress	kube-apiserver entity	443	SA TokenReview during k8s auth method

vault-agent-injector:

Direction	Peer	Port
Ingress	kube-apiserver / host / remote-node	8080 (svc 443→8080) — admission webhook
Ingress	prometheus pods	8080 if metrics on same port (verify)
Egress	kube-dns	53
Egress	kube-apiserver	443
Egress	vault server pods	8200 — injector queries vault for auth/tokens

File Structure

Path	Action	Purpose
argocd/app-configs/vault-network-policies/network-policies.yaml	CREATE	All vault + injector CNPs
argocd/app-configs/vault-network-policies/kustomization.yaml	CREATE	Kustomize
argocd/cluster-app/templates/vault-network-policies.yaml	CREATE	ArgoCD Application with finalizer
docs/operations/cilium.md	MODIFY	Update “Currently enforced” list

Recovery recipe (paste this into the PR description)

If anything breaks post-merge:

# Nuclear: restore allow-all in vault namespace
kubectl delete cnp -n vault --all

# Surgical: identify what's dropped and fix
kubectl exec -n kube-system ds/cilium -c cilium-agent -- \
  hubble observe --namespace vault --verdict DROPPED --since 10m

Most likely failure modes:

1. Raft broken: vault pods enter “removed from cluster” state, vault status reports ha_connection_healthy: false. Fix: verify raft CNP allows port 8201 intra-namespace. Recovery: delete the raft CNP, debug, re-apply.
2. Agent-injector webhook fails: new pods with vault.hashicorp.com/agent-inject: true annotation can’t start. Fix: verify host/remote-node in the injector ingress CNP (#1045 lesson).
3. ESO refresh fails: kubectl get externalsecret -A shows SecretSyncedError. Fix: verify the ESO operator’s egress to vault is allowed (PR4b egress CNP) AND the vault ingress accepts external-secrets pods.
4. cert-manager renewal fails: kubectl get certificate -A shows Ready=False. Same as ESO — verify both sides.

Tasks

• Pre-flight: confirm vault healthy (vault status shows 1 active + 2 standby); confirm no existing CNPs in vault namespace.
• Write argocd/app-configs/vault-network-policies/network-policies.yaml with vault server + vault-agent-injector CNPs.
• Create kustomization.yaml (namespace: vault).
• Create ArgoCD Application template with finalizer.
• Update docs/operations/cilium.md enforced-list.
• yamllint + kustomize-build (CNP count check).
• Branch-wide rumdl.
• Push + PR with explicit revert recipe in the body.

Post-merge verification (run ALL of these)

1. Raft healthy: kubectl exec -n vault vault-0 -- vault status -format=json | jq '.ha_enabled, .ha_connection_healthy' — both true.
2. Vault leader stable: kubectl exec -n vault vault-0 -- vault operator raft list-peers — 3 peers, all Voter.
3. Zero DROPPED on vault namespace legit traffic: hubble observe --namespace vault --verdict DROPPED --since 5m — empty.
4. ESO refresh: kubectl annotate externalsecret -n dolt dolt-credentials force-sync="$(date +%s)" --overwrite — succeeds; Secret data updated.
5. cert-manager Vault PKI: trigger a non-ACME Certificate renewal (vault-issuer) — Certificate reaches Ready.
6. Vault UI: curl -sI https://vault.fzymgc.house/v1/sys/health (via Traefik) — returns 200 (or expected status code per cluster).
7. vault-agent injection: create a test pod with vault.hashicorp.com/agent-inject: "true" annotation and a token role — pod starts and sidecar injects.
8. probe-deny test: probe pod in default ns tries vault:8200 — DROPPED with “Policy denied”.

24h soak

Watch for any DroppedFlows alerts in Grafana on vault namespace. The riskiest tail is at the 8/12/16h marks when normal token TTLs expire and renewal cycles run.