Cilium PR4a: Crown-jewel CNPs for dolt — Implementation Plan

For agentic workers: Use superpowers:subagent-driven-development or superpowers:executing-plans. Checkboxes track tasks.

Goal: Land the first crown-jewel CiliumNetworkPolicy set, for the dolt namespace. Validates the per-namespace pattern (label-keyed selectors + DNS L7 rule + Hubble flow visibility) before scaling to cert-manager and ESO in PR4b.

Scope decision: PR4 in the spec covered three namespaces (dolt, cert-manager, external-secrets); splitting into PR4a (dolt) + PR4b (cert-manager+ESO). After PR3’s multi-iteration discovery process, smaller blast radius lets us validate each pattern empirically before scaling.

Architecture: A single new manifest file argocd/app-configs/dolt/network-policies.yaml ships six CiliumNetworkPolicy resources. The set establishes default-deny (via a baseline policy with empty ingress: []/egress: []) and then opens specific allow holes for each known traffic pattern. Selectors use app: dolt (existing label on the StatefulSet pod) — not app.kubernetes.io/name: dolt, because the upstream chart doesn’t set that label and changing it is out of scope (tracked as a label-hygiene follow-up bead).

Tech Stack: Cilium 1.19.3 (policyEnforcementMode: default), ArgoCD (GitOps), Hubble (DNS L7 visibility per-policy).

Spec: docs/engineering/specs/2026-05-12-cilium-post-migration-hardening-design.md §“PR4: Crown-Jewel CNPs (dolt, cert-manager, external-secrets)”.

Traffic Audit

Direction	Peer	Port	Source identity (Cilium)
Ingress	External SQL clients (LAN, via LoadBalancer + SNAT)	3306	host + remote-node (SNAT to node IP)
Ingress	Traefik (doltdb-remote.fzymgc.house)	50051	pods in traefik namespace, label app.kubernetes.io/name: traefik
Ingress	Prometheus scrape (ServiceMonitor dolt)	9418	pods in prometheus namespace, label app.kubernetes.io/name: prometheus
Egress	kube-dns	53 UDP/TCP + DNS L7	k8s-app: kube-dns in kube-system
Egress	kube-apiserver	443	kube-apiserver entity

No Vault egress from dolt pods (ExternalSecret operator pulls from Vault and writes Secret resources into the dolt namespace — the operator’s egress, not dolt’s).

Policies (6 CNPs in one file)

1. dolt-default-deny — baseline, empty ingress: [] / egress: [].
2. dolt-ingress-sql — 3306 from host + remote-node + world (permissive: LB+SNAT erases source IP).
3. dolt-ingress-remotesapi — 50051 from traefik pods.
4. dolt-ingress-metrics — 9418 from prometheus pods.
5. dolt-egress-dns — 53 to kube-dns with DNS L7 rule (engages Hubble visibility for dolt).
6. dolt-egress-kube-apiserver — to the kube-apiserver entity.

Tasks

• Pre-flight: confirm PR3+#1032 chart state, dolt healthy, no existing CNPs.
• Write argocd/app-configs/dolt/network-policies.yaml (6 CNPs).
• Append network-policies.yaml to argocd/app-configs/dolt/kustomization.yaml.
• Add a “currently enforced” inventory line for dolt to docs/operations/cilium.md.
• yamllint + kubectl kustomize argocd/app-configs/dolt/ returns 6 CNPs.
• Branch-wide rumdl (memory: lefthook-jj-lint-scope-gap).
• Push + PR.

Post-merge verification

• dolt ArgoCD Application stays Synced/Healthy.
• All 6 CNPs report VALID: True via kubectl get cnp -n dolt.
• dolt pod stays Running, no restart.
• SQL probe: mysql -h 192.168.20.147 -P 3306 -e "SELECT 1" from Sean’s workstation.
• Remote API probe: curl -sI https://doltdb-remote.fzymgc.house/ returns expected status.
• Prometheus targets page shows dolt/dolt still UP.
• Hubble surfaces DNS flows from dolt: hubble observe --type l7 --namespace dolt --since 60s non-empty.

24h soak before PR4b

After all post-merge probes pass, leave the policies in place for 24h before opening PR4b (cert-manager + ESO). Watch for any DroppedFlows alerts on dolt pods.

Failure recovery

If a probe fails post-merge:

1. Identify the broken path via hubble observe --namespace dolt --verdict DROPPED --since 5m.
2. Edit the relevant CNP, OR temporarily delete dolt-default-deny to restore allow-all (kubectl delete cnp -n dolt dolt-default-deny).
3. Open a follow-up PR with the fix.