Cilium PR4b: Crown-jewel CNPs for cert-manager + external-secrets — Implementation Plan

For agentic workers: Use superpowers:subagent-driven-development or superpowers:executing-plans.

Goal: Crown-jewel CiliumNetworkPolicy coverage for cert-manager and external-secrets namespaces, completing the spec’s PR4 scope (PR4a was dolt; this is PR4b for the other two). Validates toFQDNs egress (cert-manager → ACME + Cloudflare API) as the most complex new pattern.

Architecture: Two new ArgoCD-managed directories with their own Applications, each carrying resources-finalizer.argocd.argoproj.io:

• argocd/app-configs/cert-manager-network-policies/ — CNPs scoped to cert-manager components (controller, webhook, cainjector).
• argocd/app-configs/external-secrets-network-policies/ — CNPs scoped to ESO components (operator, webhook, cert-controller).

Lessons applied from PR4a:

• No baseline default-deny CNP. Each direction-specific CNP engages default-deny for its direction. Empty ingress: [] + egress: [] is rejected by Cilium’s validator anyway (memory: cilium-enabledefaultdeny-false-l7-trap).
• DNS L7 rule in the same CNP as toFQDNs. The proxy must intercept DNS in the policy that uses toFQDNs, otherwise IP-learning doesn’t happen.
• Empirical probe-deny test post-merge. Run a probe pod in an unauthorized namespace; expect DROPPED with “Policy denied” — same recipe used to validate PR4a + #1041.

Spec: docs/engineering/specs/2026-05-12-cilium-post-migration-hardening-design.md §“PR4: Crown-Jewel CNPs (dolt, cert-manager, external-secrets)”.

Traffic Audit

cert-manager (3 components, separate selectors)

Component	Labels	Ingress	Egress
controller	app.kubernetes.io/name=cert-manager, component=controller	Prometheus → 9402	DNS L7 (kube-dns) + kube-apiserver + vault pods (name=vault, instance=vault, component=server) on 8200 + toFQDNs: acme-v02.api.letsencrypt.org, *.letsencrypt.org, api.cloudflare.com on 443
webhook	app.kubernetes.io/name=webhook, component=webhook	kube-apiserver → 10250 (webhook); Prometheus → 9402	DNS L7 + kube-apiserver
cainjector	app.kubernetes.io/name=cainjector, component=cainjector	Prometheus → 9402	DNS L7 + kube-apiserver

external-secrets (3 components)

Component	Labels	Ingress	Egress
operator (controller)	app.kubernetes.io/name=external-secrets	Prometheus → 8080 (if metrics enabled)	DNS L7 + kube-apiserver + vault pods on 8200
webhook	app.kubernetes.io/name=external-secrets-webhook	kube-apiserver → 443	DNS L7 + kube-apiserver
cert-controller	app.kubernetes.io/name=external-secrets-cert-controller	(no service)	DNS L7 + kube-apiserver

CNP Files

Each namespace gets ONE network-policies.yaml file with multiple ----separated CNPs (mirrors the dolt pattern in PR4a). Selectors use the actual app.kubernetes.io/name label values observed empirically.

Tasks

• Pre-flight: confirm cert-manager + ESO healthy; no existing CNPs in either namespace.
• Create argocd/app-configs/cert-manager-network-policies/{network-policies.yaml,kustomization.yaml}.
• Create argocd/app-configs/cluster-app/templates/cert-manager-network-policies.yaml Application with finalizer.
• Create argocd/app-configs/external-secrets-network-policies/{network-policies.yaml,kustomization.yaml}.
• Create argocd/app-configs/cluster-app/templates/external-secrets-network-policies.yaml Application with finalizer.
• Update docs/operations/cilium.md “Currently enforced” list.
• yamllint + helm-template-render cluster-app + branch-wide rumdl.
• Push + PR.

Post-merge verification

For EACH namespace:

1. All CNPs VALID=True via kubectl get cnp -n <ns>.
2. Application stays Synced/Healthy.
3. Pods stable (no restart).
4. Probe-deny pod in default namespace gets DROPPED with “Policy denied” when attempting to reach a workload in the namespace.
5. For cert-manager specifically: trigger a force-renewal of a Certificate (annotate with cert-manager.io/issue-temporary-certificate) and watch Hubble for cert-manager pod making egress to LE + Cloudflare. Confirm the connection succeeds (no DROPPED for ACME flows).

Soak

24h before considering PR4b “done.” If toFQDNs has any issues, they’ll surface during the next certificate renewal cycle.

Failure recovery

If cert-manager loses ability to renew certs:

1. hubble observe --namespace cert-manager --verdict DROPPED --since 10m shows the dropped egress.
2. Either widen the toFQDNs matchPattern (e.g., add a redirect destination) OR delete the offending CNP to restore allow.
3. Worst case: kubectl delete cnp -n cert-manager --all to revert to allow-all while debugging.

For ESO secret-refresh failures:

1. hubble observe --namespace external-secrets --verdict DROPPED --since 10m shows the dropped egress.
2. Most likely culprit: Vault egress selector mismatch (vault pod labels).
3. Same revert recipe.

Out of scope

• PR5 namespaces (cnpg-system, authentik, vault). Per spec, those come last with most-coupled risk profile.
• Label hygiene cleanup across crown-jewel namespaces — tracked separately.
• Hubble L7 DNS flow surfacing (hl-50v) — independent investigation.