Cilium Post-Migration Hardening — Design

Date: 2026-05-12 Author: Sean Brandt (with Claude Opus 4.7) Status: Draft — pending implementation plan Predecessor: 2026-05-10-calico-to-cilium-design.md

Context

Calico → Cilium migration completed 2026-05-12 (PR4 #1018 production cutover; PR5 #1019 legacy Calico NetworkPolicy sweep). All 8 nodes are Cilium-managed; policyEnforcementMode=default is live; the cross-CNI NetworkPolicy gap that motivated PRs #1015/#1016 is moot.

Three loose-end concerns surfaced post-cutover:

1. The cf-ssh-bastion workload — Cloudflared-tunneled SSH-to-nodes — is vestigial; the Cloudflared path it relied on has been retired.
2. NetworkPolicy coverage is extremely sparse. After bastion teardown, the cluster will have zero CiliumNetworkPolicies. With policyEnforcementMode=default, every namespace is allow-all.
3. Hubble UI is deployed (hubble-ui ClusterIP svc in kube-system) but not exposed. Operationally it is reachable only via port-forward.

Goals

• Delete the bastion cleanly with no orphaned resources.
• Adopt a “crown-jewels” NetworkPolicy posture: default-deny on namespaces that hold credentials, identity state, or persistent data. Leave application namespaces open.
• Bake label-keyed identity into all new CiliumNetworkPolicies (Strategy A from brainstorming). Avoid bare endpointSelector: {} so future pods landing in a crown-jewel namespace do not silently inherit privileges.
• Expose Hubble UI on hubble.fzymgc.house behind Authentik forward-auth.
• Use Hubble flow observation as the empirical verification ritual for each policy PR.

Non-Goals

• Cilium Mutual Authentication (SPIFFE/SPIRE). Tracked as a follow-up bead; out of scope for this initiative.
• Application-namespace NetworkPolicy coverage beyond the crown jewels.
• Migration of cert-manager Vault routing from vault-internal to vault-active. Surfaced during this audit; tracked as a separate parallel bead because it’s independent of the NP work and shares the routing-anti-pattern with ESO (memory: cilium-pr3-complete).

Architecture

Sequencing

Five PRs land in order. Each PR is independently revertable.

PR	Title	Risk	Soak
PR1	Remove cf-ssh-bastion + tunnel route	Low	Immediate
PR2	Expose Hubble UI on hubble.fzymgc.house	Low	Immediate
PR3	Enable Cilium L7 DNS proxy (prerequisite for toFQDNs egress rules)	Medium	24h
PR4	Crown-jewel CNPs: dolt → cert-manager → external-secrets	Medium	24h per ns
PR5	Crown-jewel CNPs: cnpg-system → authentik → vault	High	48h on vault

PR3 is a prerequisite to PR4/PR5 because cert-manager’s egress to Let’s Encrypt and the Cloudflare API will be expressed as toFQDNs rules.

PR4 and PR5 split the crown-jewel rollout by coupling complexity: dolt/cert-manager/ESO have narrower traffic surface; cnpg/authentik/vault have HA-internal flows (raft, replication) that demand more care.

Components

PR1: Bastion Teardown

Repo changes:

• Delete argocd/app-configs/cf-ssh-bastion/ (entire directory).
• Delete argocd/cluster-app/templates/cf-ssh-bastion.yaml. This Application carries metadata.finalizers: [resources-finalizer.argocd.argoproj.io], so deletion cascades to all child resources including the namespace.
• Audit Cloudflared tunnel config (location TBD — likely argocd/app-configs/cloudflared/ or a Cloudflare-side Terraform module under tf/) and remove the ingress route :2222 → cf-ssh-bastion.cf-ssh-bastion.svc. If a Cloudflare DNS record for the SSH host is managed via Terraform, remove it too.
• The Vault path that backed the bastion ExternalSecret (host keys, SSH config) — leave intact for now; tracked in PR1 verification step as a “document as orphaned, defer removal” item.

Verification:

• kubectl get application cf-ssh-bastion -n argocd → NotFound.
• kubectl get ns cf-ssh-bastion → NotFound.
• kubectl get pod,cnp -A | grep -i bastion returns no rows.
• Cloudflared tunnel reload succeeded; no :2222 route remains.

Bead hygiene: close hl-7h0 (PR4 Calico decommission) as stale — its work landed in #1018/#1019.

PR2: Hubble UI on hubble.fzymgc.house

New argocd/app-configs/hubble-ui/ directory:

• namespace.yaml — new hubble namespace.
• certificate.yaml — cert-manager Certificate (secretName: hubble-tls, issuerRef: vault-issuer ClusterIssuer, dnsNames: [hubble.fzymgc.house]). Matches longhorn-dashboard pattern.
• traefik-middleware.yaml — Traefik Middleware/modern-auth in hubble namespace. forwardAuth to http://authentik-server.authentik.svc/outpost.goauthentik.io/auth/traefik with the longhorn/traefik-dashboard header allowlist (Remote-User, Remote-Groups, Remote-Email, Remote-Name, plus X-authentik-*).
• ingress-route.yaml — Traefik IngressRoute in hubble namespace. Host hubble.fzymgc.house, middleware ref to local modern-auth, service ref cross-namespace to name: hubble-ui, namespace: kube-system, port: 80. TLS secretName: hubble-tls. Carries router-hosts.fzymgc.house/enabled: "true" annotation.
• kustomization.yaml — bundle.

New argocd/cluster-app/templates/hubble-ui.yaml:

• ArgoCD Application for the directory above. MUST include metadata.finalizers: [resources-finalizer.argocd.argoproj.io] so future teardown cascades cleanly (memory: argocd-application-deletion).
• This PR ships no ExternalSecret, so the standard ignoreDifferences block for ESO remote-ref drift is not required here. Add it later only if/when an ExternalSecret is introduced (memory: argocd-app-externalsecret-ignoredifferences).

Terraform changes (tf/authentik/):

• New authentik_provider_proxy resource for hubble.fzymgc.house (forward-auth domain-level mode).
• New authentik_application binding the provider.
• Group restriction: align with longhorn-dashboard’s Authentik app (probably an admin/operator group). Confirm exact group at implementation time.
• Outpost assignment: add to the existing embedded outpost.

Verification:

• DNS: dig hubble.fzymgc.house resolves via Traefik (the router-hosts controller propagates the host).
• TLS: certificate hubble-tls reaches Ready in Hubble namespace.
• Anonymous request: curl -sI https://hubble.fzymgc.house/ returns 302 to authentik login.
• Authenticated browser request: Hubble UI loads with live flow data; UI can switch namespaces.

PR3: Cilium L7 DNS Proxy Enablement

Prerequisite for PR4/PR5 cert-manager toFQDNs rules.

Repo changes (ansible/roles/cilium/templates/values.yaml.j2):

• Confirm/enable enableL7Proxy: true (likely already on by default in 1.19.x but verify).
• Enable DNS visibility / capture via the appropriate chart value. Specific value to be confirmed against Cilium 1.19.x chart docs at spec-implementation time; candidates are dnsPolicy.enabled / proxy.dns.enable / a permissive CiliumNetworkPolicy with toFQDNs: ["*"] and toPorts: dns-rules.

Apply: re-run ansible/roles/cilium against control-plane to helm upgrade. Watch cilium-agent pod restart-rolls per node.

Verification:

• hubble observe --to-fqdn '*' --from-namespace kube-system --last 100 shows FQDN labels on flows (today, this query returns nothing — that’s the symptom).
• No application-level regression: ESO refreshes succeed, traefik continues to route correctly.

PR4: Crown-Jewel CNPs (dolt, cert-manager, external-secrets)

Per-namespace pattern. Each namespace gets a small set of CNPs (default-deny + explicit allows). Selectors use app.kubernetes.io/name and app.kubernetes.io/component labels — never bare endpointSelector: {}, never namespaceSelector alone.

dolt CNPs:

• default-deny: endpointSelector: {matchLabels: {app.kubernetes.io/name: dolt}}, no ingress/egress.
• Ingress: from traefik namespace pods to SQL port (3306) and remote API (50051). Match by app.kubernetes.io/name: traefik.
• Ingress: from monitoring namespace Prometheus to dolt metrics (port 9418, per memory: dolt-deployment).
• Egress: kube-dns (UDP/TCP 53 to k8s-app=kube-dns in kube-system).
• Egress: kube-apiserver entity.

cert-manager CNPs:

• default-deny for each cert-manager component (app.kubernetes.io/name: cert-manager and component: controller|webhook|cainjector).
• Ingress: Prometheus → :9402 on all three components.
• Ingress: kube-apiserver → cert-manager-webhook for admission webhook.
• Egress: kube-dns, kube-apiserver entity.
• Egress: vault-internal.vault.svc:8200 — match by label app.kubernetes.io/name: vault, component: server in namespace vault. (NB: cert-manager today points at vault-internal headless service; the routing-fix bead may move this to vault-active before this NP lands. If so, no policy change needed since both resolve to the same Vault pods.)
• Egress (toFQDNs, PR3-enabled): acme-v02.api.letsencrypt.org:443, api.cloudflare.com:443, plus any subdomains LE redirects to.

external-secrets CNPs:

• default-deny on app.kubernetes.io/name: external-secrets, component: controller|webhook|cert-controller.
• Ingress: Prometheus → controller metrics endpoint.
• Ingress: kube-apiserver → ESO webhook.
• Egress: kube-dns, kube-apiserver entity.
• Egress: vault-active.vault.svc:8200 — match by label app.kubernetes.io/name: vault, component: server (the namespace-and-label combo, not raw service name). Confirms intent matches the memory note about leader-only routing.

Per-policy verification (Hubble flow-driven ritual):

1. Apply policy via ArgoCD.
2. Wait for Cilium identity allocation.
3. Trigger the workload’s outbound activity explicitly:
   • dolt: run bd push/pull from operator machine, query via traefik.
   • cert-manager: annotate a Certificate for force-renewal (cert-manager.io/issue-temporary-certificate).
   • external-secrets: annotate an ExternalSecret for force-refresh (force-sync: <timestamp>).
4. kubectl exec -n kube-system ds/cilium -- hubble observe --namespace <ns> --last 200 for 60–120s. Confirm:
   • No DROPPED flows for traffic the policy should permit.
   • All FORWARDED flows match expected egress destinations.
5. 24h soak with monitoring alerts watched. If clean, proceed to next namespace.

PR5: Crown-Jewel CNPs (cnpg-system, authentik, vault)

These three have HA-internal flows that demand extra care.

cnpg-system CNPs:

• Operator and per-cluster pods need policy. Selectors include cnpg.io/cluster label for postgres pods (operator labels are app.kubernetes.io/name: cloudnative-pg).
• Ingress: operator API from kube-apiserver (webhook), Prometheus → operator metrics.
• Ingress (per-postgres-cluster ns): traffic from owning app pods only, by label (postgres port 5432).
• Egress: kube-dns, kube-apiserver, intra-namespace replication (port 5432 between pods sharing cnpg.io/cluster label).
• CRITICAL: validate streaming replication remains unbroken. Memory note cnpg-cpu-format recalls how subtle config drift can break CNPG; replication is on the same risk surface here.

authentik CNPs:

• default-deny on app.kubernetes.io/name: authentik (separate for component: server | worker | outpost).
• Ingress: from traefik to authentik-server port 9000.
• Ingress (intra-namespace): outpost ↔ server.
• Egress: own postgres (cnpg.io/cluster: authentik in same ns or external — confirm topology), redis if present (check chart values), kube-dns, kube-apiserver.

vault CNPs (last, most coupled):

• default-deny on app.kubernetes.io/name: vault, component: server.
• Ingress: from traefik (port 8200), from external-secrets controllers (port 8200), cluster-internal from agent-injector webhook callers (port 8443 on injector pod). Match by labels.
• Ingress: from cert-manager (port 8200).
• Ingress: kube-apiserver → vault-agent-injector mutating webhook.
• Egress: kube-dns, kube-apiserver, k8s-API for SA TokenReview.
• CRITICAL — intra-namespace raft heartbeat (port 8201) between vault server pods. Memory note cilium-pr3-complete flags this as the highest-risk failure mode if missed. Allow rule: endpointSelector: app.kubernetes.io/name=vault, component=server to itself on port 8201.

Verification ritual: same Hubble flow-driven pattern as PR4, plus:

• Authentik: log in via SSO, verify all downstream apps (longhorn, traefik dashboard) still forward-auth correctly. 48h soak before considering complete.
• Vault: validate ESO refresh cycle, cert-manager cert issuance, vault-active leader election (kill leader pod, confirm new leader within 30s).

Data Flow

Per-PR flows are described in the per-PR sections above. Cross-PR data flow during rollout:

Operator (workstation)
   ↓ kubectl apply (via ArgoCD GitOps)
ArgoCD Application Controller
   ↓ reconciles CiliumNetworkPolicy CRDs
Cilium Operator + cilium-agent (per node)
   ↓ allocates identity, programs eBPF
Workload pods (now bound by policy)
   ↓ traffic
Hubble (relay → CLI on workstation)
   ↓ flow observation
Operator (verification feedback loop)

Error Handling and Rollback

Per-PR rollback strategy:

• PR1: revert the ArgoCD Application deletion commit; namespace and resources recreate.
• PR2: revert the new hubble-ui Application + directory; ArgoCD prunes.
• PR3: revert ansible role change, re-run ansible to roll back Cilium chart values. Cilium agents restart-roll.
• PR4/PR5: each per-namespace CNP set is independently revertable. Roll back by deleting the CNPs from the relevant argocd/app-configs/<ns>/ directory. Critical: revert applies immediately; traffic begins flowing again the moment the CNP is removed.

Failure modes to watch:

• Vault raft heartbeat dropped (PR5): symptom is split-brain or all-followers state, Vault API returns HTTP 474. Per memory, read full /v1/sys/health JSON body — don’t trust the status code alone.
• ESO refresh failures (PR4): ExternalSecret SecretSyncedError events.
• Cert-manager renewal failures (PR4): Certificate Ready: False, CertificateRequest events showing connection refused to vault or ACME.
• Authentik forward-auth 502 (PR5): any Authentik-protected app returns 502 — likely an ingress block on authentik-server. The traefik forward-auth is in the authentik namespace and is reached from many namespaces; verify the policy allows ingress from all currently-using namespaces.

Testing Strategy

Per-PR functional tests:

• PR1: bastion namespace gone, no orphaned resources.
• PR2: anonymous → 302, authenticated → Hubble UI loads.
• PR3: hubble observe --to-fqdn '*' shows FQDN labels.
• PR4/PR5: Hubble flow-driven ritual described above. 24h soak per namespace (48h for vault).

Regression watch:

• Grafana dashboards for cluster traffic, error rates, certificate expiry, ESO sync success.
• ArgoCD app sync status: all apps remain Synced and Healthy.

Quality gates:

• yamllint argocd/ clean.
• helm template argocd/cluster-app --name-template cluster-app --namespace argocd renders (memory: cluster-app-helm-template-trap).
• rumdl fmt docs/ clean if touching docs.
• mkdocs build warning-free if touching docs (memory: mkdocs-nav-curated).

Documentation Pointers

New runbooks to add (each in its respective PR):

• docs/operations/hubble-ui.md — how to access, how to read flows, common debugging recipes.
• docs/operations/cilium-network-policies.md — crown-jewel policy inventory, label conventions, verification ritual, rollback recipes.
• Update mkdocs.yml nav: to list the new docs (memory: mkdocs-nav-curated).

Open Questions for Implementation Time

1. Exact Cilium chart values for L7 DNS visibility on 1.19.x — confirm against upstream docs.
2. Cloudflared tunnel config location for the bastion route — needs grep at PR1 implementation time.
3. Authentik group restriction for hubble.fzymgc.house — align with longhorn dashboard’s existing app config.
4. CNPG postgres clusters that need per-cluster policy in PR5 — enumerate live: kubectl get cluster.postgresql.cnpg.io -A.

Bead Plan

• Close: hl-7h0 (PR4 Calico decommission) — stale, work merged.
• New beads to file:
  • “Re-point cert-manager vault-issuer from vault-internal to vault-active” (P2, parallel track, not blocking this initiative).
  • “Spike Cilium Mutual Auth + SPIRE for crown-jewel namespaces” (P3, depends on PR5 complete).
  • “Audit app.kubernetes.io/* label hygiene across all workloads” (P3, useful beyond NP work).
  • One bead per PR (PR1..PR5) to track status, soak, blockers.