Cilium Crown-Jewel CNP Redesign Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

DESIGN UPDATE 2026-05-13 — L4-ONLY CONSTRAINT: This cluster’s Cilium L7 DNS proxy is broken (bead hl-bdr, tcpdump-confirmed). The upstream-recommended workaround (dnsproxy-enable-transparent-mode=false, PR #1071) DID NOT FIX it. Until either (a) someone roots out the deeper bug or (b) Cilium 1.20+ Standalone DNS Proxy graduates and we adopt it (bead hl-r39), all CNPs ship as L4-only: no rules.dns: matchPattern, no toFQDNs egress. The “Components” sections below predate this constraint and reference L7 DNS / toFQDNs rules; strip those when implementing. See “L4-only retrofit” section near the end for what changes vs. the original plan.

Goal: Re-introduce CiliumNetworkPolicy coverage on external-secrets, cert-manager, and vault namespaces — reverted in PR #1051 — using empirically-gated functional probes (not admission-webhook annotations, not Hubble L7 stream) and per-namespace soak between PRs. Updated: L4-only design only.

Architecture: Three sequential PRs, one namespace per PR, in least-to-most-coupled order: external-secrets → cert-manager → vault. Each PR ships a per-namespace <ns>-network-policies directory + ArgoCD Application (matches the reverted PR4b/PR5a structure). Each is verified with a functional probe — for ESO an force-sync annotation, for cert-manager a Certificate renewal, for vault a leader-kill drill. Out-of-band kubectl apply precedes ArgoCD adoption to enable fast revert; the merged Git state matches what kubectl applied.

Tech Stack: Cilium 1.19.x CNPs, ArgoCD ApplicationSet pattern, kustomize, kubectl + Hubble CLI on cilium-agent pods, bash for probe loops.

Key constraints (don’t violate, the spec is more detailed):

Verification gate is functional probes only. hubble observe --type l7 is permanently off the table (memory: hubble-l7-flow-stream-empty-empirical).
24h soak between PRs. Don’t stack PR-β until PR-α has 24h of healthy ESO refresh activity.
K3s API server identity caveat: every admission-webhook ingress rule MUST carry [kube-apiserver, host, remote-node] (memory: cilium-enabledefaultdeny-false-l7-trap).
ESO recovery procedure pre-staged: kubectl rollout restart deploy/external-secrets-operator -n external-secrets is the post-failure ritual — bake it into runbook before merge.
Sean’s directive on vault egress: vault egress should be permissive (toEntities: [all] or no egress CNP). Restrict who can CALL vault, not what vault reaches.
No enableDefaultDeny: false — direction-specific CNPs implicitly engage default-deny, and the validator rejects empty-rule-list “baseline” CNPs (memory: cilium-enabledefaultdeny-false-l7-trap / third gotcha).
OOB kubectl apply IS approved for the verification window — explicit Sean approval overrides the MUST NOT apply kubectl changes directly rule in argocd/CLAUDE.md for THIS plan only.
No sandbox ExternalSecret needed — probe existing low-risk ExternalSecrets (e.g., tandoor-secrets).

Pre-Flight: Apply Once, Universal to All Three PRs

Task PF-1: Verify k3s API-server identity reaches admission-webhook pods as `host` / `remote-node`

Why: The PR4b/PR5a failure root cause was admission-webhook ingress only allowing kube-apiserver entity, which doesn’t match in k3s. The fix exists (#1045 added host + remote-node) but the reverted CNPs never re-landed with that fix proven. This task confirms our assumption is still valid before any new CNP touches an admission-webhook flow.

Files: none (investigation only)

Step 1: Find a cilium-agent on the same node as the cert-manager-webhook pod

kubectl --context fzymgc-house get pods -n cert-manager -l app.kubernetes.io/component=webhook -o wide
# Note NODE column, then:
kubectl --context fzymgc-house get pods -n kube-system -l k8s-app=cilium -o wide | grep <NODE>

Record the cilium-agent pod name (e.g., cilium-xlk88).

Step 2: Observe admission-webhook ingress flows for 30s

kubectl --context fzymgc-house exec -n kube-system <cilium-agent-pod> -c cilium-agent -- \
  hubble observe --to-namespace cert-manager --to-port 10250 --last 50 --follow &
PID=$!
sleep 30
kill $PID

Then trigger admission activity by force-renewing any cert-manager Certificate:

kubectl --context fzymgc-house get certificate -A -o json | jq -r \
  '.items[] | select(.spec.issuerRef.name=="vault-issuer") | "\(.metadata.namespace) \(.metadata.name)"' | head -1
# Take the first one returned and:
kubectl --context fzymgc-house annotate certificate -n <ns> <name> \
  cert-manager.io/issue-temporary-certificate="$(date +%s)" --overwrite

Expected output from hubble observe: source identity is (host) or (remote-node) for traffic into cert-manager/cert-manager-webhook-*:10250. NOT (kube-apiserver).

Step 3: Record the finding

If observed identity is host or remote-node: assumption confirmed. Proceed to PR-α.

If observed identity is kube-apiserver: STOP. The k3s gotcha was fixed upstream and our memory is stale. Update memory cilium-enabledefaultdeny-false-l7-trap to reflect new behavior, then re-evaluate whether admission-webhook ingress rules need adjusting.

Task PF-2: Pin live label snapshots for all three namespaces

Why: The reverted CNPs had correct selectors but the labels are independent of our control (Helm chart upgrades can change them). Capture a frozen snapshot to detect chart-level changes between writing and applying.

Files: docs/engineering/plans/cilium-cnp-redesign-label-snapshot-2026-05-13.md (frozen reference)

Step 1: Snapshot all relevant pod labels into the plan reference doc

Run and copy output verbatim into the new file:

echo "=== external-secrets ==="
kubectl --context fzymgc-house get pods -n external-secrets --show-labels
echo ""
echo "=== cert-manager ==="
kubectl --context fzymgc-house get pods -n cert-manager --show-labels
echo ""
echo "=== vault ==="
kubectl --context fzymgc-house get pods -n vault --show-labels
echo ""
echo "Snapshot taken: $(date -u +%Y-%m-%dT%H:%M:%SZ)"

Step 2: Create the snapshot file

Path: docs/engineering/plans/cilium-cnp-redesign-label-snapshot-2026-05-13.md

Contents: title + the verbatim output from Step 1 inside a text fenced block. Add a short note at top: “Frozen snapshot for the cilium-cnp-redesign plan — selectors in the implementation MUST match these labels.”

Step 3: Commit the snapshot

jj commit -m "docs(plans): snapshot pod labels for cilium-cnp-redesign"

PR-α: external-secrets Crown-Jewel CNPs

Risk: Highest of the three because of the documented ESO operator vault-client-cache poisoning failure mode (memory: cilium-cnp-rollout-empirical-gates). Lowest in coupling because external-secrets only talks to vault and kube-apiserver.

Functional probe: kubectl annotate externalsecret ... force-sync=$(date +%s) --overwrite on three real ExternalSecrets across different namespaces; each must reach Ready=True within 60s.

Recovery procedure (PRE-STAGE): before applying anything, open a separate terminal, type but do NOT execute:

kubectl --context fzymgc-house rollout restart deploy/external-secrets-operator -n external-secrets

If the probe in Step 11 fails, this is the very next command after kubectl delete -f.

Task α-1: Create `argocd/app-configs/external-secrets-network-policies/` directory

Files:

Create: argocd/app-configs/external-secrets-network-policies/kustomization.yaml
Create: argocd/app-configs/external-secrets-network-policies/network-policies.yaml

Steps:

Step 1: Create the kustomization

argocd/app-configs/external-secrets-network-policies/kustomization.yaml:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: external-secrets

resources:
  - network-policies.yaml

Step 2: Write the network-policies.yaml

Use the verified pod labels (per snapshot doc) as endpointSelectors:

operator: app.kubernetes.io/name: external-secrets
webhook: app.kubernetes.io/name: external-secrets-webhook
cert-controller: app.kubernetes.io/name: external-secrets-cert-controller

argocd/app-configs/external-secrets-network-policies/network-policies.yaml:

# Cilium Network Policies for the external-secrets namespace.
#
# Three components, each selected separately by app.kubernetes.io/name
# (verified empirically against live pods 2026-05-13; see
# cilium-cnp-redesign-label-snapshot-2026-05-13.md).
#
# Each direction-specific CNP engages default-deny for its direction.
# K3s admission-webhook ingress includes [kube-apiserver, host, remote-node]
# from the start (memory: cilium-enabledefaultdeny-false-l7-trap, gotcha 4).

# ============================================================
# OPERATOR (controller)
# ============================================================

# Operator: egress for DNS, kube-apiserver, and Vault. Operator pulls
# secrets from Vault on every ExternalSecret refresh — critical path.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: external-secrets-operator-egress
  namespace: external-secrets
  labels:
    app.kubernetes.io/component: network-policy
    app.kubernetes.io/part-of: external-secrets
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: external-secrets
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s:k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: ANY
          rules:
            dns:
              - matchPattern: "*"
    - toEntities:
        - kube-apiserver
    # Vault server pods (operator → vault-active.vault.svc:8200, leader-only)
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: vault
            k8s:app.kubernetes.io/name: vault
            k8s:component: server
      toPorts:
        - ports:
            - port: "8200"
              protocol: TCP
---
# ============================================================
# WEBHOOK
# ============================================================

# Webhook: ingress for admission from kube-apiserver. K3s requires
# host + remote-node alongside kube-apiserver entity (the entity alone
# silently never matches in k3s).
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: external-secrets-webhook-ingress
  namespace: external-secrets
  labels:
    app.kubernetes.io/component: network-policy
    app.kubernetes.io/part-of: external-secrets
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: external-secrets-webhook
  ingress:
    - fromEntities:
        - kube-apiserver
        - host
        - remote-node
      toPorts:
        - ports:
            - port: "10250"
              protocol: TCP
        - ports:
            - port: "443"
              protocol: TCP
---
# Webhook: egress for DNS + kube-apiserver.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: external-secrets-webhook-egress
  namespace: external-secrets
  labels:
    app.kubernetes.io/component: network-policy
    app.kubernetes.io/part-of: external-secrets
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: external-secrets-webhook
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s:k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: ANY
          rules:
            dns:
              - matchPattern: "*"
    - toEntities:
        - kube-apiserver
---
# ============================================================
# CERT-CONTROLLER
# ============================================================

# Cert-controller: egress only (no service exposed).
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: external-secrets-cert-controller-egress
  namespace: external-secrets
  labels:
    app.kubernetes.io/component: network-policy
    app.kubernetes.io/part-of: external-secrets
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: external-secrets-cert-controller
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s:k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: ANY
          rules:
            dns:
              - matchPattern: "*"
    - toEntities:
        - kube-apiserver

Task α-2: Validate manifest locally

Step 1: yamllint pass

yamllint argocd/app-configs/external-secrets-network-policies/

Expected: no errors. If errors, fix and re-run.

Step 2: Kustomize render

kubectl --context fzymgc-house kustomize argocd/app-configs/external-secrets-network-policies/

Expected: 4 documents emitted (1 operator-egress, 2 webhook policies, 1 cert-controller-egress).

Step 3: Server-side dry-run validates against Cilium CRD schema

kubectl --context fzymgc-house apply --server-side --dry-run=server \
  -k argocd/app-configs/external-secrets-network-policies/

Expected: <each-cnp-name> serverside-applied (server dry run). NO Valid=False messages.

Task α-3: Create the ArgoCD Application template

Files:

Create: argocd/cluster-app/templates/external-secrets-network-policies.yaml
Step 1: Write the Application template

Mirrors the reverted PR4b structure, with the resources-finalizer included for clean cascade if reverted:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets-network-policies
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  project: core-services
  sources:
    - repoURL: https://github.com/fzymgc-house/selfhosted-cluster
      targetRevision: HEAD
      path: argocd/app-configs/external-secrets-network-policies
  destination:
    server: https://kubernetes.default.svc
    namespace: external-secrets
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
      - RespectIgnoreDifferences=true

Step 2: Verify cluster-app helm template still renders

helm template argocd/cluster-app --name-template cluster-app --namespace argocd \
  > /tmp/cluster-app-render.yaml
echo "exit=$?"
grep -c "kind: Application" /tmp/cluster-app-render.yaml

Expected: exit=0; the Application count is one higher than before (use git diff main -- argocd/cluster-app/templates/ to confirm one new entry).

Task α-4: Commit the manifests (NOT pushing yet)

Step 1: jj commit

jj commit -m "feat(external-secrets): crown-jewel CNPs (PR-α/redesign)

Three direction-specific CiliumNetworkPolicies for the external-secrets
namespace covering operator, webhook, and cert-controller. Selectors
verified against live pod labels (snapshot 2026-05-13). Webhook ingress
includes [kube-apiserver, host, remote-node] from the start to handle
the k3s API-server identity gotcha.

Replaces the reverted PR #1044 (PR4b) with the cilium-cnp-redesign
plan's verification ritual: out-of-band kubectl apply, three-ES
force-sync probe, then commit + ArgoCD adoption.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>"

Task α-5: Capture probe baseline (BEFORE applying CNPs)

Why: to detect probe regressions caused by the CNPs we’re about to apply, you need a known-healthy baseline.

Step 1: Pick three ExternalSecrets across different namespaces for the probe set

kubectl --context fzymgc-house get externalsecret -A \
  -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' \
  | head -20

Pick 3 across distinct namespaces, prioritizing low-blast-radius (NOT vault unsealing keys, NOT critical infra). Record them in scratch:

PROBE_ES=(
  "tandoor:tandoor-secrets"
  "<ns2>:<es2>"
  "<ns3>:<es3>"
)

Step 2: Verify all three are currently Ready=True

for entry in "${PROBE_ES[@]}"; do
  ns="${entry%%:*}"; name="${entry##*:}"
  status=$(kubectl --context fzymgc-house get externalsecret -n "$ns" "$name" \
    -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')
  echo "$ns/$name -> $status"
done

Expected: all three print True. If any prints anything else, swap that entry for a healthy ES before continuing.

Task α-6: OOB-apply the CNPs

This is the explicit-Sean-approved out-of-band application. It precedes the git push.

Step 1: Apply via kubectl directly

kubectl --context fzymgc-house apply --server-side \
  -k argocd/app-configs/external-secrets-network-policies/

Expected: 4 serverside-applied messages (or unchanged on a re-run).

Step 2: Verify CNPs are Valid=True

kubectl --context fzymgc-house get cnp -n external-secrets

Expected: 4 CNPs, all VALID=True. If any are VALID=False, run kubectl describe cnp -n external-secrets <name> for the validator message and fix.

Step 3: Verify enforcement state on a sampled endpoint

Find a cilium-agent on the same node as external-secrets-operator-*:

NODE=$(kubectl --context fzymgc-house get pod -n external-secrets \
  -l app.kubernetes.io/name=external-secrets \
  -o jsonpath='{.items[0].spec.nodeName}')
AGENT=$(kubectl --context fzymgc-house get pod -n kube-system -l k8s-app=cilium \
  -o wide | awk -v node="$NODE" '$7==node{print $1}')
echo "operator on $NODE, agent $AGENT"
kubectl --context fzymgc-house exec -n kube-system "$AGENT" -c cilium-agent -- \
  cilium-dbg endpoint list | grep -i "external-secrets" | head -5

Expected: the operator endpoint shows Enabled for at least Egress (a direction-specific CNP engages default-deny for that direction).

Task α-7: Run the functional probe

Step 1: Probe loop

for entry in "${PROBE_ES[@]}"; do
  ns="${entry%%:*}"; name="${entry##*:}"
  echo "--- probing $ns/$name ---"
  kubectl --context fzymgc-house annotate externalsecret -n "$ns" "$name" \
    force-sync="$(date +%s)" --overwrite
  # Poll for Ready=True
  for i in $(seq 1 12); do
    status=$(kubectl --context fzymgc-house get externalsecret -n "$ns" "$name" \
      -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')
    age=$(kubectl --context fzymgc-house get externalsecret -n "$ns" "$name" \
      -o jsonpath='{.status.conditions[?(@.type=="Ready")].lastTransitionTime}')
    echo "  [$i/12] Ready=$status (lastTransition=$age)"
    [ "$status" = "True" ] && break
    sleep 5
  done
  if [ "$status" != "True" ]; then
    echo "PROBE FAILED for $ns/$name"
    break
  fi
done

Expected: each ES reaches Ready=True within 60s of force-sync.

Step 2: Confirm ClusterSecretStore is still Ready

kubectl --context fzymgc-house get clustersecretstore vault \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status} {.status.conditions[?(@.type=="Ready")].message}'
echo

Expected: True store validated. If False: probe FAILED.

Step 3: Survey hubble for unexpected drops

kubectl --context fzymgc-house exec -n kube-system "$AGENT" -c cilium-agent -- \
  hubble observe --namespace external-secrets --verdict DROPPED --last 100

Expected: empty (no drops). If drops appear, examine source/destination — the probe Ready=True alone isn’t sufficient if hubble shows drops on tangential flows.

Task α-8: Branch on probe result

If probe PASSED: continue to Task α-9 (commit + push).
If probe FAILED: execute the recovery sequence:

# 1. Remove the CNPs
kubectl --context fzymgc-house delete -k argocd/app-configs/external-secrets-network-policies/

# 2. Restart ESO operator (per memory: required even after CNPs removed)
kubectl --context fzymgc-house rollout restart deploy/external-secrets-operator -n external-secrets

# 3. Wait for operator to come back
kubectl --context fzymgc-house rollout status deploy/external-secrets-operator -n external-secrets --timeout=2m

# 4. Re-verify CSS health
kubectl --context fzymgc-house get clustersecretstore vault \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'

# 5. Re-run the probe loop from Task α-7 to confirm recovery

Then DO NOT commit/push. Instead: investigate root cause (likely a label drift caught in the snapshot diff), file a beads follow-up, and abandon the jj change with jj abandon @-.

Task α-9: Push the branch and open a PR

Only execute if Task α-7 PASSED AND ClusterSecretStore is still Ready.

Step 1: Push the bookmark

jj git push --bookmark cilium-cnp-redesign

If push is rejected because the branch is new, use jj git push --bookmark cilium-cnp-redesign --allow-new.

Step 2: Open the PR

gh pr create --title "feat(external-secrets): crown-jewel CNPs (PR-α/redesign)" \
  --body "$(cat <<'EOF'
## Summary

- Re-introduces CiliumNetworkPolicy coverage for external-secrets (operator, webhook, cert-controller) — reverted in #1051
- Selectors verified against live pod labels (snapshot 2026-05-13)
- Webhook ingress includes \`[kube-apiserver, host, remote-node]\` for k3s API server identity
- Out-of-band applied + functional-probe verified (3 ExternalSecrets force-synced, all reached Ready=True within 60s) before this PR

## Verification

- \`kubectl get cnp -n external-secrets\`: 4 CNPs Valid=True
- ClusterSecretStore \`vault\` Ready=True
- Three ExternalSecrets in distinct namespaces force-synced and reached Ready=True within 60s
- \`hubble observe --namespace external-secrets --verdict DROPPED\`: no drops

## Pre-merge soak

This PR has been live in the cluster (out-of-band) for ~30 minutes. ArgoCD will adopt on merge.

Per the cilium-cnp-redesign plan, this PR must soak for 24h before PR-β (cert-manager) lands.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

Step 3: Confirm ArgoCD will adopt on merge

kubectl --context fzymgc-house get application external-secrets-network-policies -n argocd 2>&1

Expected: NotFound (not yet created). After merge, cluster-app will create it on the next sync wave.

Task α-10: Update bd and the spec

Step 1: Update hl-7q1 bead with PR-α status

bd update hl-7q1 --status in_progress
bd comment hl-7q1 --body "PR-α (external-secrets) opened as PR #<N>. Out-of-band applied, 3-ES probe passed, ClusterSecretStore healthy. 24h soak begins on merge."

Step 2: After merge, verify ArgoCD adoption

(This step happens after PR-α is merged — out of scope for this plan’s initial execution session, but document the expected check.)

# After merge:
kubectl --context fzymgc-house get application external-secrets-network-policies -n argocd \
  -o jsonpath='{.status.sync.status} {.status.health.status}'

Expected: Synced Healthy.

Task α-11: Wait 24h, then proceed to PR-β

Do not start PR-β work until 24h after PR-α merge. During that window, watch for any ESO-related anomaly (Grafana dashboards, vault-related Slack/email alerts). If any anomaly surfaces, file a bead and decide whether to revert PR-α or continue.

PR-β: cert-manager Crown-Jewel CNPs

Risk: Medium. Cert-manager has the toFQDNs egress complexity (LE + Cloudflare) but no operator-cache poisoning failure mode like ESO. K3s admission-webhook gotcha applies.

Functional probe: force-renew a non-production Certificate (e.g., a router-hosts test cert), confirm Ready=True within 5min.

Sub-skill note: all the Pre-Flight tasks (PF-1, PF-2) have already been completed for the initiative. PR-β reuses the snapshot from PF-2 — no need to re-snapshot.

Task β-1: Create `argocd/app-configs/cert-manager-network-policies/`

Files:

Create: argocd/app-configs/cert-manager-network-policies/kustomization.yaml
Create: argocd/app-configs/cert-manager-network-policies/network-policies.yaml

Steps:

Step 1: kustomization.yaml

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: cert-manager

resources:
  - network-policies.yaml

Step 2: network-policies.yaml

Pod labels per snapshot:

controller: app.kubernetes.io/name: cert-manager + app.kubernetes.io/component: controller
webhook: app.kubernetes.io/name: webhook + app.kubernetes.io/component: webhook
cainjector: app.kubernetes.io/name: cainjector + app.kubernetes.io/component: cainjector

# Cilium Network Policies for the cert-manager namespace.
#
# Three components selected by app.kubernetes.io/name (cert-manager /
# webhook / cainjector). Note: cert-manager's chart uses bare component
# names ("webhook", "cainjector") for the name label, NOT the prefixed
# form ("cert-manager-webhook"). Verified against live pods in the
# 2026-05-13 snapshot.
#
# Each direction-specific CNP engages default-deny for its direction.
# Webhook ingress includes [kube-apiserver, host, remote-node] for k3s.

# ============================================================
# CONTROLLER
# ============================================================

# Controller egress: DNS (with L7 rule for matching toFQDNs IPs) +
# kube-apiserver + Vault PKI + ACME/Cloudflare via toFQDNs. The DNS
# L7 rule MUST be co-located with toFQDNs in the same policy — Cilium's
# DNS proxy intercepts the query and learns the IP for toFQDNs matching.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: cert-manager-controller-egress
  namespace: cert-manager
  labels:
    app.kubernetes.io/component: network-policy
    app.kubernetes.io/part-of: cert-manager
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: cert-manager
      app.kubernetes.io/component: controller
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s:k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: ANY
          rules:
            dns:
              - matchPattern: "*"
    - toEntities:
        - kube-apiserver
    # Vault server (PKI issuance via vault-issuer ClusterIssuer)
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: vault
            k8s:app.kubernetes.io/name: vault
            k8s:component: server
      toPorts:
        - ports:
            - port: "8200"
              protocol: TCP
    # ACME (Let's Encrypt) + Cloudflare API for DNS-01 challenges
    - toFQDNs:
        - matchName: acme-v02.api.letsencrypt.org
        - matchPattern: "*.letsencrypt.org"
        - matchName: api.cloudflare.com
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP
---
# ============================================================
# WEBHOOK
# ============================================================

# Webhook ingress: admission from kube-apiserver (k3s identity dance).
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: cert-manager-webhook-ingress
  namespace: cert-manager
  labels:
    app.kubernetes.io/component: network-policy
    app.kubernetes.io/part-of: cert-manager
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: webhook
      app.kubernetes.io/component: webhook
  ingress:
    - fromEntities:
        - kube-apiserver
        - host
        - remote-node
      toPorts:
        - ports:
            - port: "10250"
              protocol: TCP
        - ports:
            - port: "443"
              protocol: TCP
---
# Webhook egress: DNS + kube-apiserver.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: cert-manager-webhook-egress
  namespace: cert-manager
  labels:
    app.kubernetes.io/component: network-policy
    app.kubernetes.io/part-of: cert-manager
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: webhook
      app.kubernetes.io/component: webhook
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s:k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: ANY
          rules:
            dns:
              - matchPattern: "*"
    - toEntities:
        - kube-apiserver
---
# ============================================================
# CAINJECTOR
# ============================================================

# Cainjector egress: DNS + kube-apiserver (no service exposed).
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: cert-manager-cainjector-egress
  namespace: cert-manager
  labels:
    app.kubernetes.io/component: network-policy
    app.kubernetes.io/part-of: cert-manager
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: cainjector
      app.kubernetes.io/component: cainjector
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s:k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: ANY
          rules:
            dns:
              - matchPattern: "*"
    - toEntities:
        - kube-apiserver

Task β-2: Validate manifest locally

Step 1: yamllint

yamllint argocd/app-configs/cert-manager-network-policies/

Step 2: Server-side dry-run

kubectl --context fzymgc-house apply --server-side --dry-run=server \
  -k argocd/app-configs/cert-manager-network-policies/

Expected: 4 CNPs, all valid.

Task β-3: Create the ArgoCD Application template

Files:

Create: argocd/cluster-app/templates/cert-manager-network-policies.yaml
Step 1:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cert-manager-network-policies
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  project: core-services
  sources:
    - repoURL: https://github.com/fzymgc-house/selfhosted-cluster
      targetRevision: HEAD
      path: argocd/app-configs/cert-manager-network-policies
  destination:
    server: https://kubernetes.default.svc
    namespace: cert-manager
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
      - RespectIgnoreDifferences=true

Step 2: Verify cluster-app helm template still renders

helm template argocd/cluster-app --name-template cluster-app --namespace argocd \
  > /tmp/cluster-app-render.yaml
echo "exit=$?"

Expected: exit=0.

Task β-4: Pick the probe Certificate

Step 1: Identify a non-production Certificate to renew

kubectl --context fzymgc-house get certificate -A \
  -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.spec.issuerRef.name}{"\n"}{end}' \
  | rg "vault-issuer|self-signed" | head -10

Pick one with low blast radius (not the dolt-tls cert — that gates SQL access; not the vault-tls — that gates everything). A router-hosts cert or a hubble-tls cert is ideal — single-consumer, easy to rebuild if it breaks.

Record: PROBE_CERT_NS=<ns> PROBE_CERT_NAME=<name>.

Task β-5: Commit, OOB-apply, probe, branch on result

Step 1: jj commit (manifest only, no apply yet)

jj commit -m "feat(cert-manager): crown-jewel CNPs (PR-β/redesign)

Three direction-specific CNPs for cert-manager (controller, webhook,
cainjector). Selectors match live pod labels (cert-manager chart
uses bare component names like 'webhook', not 'cert-manager-webhook').
Webhook ingress includes [kube-apiserver, host, remote-node] for k3s.
Controller egress allows toFQDNs to ACME + Cloudflare with co-located
DNS L7 rule for FQDN IP learning.

Replaces the cert-manager half of reverted PR #1044 (PR4b).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>"

Step 2: OOB-apply

kubectl --context fzymgc-house apply --server-side \
  -k argocd/app-configs/cert-manager-network-policies/

Step 3: Verify Valid=True

kubectl --context fzymgc-house get cnp -n cert-manager

Step 4: Force-renew the probe cert

kubectl --context fzymgc-house annotate certificate -n "$PROBE_CERT_NS" "$PROBE_CERT_NAME" \
  cert-manager.io/issue-temporary-certificate="$(date +%s)" --overwrite

Step 5: Poll for renewal

for i in $(seq 1 30); do
  status=$(kubectl --context fzymgc-house get certificate -n "$PROBE_CERT_NS" "$PROBE_CERT_NAME" \
    -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')
  echo "[$i/30] $PROBE_CERT_NS/$PROBE_CERT_NAME Ready=$status"
  [ "$status" = "True" ] && break
  sleep 10
done

Expected: Ready=True within 5 minutes (30 × 10s).

Step 6: Survey hubble for drops

NODE=$(kubectl --context fzymgc-house get pod -n cert-manager \
  -l app.kubernetes.io/component=controller \
  -o jsonpath='{.items[0].spec.nodeName}')
AGENT=$(kubectl --context fzymgc-house get pod -n kube-system -l k8s-app=cilium \
  -o wide | awk -v node="$NODE" '$7==node{print $1}')
kubectl --context fzymgc-house exec -n kube-system "$AGENT" -c cilium-agent -- \
  hubble observe --namespace cert-manager --verdict DROPPED --last 100

Expected: empty.

Task β-6: Branch on probe result

If PASSED: push + open PR (mirror of Task α-9, with cert-manager body):

jj git push --bookmark cilium-cnp-redesign
gh pr create --title "feat(cert-manager): crown-jewel CNPs (PR-β/redesign)" --body "...adapted body..."

If FAILED: revert sequence:

kubectl --context fzymgc-house delete -k argocd/app-configs/cert-manager-network-policies/
# Probe again to confirm cert renews now
kubectl --context fzymgc-house annotate certificate -n "$PROBE_CERT_NS" "$PROBE_CERT_NAME" \
  cert-manager.io/issue-temporary-certificate="$(date +%s)" --overwrite

If recovery succeeds, abandon the jj change. File a beads follow-up with the failure mode.

Task β-7: 24h soak before PR-γ

PR-γ: vault Crown-Jewel CNPs

Risk: Highest. Vault has raft heartbeat (port 8201), API forwarding (port 8200), and is the keystone of the whole secret-distribution path. Egress is permissive per Sean’s directive.

Functional probe: kill the current vault leader; confirm new leader election within 30s; confirm raft Ready=True for all three nodes; confirm ESO can still refresh (re-run α-7 probe).

Pre-stage recovery commands:

# If vault gets into a bad state:
kubectl --context fzymgc-house delete -k argocd/app-configs/vault-network-policies/
# If leader-kill probe doesn't see a new leader within 60s:
#   investigate via 'kubectl exec vault-0 -- vault status -format=json' on each pod

Task γ-1: Create `argocd/app-configs/vault-network-policies/`

Files:

Create: argocd/app-configs/vault-network-policies/kustomization.yaml
Create: argocd/app-configs/vault-network-policies/network-policies.yaml

Steps:

Step 1: kustomization.yaml

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: vault

resources:
  - network-policies.yaml

Step 2: network-policies.yaml — ingress-restrictive, egress-permissive

Pod labels per snapshot:

vault server: app.kubernetes.io/name: vault + component: server
vault-agent-injector: app.kubernetes.io/name: vault-agent-injector + component: webhook

# Cilium Network Policies for the vault namespace.
#
# Two workload identities:
#   - vault server (StatefulSet): app.kubernetes.io/name=vault, component=server
#   - vault-agent-injector (Deployment): app.kubernetes.io/name=vault-agent-injector, component=webhook
#
# DESIGN PER SEAN'S DIRECTIVE: ingress is restrictive (control who can
# CALL vault). Egress is intentionally OMITTED — vault may need to reach
# arbitrary external systems (cloud KMS, audit log endpoints, future
# integrations). Adding an egress CNP that's missing one toEndpoints
# entry would break vault silently. Restrict callers, not destinations.
#
# CRITICAL: raft heartbeat (port 8201, intra-namespace server↔server)
# MUST be allowed in ingress. Memory cilium-pr3-complete documents the
# class of split-brain failure if missed.

# ============================================================
# VAULT SERVER — INGRESS
# ============================================================

# Raft heartbeat (CRITICAL). Intra-namespace, server-to-server, port 8201.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: vault-server-ingress-raft
  namespace: vault
  labels:
    app.kubernetes.io/component: network-policy
    app.kubernetes.io/part-of: vault
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: vault
      component: server
  ingress:
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: vault
            k8s:app.kubernetes.io/name: vault
            k8s:component: server
      toPorts:
        - ports:
            - port: "8201"
              protocol: TCP
---
# Vault API (8200) — multiple authorized sources.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: vault-server-ingress-api
  namespace: vault
  labels:
    app.kubernetes.io/component: network-policy
    app.kubernetes.io/part-of: vault
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: vault
      component: server
  ingress:
    # Peer vault server pods (API forwarding from standby to leader)
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: vault
            k8s:app.kubernetes.io/name: vault
            k8s:component: server
      toPorts:
        - ports:
            - port: "8200"
              protocol: TCP
    # vault-agent-injector
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: vault
            k8s:app.kubernetes.io/name: vault-agent-injector
      toPorts:
        - ports:
            - port: "8200"
              protocol: TCP
    # Traefik (UI/API ingress)
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: traefik
            k8s:app.kubernetes.io/name: traefik
      toPorts:
        - ports:
            - port: "8200"
              protocol: TCP
    # cert-manager controller (Vault PKI issuance)
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: cert-manager
            k8s:app.kubernetes.io/name: cert-manager
            k8s:app.kubernetes.io/component: controller
      toPorts:
        - ports:
            - port: "8200"
              protocol: TCP
    # external-secrets operator (every ExternalSecret refresh)
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: external-secrets
            k8s:app.kubernetes.io/name: external-secrets
      toPorts:
        - ports:
            - port: "8200"
              protocol: TCP
    # host + remote-node (vault-agent sidecars on hostNetwork pods,
    # operator port-forwards via node IP, router-hosts vault-unseal flows)
    - fromEntities:
        - host
        - remote-node
      toPorts:
        - ports:
            - port: "8200"
              protocol: TCP
---
# ============================================================
# VAULT-AGENT-INJECTOR — INGRESS
# ============================================================

# Admission webhook from kube-apiserver. K3s identity dance applies.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: vault-agent-injector-ingress
  namespace: vault
  labels:
    app.kubernetes.io/component: network-policy
    app.kubernetes.io/part-of: vault
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: vault-agent-injector
      component: webhook
  ingress:
    - fromEntities:
        - kube-apiserver
        - host
        - remote-node
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP

(NB: no egress CNPs at all — direction-specific default-deny only engages for INGRESS by virtue of these CNPs. Egress for vault server + vault-agent-injector remains allow-all. This is the deliberate Sean-permissive design.)

Task γ-2: Validate locally

Step 1: yamllint + dry-run

yamllint argocd/app-configs/vault-network-policies/
kubectl --context fzymgc-house apply --server-side --dry-run=server \
  -k argocd/app-configs/vault-network-policies/

Expected: 3 CNPs, all valid.

Task γ-3: ArgoCD Application template

Files:

Create: argocd/cluster-app/templates/vault-network-policies.yaml
Step 1:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: vault-network-policies
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  project: core-services
  sources:
    - repoURL: https://github.com/fzymgc-house/selfhosted-cluster
      targetRevision: HEAD
      path: argocd/app-configs/vault-network-policies
  destination:
    server: https://kubernetes.default.svc
    namespace: vault
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
      - RespectIgnoreDifferences=true

Step 2: Verify cluster-app render

helm template argocd/cluster-app --name-template cluster-app --namespace argocd \
  > /tmp/cluster-app-render.yaml
echo "exit=$?"

Task γ-4: Capture vault-state baseline

Step 1: Identify current leader

for pod in vault-0 vault-1 vault-2; do
  status=$(kubectl --context fzymgc-house exec -n vault $pod -- vault status -format=json 2>/dev/null | jq -r '.is_self + " " + (.ha_enabled|tostring)')
  echo "$pod: leader=$status"
done

Record the current leader pod (e.g., LEADER=vault-1).

Step 2: Confirm raft state

kubectl --context fzymgc-house exec -n vault $LEADER -- vault operator raft list-peers

Expected: 3 voters, all alive.

Task γ-5: Commit, OOB-apply, leader-kill probe

Step 1: jj commit (manifest only)

jj commit -m "feat(vault): crown-jewel CNPs — ingress-restrictive, egress-permissive (PR-γ/redesign)

Three direction-specific CNPs for vault. Vault server gets two ingress
policies (raft heartbeat 8201, API 8200 from authorized callers).
Vault-agent-injector gets one ingress policy (admission webhook 8080
with k3s identity dance: kube-apiserver + host + remote-node).

Egress is intentionally omitted per Sean's directive: restrict who can
CALL vault, not what vault can reach. Vault may need cloud KMS, audit
log destinations, etc. — easier to keep egress open than to play
whack-a-mole with toEndpoints entries.

Replaces the reverted PR #1048 (PR5a) with the cilium-cnp-redesign
plan's leader-kill verification ritual.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>"

Step 2: OOB-apply

kubectl --context fzymgc-house apply --server-side \
  -k argocd/app-configs/vault-network-policies/

Step 3: Verify Valid=True

kubectl --context fzymgc-house get cnp -n vault

Expected: 3 CNPs, all VALID=True.

Step 4: Wait 60s for endpoint policy to settle

sleep 60

Step 5: Pre-probe — verify ESO refresh still works (no-regression check on PR-α path)

Re-run the probe loop from Task α-7. Expected: all three ExternalSecrets reach Ready=True within 60s.

If this fails: vault ingress is too restrictive. Add the missing source. DO NOT proceed to leader-kill.

Step 6: Pre-probe — verify cert-manager renewal still works (no-regression check on PR-β path)

Re-run the cert renewal probe from Task β-5 Step 4-5.

If this fails: vault ingress doesn’t allow cert-manager. Add the missing source.

Step 7: Leader-kill probe — induce a re-election

# Note current leader (from Task γ-4)
echo "Killing current leader: $LEADER"
kubectl --context fzymgc-house delete pod -n vault "$LEADER" --grace-period=10

Step 8: Watch for new leader within 60s

for i in $(seq 1 12); do
  for pod in vault-0 vault-1 vault-2; do
    is_self=$(kubectl --context fzymgc-house exec -n vault $pod -- vault status -format=json 2>/dev/null | jq -r '.is_self // false')
    [ "$is_self" = "true" ] && echo "[$i/12] new leader: $pod" && NEW_LEADER=$pod && break 2
  done
  sleep 5
done

Expected: NEW_LEADER set within 60s.

Step 9: Re-confirm raft list-peers

sleep 30  # let killed pod re-join
kubectl --context fzymgc-house exec -n vault $NEW_LEADER -- vault operator raft list-peers

Expected: 3 voters, all alive.

Step 10: Final no-regression check on ESO

Re-run the α-7 probe loop.

Expected: 3 ESes Ready=True within 60s.

Task γ-6: Branch on probe result

If PASSED: push + PR (mirror of α-9):

jj git push --bookmark cilium-cnp-redesign
gh pr create --title "feat(vault): crown-jewel CNPs (PR-γ/redesign)" --body "..."

If FAILED: revert + investigate:

kubectl --context fzymgc-house delete -k argocd/app-configs/vault-network-policies/
# Re-run α-7 probe to confirm baseline restored

If a vault pod is wedged after the leader-kill, try kubectl delete pod on it (idempotent).

Task γ-7: After PR-γ merges, close hl-7q1

Step 1: Update bd

bd update hl-7q1 --status closed
bd comment hl-7q1 --body "All three CNP PRs merged: external-secrets (PR-α), cert-manager (PR-β), vault (PR-γ). 24h soak between each. No incidents. Resolves the PR4b/PR5a revert (#1051)."

Step 2: Update the cilium-post-migration-hardening spec

In docs/engineering/specs/2026-05-12-cilium-post-migration-hardening-design.md, add a section near the bottom:

## Update 2026-05-XX (post-redesign)

PR4b + PR5a were reverted in #1051 due to the k3s admission-webhook
gotcha. They were re-shipped in three separate PRs (PR-α, PR-β, PR-γ)
following the cilium-cnp-redesign plan with empirical functional
probes as verification gates. Vault egress was changed from restrictive
to permissive per Sean's directive ("restrict who can CALL vault, not
what vault can reach").

Self-Review Notes

Spec coverage:

PR1 (bastion teardown): out of scope; landed in PR #1023 (already done)
PR2 (Hubble UI): out of scope; landed in PR #1027 (already done)
PR3 (Cilium L7 DNS proxy): out of scope; landed in PR #1028 + #1032 (already done)
PR4a (dolt CNPs): out of scope; landed in PR #1039 + #1041 (already done, working baseline)
PR4b (cert-manager + ESO): COVERED here as PR-α (ESO) + PR-β (cert-manager). Split per Sean’s “don’t bundle multiple namespaces” directive.
PR5a (vault): COVERED here as PR-γ with permissive-egress redesign per Sean’s directive.

Placeholder scan: None remaining. All commands are concrete, all YAMLs complete, all expected outputs explicit.

Type consistency: Selectors verified against live pod labels (Task PF-2 captures the snapshot to compare against). All policy endpointSelector matches a real pod label set as observed on 2026-05-13.

Open Items / Follow-ups (file as separate beads if they materialize)

PR4a’s dolt CNPs use bare app: dolt selector instead of app.kubernetes.io/name: dolt. Consistent with the existing dolt StatefulSet but inconsistent with the new policies. Don’t fix in this initiative — that’s a label-hygiene cleanup beyond CNP scope.
No CNPs for cnpg-system, authentik — original PR5 spec covered these. They’re explicitly out of scope here; tracked as hl-bng for the post-migration sweep.
Hubble L7 flow stream remains empty even with policy-engaged endpoints; bead hl-50v already closed as practical resolution. If a future Cilium upgrade fixes this, revisit the verification ritual (could shift back to flow-driven gates).

L4-only retrofit (added 2026-05-13 post-incident)

The original Components sections above were written before bead hl-bdr was discovered. They reference L7 DNS rules and toFQDNs egress — neither is usable on this cluster (workaround in PR #1071 didn’t fix it; see memory cilium-l7-dns-proxy-degrades-under-load). When implementing, strip those constructs as follows:

Universal substitutions (apply to every CNP in this plan)

Original (delete)	Replacement
`toPorts: [...] rules: dns: [{matchPattern: "*"}]` on the kube-dns egress	Just the L4 `toPorts` block (no `rules:` section) — pattern matches `argocd/app-configs/dolt/network-policies.yaml` after PR #1070
`toFQDNs: [{matchName: ...}, {matchPattern: ...}]` egress	Either `toEntities: [world]` (broad), or `toCIDR: [...]` if you can enumerate the destinations, or omit egress restriction entirely (per Sean’s vault directive)

Per-namespace impact

PR-α (external-secrets): No toFQDNs exist in the original. Just strip the rules.dns block from the operator’s kube-dns egress. Same for webhook-egress and cert-controller-egress. The K3s admission-webhook ingress fix ([kube-apiserver, host, remote-node]) stays.

PR-β (cert-manager): Original controller-egress has BOTH a rules.dns: matchPattern: "*" block AND toFQDNs: [acme-v02..., *.letsencrypt.org, api.cloudflare.com]. With L4-only:

Strip the rules.dns block (kube-dns L4 egress remains)
Replace the toFQDNs block with toEntities: [world] to permit external HTTPS to ACME and Cloudflare — accept the broader allow because we cannot enumerate ACME’s IP set safely
Document the trade-off in the file comment header

PR-γ (vault): Original was already L4-only with permissive egress per Sean’s directive. No retrofit needed.

Probe ritual unchanged

Functional probes (ESO force-sync, cert-manager cmctl renew, vault leader-kill) all work without any L7 DNS observability. The hubble observe --verdict DROPPED --namespace <ns> post-merge survey remains the L3/L4 health check.

What we lose vs. original design

Cluster has zero CNPs with L7 DNS rules after this plan ships — accepted constraint
Hubble L7 DNS attribution unavailable for any namespace — accept; relevant for incident debugging only, not steady-state operation
cert-manager egress is broader (world instead of three specific FQDNs) — small surface increase; cert-manager already has Vault credentials so an exfiltration via cert-manager is not a meaningful new risk vs. what an attacker could already do

When to revisit (revert this retrofit, restore L7)

Either of these triggers re-evaluation:

hl-bdr gets a real fix (someone roots out the deeper bug or upstream Cilium ships a fix that works on ARM64/Armbian)
hl-r39 lands (Cilium 1.20+ Standalone DNS Proxy in production)

In both cases: re-test L7 DNS on a probe pod first; only restore L7 rules to crown-jewel CNPs if the probe passes cleanly.

Cilium Crown-Jewel CNP Redesign Implementation Plan

Pre-Flight: Apply Once, Universal to All Three PRs

Task PF-1: Verify k3s API-server identity reaches admission-webhook pods as host / remote-node

Task PF-2: Pin live label snapshots for all three namespaces

PR-α: external-secrets Crown-Jewel CNPs

Task α-1: Create argocd/app-configs/external-secrets-network-policies/ directory

Task α-2: Validate manifest locally

Task α-3: Create the ArgoCD Application template

Task α-4: Commit the manifests (NOT pushing yet)

Task α-5: Capture probe baseline (BEFORE applying CNPs)

Task α-6: OOB-apply the CNPs

Task α-7: Run the functional probe

Task α-8: Branch on probe result

Task α-9: Push the branch and open a PR

Task α-10: Update bd and the spec

Task α-11: Wait 24h, then proceed to PR-β

PR-β: cert-manager Crown-Jewel CNPs

Task β-1: Create argocd/app-configs/cert-manager-network-policies/

Task β-2: Validate manifest locally

Task β-3: Create the ArgoCD Application template

Task β-4: Pick the probe Certificate

Task β-5: Commit, OOB-apply, probe, branch on result

Task β-6: Branch on probe result

Task β-7: 24h soak before PR-γ

PR-γ: vault Crown-Jewel CNPs

Task γ-1: Create argocd/app-configs/vault-network-policies/

Task γ-2: Validate locally

Task γ-3: ArgoCD Application template

Task γ-4: Capture vault-state baseline

Task γ-5: Commit, OOB-apply, leader-kill probe

Task γ-6: Branch on probe result

Task γ-7: After PR-γ merges, close hl-7q1

Self-Review Notes

Open Items / Follow-ups (file as separate beads if they materialize)

L4-only retrofit (added 2026-05-13 post-incident)

Universal substitutions (apply to every CNP in this plan)

Per-namespace impact

Probe ritual unchanged

What we lose vs. original design

When to revisit (revert this retrofit, restore L7)

Task PF-1: Verify k3s API-server identity reaches admission-webhook pods as `host` / `remote-node`

Task α-1: Create `argocd/app-configs/external-secrets-network-policies/` directory

Task β-1: Create `argocd/app-configs/cert-manager-network-policies/`

Task γ-1: Create `argocd/app-configs/vault-network-policies/`