Cilium PR3: L7 DNS Proxy Enablement — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Make Cilium’s L7 DNS proxy explicitly-configured (it is on by default but undocumented in values) AND deploy a single cluster-wide DNS visibility CiliumClusterwideNetworkPolicy (CCNP) so DNS flows are surfaced in Hubble and so PR4’s per-namespace toFQDNs policies have a working baseline.

Architecture: Two coupled changes. (1) ansible/roles/cilium/templates/values.yaml.j2 gets an explicit l7Proxy: true so the dependency is documented (no behavior change — chart default in 1.19.x is true). (2) A new ArgoCD-managed argocd/app-configs/cilium-policies/ directory ships a CCNP dns-visibility with enableDefaultDeny: {egress: false, ingress: false}, which adds DNS L7 interception cluster-wide WITHOUT flipping any namespace to default-deny. Together they unlock toFQDNs rules in PR4/PR5 and immediately surface DNS flows in Hubble UI (which became reachable in PR2).

Tech Stack: Cilium (chart v1.19.3), Ansible (roles/cilium), ArgoCD (GitOps), Hubble (already deployed).

Spec: docs/engineering/specs/2026-05-12-cilium-post-migration-hardening-design.md §“PR3: Cilium L7 DNS Proxy Enablement”.

File Structure

Path	Action	Purpose
`ansible/roles/cilium/defaults/main.yml`	MODIFY	Add `cilium_l7_proxy_enabled: true` variable
`ansible/roles/cilium/templates/values.yaml.j2`	MODIFY	Add `l7Proxy: {{ cilium_l7_proxy_enabled \| lower }}` to the Helm values
`argocd/app-configs/cilium-policies/dns-visibility.yaml`	CREATE	`CiliumClusterwideNetworkPolicy` with `enableDefaultDeny: {egress: false, ingress: false}` — adds DNS L7 visibility without restricting any traffic
`argocd/app-configs/cilium-policies/kustomization.yaml`	CREATE	Kustomize manifest for the directory
`argocd/cluster-app/templates/cilium-policies.yaml`	CREATE	ArgoCD `Application` managing `argocd/app-configs/cilium-policies/`. MUST include `resources-finalizer.argocd.argoproj.io`
`docs/operations/cilium.md`	MODIFY	Add a “DNS visibility” section near the existing “NetworkPolicy enforcement” section; cross-link to Hubble UI runbook

Why `enableDefaultDeny: {egress: false, ingress: false}`

Default Cilium CCNP/CNP semantics: any pod selected by a policy is in default-deny mode for the directions that policy covers. The pattern endpointSelector: {} selects every pod cluster-wide; if enableDefaultDeny is not set, every pod would suddenly be in egress default-deny, allowing only what the policy explicitly permits.

enableDefaultDeny: {egress: false, ingress: false} flips the CCNP into “additive only” mode: the rules contribute observability/interception capabilities (here, DNS L7 proxy) but do NOT remove any allow-by-default behavior. This is the Cilium-blessed pattern for “make Hubble DNS-aware cluster-wide without changing traffic.” It was added in Cilium v1.14; v1.19 supports it natively.

Pre-flight

Task 0: Verify Pre-State

Files: none (verification only).

Step 0.1: Confirm Cilium version 1.19.x

Run:

kubectl exec -n kube-system ds/cilium -c cilium-agent -- cilium-dbg version 2>&1 | head -5

Expected: client/server reports 1.19.x. If not 1.19.x, STOP — enableDefaultDeny semantics may differ.

Step 0.2: Confirm no CCNPs exist (clean slate for new policy)

kubectl get ciliumclusterwidenetworkpolicies 2>&1

Expected: No resources found.

Step 0.3: Confirm no DNS L7 visibility today (the gap PR3 closes)

CP=$(kubectl get pod -n kube-system -l k8s-app=cilium -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n kube-system "$CP" -c cilium-agent -- hubble observe --to-fqdn "*" --last 100 2>&1 | head -5

Expected: empty (no FQDN-labeled flows). Confirms PR3 is needed.

Step 0.4: Confirm AppProject core-services exists

kubectl get appprojects -n argocd core-services -o jsonpath='{.metadata.name}{"\n"}'

Expected: core-services. The new cilium-policies Application will live in this project (matches traefik, cnpg, etc.).

Implementation

Task 1: Add `cilium_l7_proxy_enabled` Variable

Files: ansible/roles/cilium/defaults/main.yml

Step 1.1: Start a new commit on top of the plan commit

jj --no-pager new -m "feat(ansible/cilium): explicitly enable L7 proxy in chart values

Cilium 1.19.x defaults l7Proxy to true, but the values.yaml.j2 template
did not surface it. Make the dependency explicit so PR4/PR5's toFQDNs
rules have a documented prerequisite.

No behavior change in the live cluster — the helm upgrade is a no-op
diff. Plumbing is for documentation/intent.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>"

Step 1.2: Edit ansible/roles/cilium/defaults/main.yml

Add a new variable in the “Feature scope” section, after cilium_kube_proxy_replacement: "false" and before cilium_hubble_enabled: true:

Before:

# String value, not bool (Cilium 1.19 Helm schema).
cilium_kube_proxy_replacement: "false"
cilium_hubble_enabled: true

After:

# String value, not bool (Cilium 1.19 Helm schema).
cilium_kube_proxy_replacement: "false"

# L7 proxy enables DNS interception (toFQDNs rules) and HTTP L7 visibility in Hubble.
# Chart default in 1.19.x is true; setting explicitly documents the PR4/PR5 dependency.
cilium_l7_proxy_enabled: true

cilium_hubble_enabled: true

Step 1.3: Edit ansible/roles/cilium/templates/values.yaml.j2

Add an explicit l7Proxy: line in the Helm values, positioned near the other top-level feature toggles (after kubeProxyReplacement: and before the hubble: block).

Before:

# String value, not bool (Cilium 1.19 Helm schema).
# Set to "false" during migration: keep kube-proxy running. Full kube-proxy
# replacement is a separate follow-up project to reduce blast radius.
kubeProxyReplacement: "{{ cilium_kube_proxy_replacement }}"

hubble:

After:

# String value, not bool (Cilium 1.19 Helm schema).
# Set to "false" during migration: keep kube-proxy running. Full kube-proxy
# replacement is a separate follow-up project to reduce blast radius.
kubeProxyReplacement: "{{ cilium_kube_proxy_replacement }}"

# L7 proxy: enables DNS interception (toFQDNs in CNP egress rules) and HTTP
# L7 visibility in Hubble. Chart default in 1.19.x is true; setting
# explicitly so PR4/PR5 toFQDNs work has a documented prerequisite.
l7Proxy: {{ cilium_l7_proxy_enabled | lower }}

hubble:

Step 1.4: Validate templates

yamllint ansible/roles/cilium/defaults/main.yml
echo "exit: $?"
ansible-lint ansible/roles/cilium/ 2>&1 | tail -10 || echo "(ansible-lint may report unrelated warnings; verify no new errors)"

Expected: yamllint exit 0; ansible-lint output unchanged from baseline.

Step 1.5: Render the template locally

ansible -i ansible/inventory.yml -m template -a "src=ansible/roles/cilium/templates/values.yaml.j2 dest=/tmp/cilium-values-rendered.yaml" localhost 2>&1 | tail -3 || cat /tmp/cilium-values-rendered.yaml 2>&1 | grep -A 2 "l7Proxy"

Expected: l7Proxy: true appears in the rendered output near the kubeProxyReplacement block. If ansible isn’t available in this environment, skip this step — the j2 syntax is trivial enough that yamllint already covers correctness.

Step 1.6: Verify commit diff

jj --no-pager diff --summary

Expected: 2 M entries: ansible/roles/cilium/defaults/main.yml and ansible/roles/cilium/templates/values.yaml.j2.

Task 2: Create cilium-policies App-Config Directory

Files: argocd/app-configs/cilium-policies/dns-visibility.yaml, argocd/app-configs/cilium-policies/kustomization.yaml

Step 2.1: Start a new commit

jj --no-pager new -m "feat(cilium-policies): cluster-wide DNS visibility CCNP (no traffic change)

Adds a single CiliumClusterwideNetworkPolicy that activates Cilium's
DNS L7 proxy for every pod (endpointSelector: {}) without restricting
egress (enableDefaultDeny.egress: false). This:

1. Surfaces DNS flows in Hubble (visible at hubble.fzymgc.house after
   PR2's ingress landed). 'hubble observe --to-fqdn \"*\"' now returns
   labeled DNS flows.
2. Unlocks toFQDNs rules in PR4/PR5 cert-manager egress policies
   (otherwise toFQDNs has no IP-learning mechanism).

The enableDefaultDeny flip is the Cilium-blessed pattern for cluster-wide
observability without changing the default-allow-when-no-policy behavior
of policyEnforcementMode=default.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>"

Step 2.2: Create argocd/app-configs/cilium-policies/dns-visibility.yaml

# Cluster-wide DNS L7 visibility.
#
# Activates Cilium's DNS proxy for every pod so DNS flows are visible in
# Hubble AND so toFQDNs rules in subsequent CiliumNetworkPolicies (e.g.
# cert-manager in PR4) can learn IPs from DNS responses.
#
# enableDefaultDeny: {egress: false, ingress: false} makes this CCNP
# "additive only" — it adds the DNS L7 proxy capability cluster-wide
# without removing the default-allow-when-no-policy behavior
# (policyEnforcementMode=default).
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: dns-visibility
  labels:
    app.kubernetes.io/name: cilium-policies
    app.kubernetes.io/component: dns-visibility
spec:
  enableDefaultDeny:
    egress: false
    ingress: false
  endpointSelector: {}
  egress:
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: ANY
          rules:
            dns:
              - matchPattern: "*"

Step 2.3: Create argocd/app-configs/cilium-policies/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - dns-visibility.yaml

Step 2.4: Validate

yamllint argocd/app-configs/cilium-policies/
echo "exit: $?"
kubectl kustomize argocd/app-configs/cilium-policies/ >/tmp/cilium-policies-render.yaml 2>&1
echo "exit: $?"
grep -c "^kind: CiliumClusterwideNetworkPolicy$" /tmp/cilium-policies-render.yaml

Expected: yamllint exit 0; kustomize exit 0; CCNP kind-count is 1.

Step 2.5: Verify commit diff

jj --no-pager diff --summary

Expected: 2 A entries under argocd/app-configs/cilium-policies/.

Task 3: Add ArgoCD Application for cilium-policies

Files: argocd/cluster-app/templates/cilium-policies.yaml

Step 3.1: Start a new commit

jj --no-pager new -m "feat(cilium-policies): ArgoCD Application template with cascading finalizer

Manages the new argocd/app-configs/cilium-policies/ directory. The
destination namespace is 'default' because CiliumClusterwideNetworkPolicy
is cluster-scoped (no namespace required); we use the same pattern as
k8s-oidc-rbac (which manages ClusterRoleBindings).

CreateNamespace=false (no namespace to create); finalizer is present so
future teardown cascades cleanly (memory: argocd-application-deletion).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>"

Step 3.2: Create argocd/cluster-app/templates/cilium-policies.yaml

# SPDX-License-Identifier: MIT-0
# ArgoCD Application for cluster-wide CiliumClusterwideNetworkPolicy resources.
# Currently ships dns-visibility (PR3). Future policies (e.g. crown-jewel
# default-deny in PR4/PR5) will be per-namespace CNPs in their own
# app-configs/<service>/ directories rather than added here.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cilium-policies
  namespace: argocd
  annotations:
    # Deploy early so DNS visibility is in place before app-namespace
    # policies that depend on it (PR4 cert-manager toFQDNs rules)
    argocd.argoproj.io/sync-wave: "-3"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: core-services
  sources:
    - repoURL: https://github.com/fzymgc-house/selfhosted-cluster
      targetRevision: HEAD
      path: argocd/app-configs/cilium-policies
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=false
      - ServerSideApply=true

Step 3.3: Render cluster-app helm template

helm template argocd/cluster-app --name-template cluster-app --namespace argocd >/tmp/cluster-app-render.yaml 2>&1
echo "exit: $?"
grep -A 25 "name: cilium-policies$" /tmp/cluster-app-render.yaml | head -30

Expected: exit 0; the new Application appears in the rendered output with finalizer, project core-services, path argocd/app-configs/cilium-policies.

Step 3.4: yamllint

yamllint argocd/cluster-app/templates/cilium-policies.yaml
echo "exit: $?"

Expected: exit 0.

Step 3.5: Verify commit diff

jj --no-pager diff --summary

Expected: 1 A argocd/cluster-app/templates/cilium-policies.yaml.

Task 4: Update `docs/operations/cilium.md`

Files: docs/operations/cilium.md

Step 4.1: Start a new commit

jj --no-pager new -m "docs(cilium): document DNS L7 visibility (PR3)

Notes the cluster-wide dns-visibility CCNP under the NetworkPolicy
enforcement section. Cross-links to the Hubble UI runbook so operators
investigating DNS flows know where to find the live view.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>"

Step 4.2: Add a DNS visibility subsection

Read docs/operations/cilium.md first to confirm the current structure. Find the section that currently reads (after PR1’s edit):

Before:

## NetworkPolicy enforcement

Cluster-wide `policyEnforcementMode: default` — pods without matching CiliumNetworkPolicy are allowed by default; pods with matching policies are restricted to what the policies permit.

No CiliumNetworkPolicies are currently in force. Crown-jewel namespace coverage is tracked under the post-migration hardening initiative (see `docs/engineering/specs/2026-05-12-cilium-post-migration-hardening-design.md`).

## Known follow-ups

After:

## NetworkPolicy enforcement

Cluster-wide `policyEnforcementMode: default` — pods without matching CiliumNetworkPolicy are allowed by default; pods with matching policies are restricted to what the policies permit.

Crown-jewel namespace coverage is tracked under the post-migration hardening initiative (see `docs/engineering/specs/2026-05-12-cilium-post-migration-hardening-design.md`).

### DNS L7 visibility (cluster-wide)

A single `CiliumClusterwideNetworkPolicy` named `dns-visibility` (in `argocd/app-configs/cilium-policies/`) activates Cilium's DNS proxy for every pod via `endpointSelector: {}`. It carries `enableDefaultDeny: {egress: false, ingress: false}` so the policy is **additive only** — DNS L7 interception is enabled cluster-wide WITHOUT removing the default-allow-when-no-policy behavior. Two consequences:

- DNS flows are visible in Hubble UI ([hubble.fzymgc.house](https://hubble.fzymgc.house)) and via `hubble observe --to-fqdn '*'`.
- `toFQDNs` rules in subsequent per-namespace CNPs (e.g. cert-manager's ACME/Cloudflare egress in PR4) can learn IPs from DNS responses.

## Known follow-ups

Step 4.3: Lint

rumdl check docs/operations/cilium.md
echo "exit: $?"

Expected: exit 0.

Step 4.4: Verify commit diff

jj --no-pager diff --summary

Expected: 1 M docs/operations/cilium.md.

Pre-Push Sanity, Push, and PR

Task 5: Lint Sweep + Push + PR

Files: none (workflow only).

Step 5.1: Full PR contents

jj --no-pager log -r 'trunk()..@'
jj --no-pager diff -r 'trunk()..@' --summary

Expected: 5 commits (plan + 4 implementation), file summary:

A docs/engineering/plans/2026-05-12-cilium-pr3-l7-dns-proxy.md
M ansible/roles/cilium/defaults/main.yml
M ansible/roles/cilium/templates/values.yaml.j2
A argocd/app-configs/cilium-policies/dns-visibility.yaml
A argocd/app-configs/cilium-policies/kustomization.yaml
A argocd/cluster-app/templates/cilium-policies.yaml
M docs/operations/cilium.md

Total: 4 A + 3 M.

Step 5.2: Branch-wide rumdl scan (memory: lefthook-jj-lint-scope-gap)

rumdl check $(jj --no-pager diff -r 'trunk()..@' --summary | awk '/^[AM] .*\.md$/{print $2}')
echo "exit: $?"

Expected: exit 0. Catches plan-doc lint issues lefthook would miss.

Step 5.3: lefthook

lefthook run pre-commit 2>&1 | tail -10
echo "exit: $?"

Expected: exit 0.

Step 5.4: Bookmark + push

jj --no-pager bookmark create feat/cilium-pr3-l7-dns-proxy -r @
jj --no-pager git push -b feat/cilium-pr3-l7-dns-proxy

Step 5.5: Open the PR

Use gh pr create --head feat/cilium-pr3-l7-dns-proxy --title "feat(cilium): enable L7 DNS proxy + cluster-wide DNS visibility CCNP (PR3/5)" --body "..." with a summary covering: (a) explicit l7Proxy: true plumbing in ansible role (no behavior change — chart default), (b) new cilium-policies Application managing a single enableDefaultDeny: false CCNP, (c) what unlocks for PR4/PR5.

Post-merge verification (controller’s job)

After Sean merges, run a state-change monitor that watches:

cluster-app re-syncs with new revision
New cilium-policies Application reaches Synced/Healthy
kubectl get ciliumclusterwidenetworkpolicies shows dns-visibility
hubble observe --to-fqdn '*' returns FQDN-labeled flows within ~2 min of DNS activity

Note that the ansible-side change (values.yaml.j2) is a no-op until Sean re-runs the cilium role (ansible-playbook ... -t cilium). Since l7Proxy: true is the chart default, the live cluster already has L7 proxy enabled — the ansible re-run can be deferred to the next routine update.

Out of scope

Per-namespace CNPs with DNS L7 rules — those land in PR4/PR5 for the crown-jewel namespaces.
Hubble Prometheus DNS metrics (hubble.metrics.enabled: [..., "dns"]) — a useful follow-up but unrelated to the toFQDNs prerequisite. File a follow-up bead if desired.
Cilium Mutual Auth (SPIRE) — tracked separately.

Cilium PR3: L7 DNS Proxy Enablement — Implementation Plan

File Structure

Why enableDefaultDeny: {egress: false, ingress: false}

Pre-flight

Task 0: Verify Pre-State

Implementation

Task 1: Add cilium_l7_proxy_enabled Variable

Task 2: Create cilium-policies App-Config Directory

Task 3: Add ArgoCD Application for cilium-policies

Task 4: Update docs/operations/cilium.md

Pre-Push Sanity, Push, and PR

Task 5: Lint Sweep + Push + PR

Post-merge verification (controller’s job)

Out of scope

Why `enableDefaultDeny: {egress: false, ingress: false}`

Task 1: Add `cilium_l7_proxy_enabled` Variable

Task 4: Update `docs/operations/cilium.md`