Keycloak DCR Cluster-Internal Hardening (A1+trust) Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Restore ADR hl-xktj’s cluster-internal-only Keycloak DCR posture by giving agentgateway’s data-plane pod TLS trust for Keycloak’s internal (ICA1-issued) certificate, then re-adding the Traefik network deny hl-thsf.1 removed — without reintroducing the exact 403 that removal fixed.

Architecture: Two sequential PRs with a mandatory live-validation gate between them. PR1 (Tasks 1-2) is pure GitOps manifest addition in argocd/app-configs/agentgateway/: vendor a public root-CA bundle as a ConfigMap, mount it alongside the already-replicated fzymgc-ica1-ca ConfigMap into one directory via the AgentgatewayParameters CR’s opaque spec.deployment.spec overlay, and point rustls-native-certs (agentgateway’s TLS trust source for the DCR proxy call) at that directory via SSL_CERT_DIR. The CoreDNS routing precondition (id.fzymgc.house → the internal keycloak-internal Service) already exists, delivered by an unrelated bead (hl-rs8m) — this plan adds no CoreDNS change. PR2 (Task 3) is a pure manifest change in argocd/app-configs/keycloak/ restoring the Traefik deny-client-registration rule + Middleware.

Tech Stack: Kubernetes ConfigMap/projected volumes, Kustomize (configMapGenerator), agentgateway.dev/v1alpha1 AgentgatewayParameters CRD (Strategic-Merge-Patch overlay), Traefik v3 IngressRoute/Middleware CRDs, ArgoCD (GitOps, targetRevision: HEAD), yamllint.

Spec: docs/engineering/specs/2026-07-02-keycloak-dcr-cluster-internal-hardening-design.md Bead: hl-thsf.2 · Predecessor: hl-thsf.1 · Depends on (already merged): hl-rs8m / PR #1538

Grounding: All line references are against origin/main (commit 0b92dd53, hl-rs8m / PR #1538 — the current tip as of this plan). The local working copy MUST be current with main@origin before editing (jj rebase -s <root> -o main@origin, or start a fresh workspace). Verify argocd/app-configs/agentgateway/parameters.yaml is 51 lines and argocd/app-configs/keycloak/ingress-route.yaml is 113 lines before starting — a different line count means the tree is stale.

Implementation Tasks — PR1 (ICA1 trust bundle for agentgateway)

Task 1: Vendor the public root CA bundle

Files:

Create: argocd/app-configs/agentgateway/cacert.pem
Modify: argocd/app-configs/agentgateway/kustomization.yaml
Step 1: Fetch curl’s maintained public root CA bundle

Run: curl -sL "https://curl.se/ca/cacert.pem" -o argocd/app-configs/agentgateway/cacert.pem

This is curl’s actively-maintained PEM export of Mozilla’s trusted root CA list — the canonical, purpose-built source for exactly this “vendor a public CA bundle” use case.

Step 2: Verify the download is well-formed, not truncated or an error page

Run:

head -3 argocd/app-configs/agentgateway/cacert.pem
rg -c "BEGIN CERTIFICATE" argocd/app-configs/agentgateway/cacert.pem

Expected: the first line is ##, the second is ## Bundle of CA Root Certificates (or similar curl-bundle header — NOT <html>/<!DOCTYPE or any HTTP-error text). The certificate count MUST be greater than 100 (a sanity floor — the exact count drifts upward over time as Mozilla’s root list changes; as of this plan it was 121). If either check fails, the download is corrupted or hit an error page — do not proceed; re-fetch.

Step 3: Wire it into kustomize as a stably-named ConfigMap

In argocd/app-configs/agentgateway/kustomization.yaml, add a configMapGenerator block after the existing resources: list:

configMapGenerator:
  - name: agentgateway-public-ca-bundle
    files:
      - cacert.pem
    options:
      disableNameSuffixHash: true

disableNameSuffixHash: true is required — kustomize’s built-in name-reference transformer only rewrites ConfigMap names inside well-known built-in fields (e.g. Deployment.spec.template.spec.volumes[].configMap.name), and cannot reach into AgentgatewayParameters’ opaque custom-resource overlay field where Task 2 references this ConfigMap by name. Without a stable name, that reference would silently break on every regeneration. This exact pattern (configMapGenerator + disableNameSuffixHash: true) is already established in this repo — see argocd/app-configs/monitoring-vector/kustomization.yaml.

Step 4: Render and lint

Run:

kubectl kustomize argocd/app-configs/agentgateway > /dev/null && echo KUSTOMIZE_OK
yamllint -c .yamllint.yaml argocd/app-configs/agentgateway/kustomization.yaml

Expected: KUSTOMIZE_OK, no yamllint errors. (cacert.pem itself is not YAML and is not yamllint’s concern.)

Step 5: Confirm the ConfigMap renders with the expected name and key

Run: kubectl kustomize argocd/app-configs/agentgateway | yq 'select(.kind == "ConfigMap" and .metadata.name == "agentgateway-public-ca-bundle") | .data | keys'

Expected: ["cacert.pem"].

Step 6: Commit

Commit using VCS-appropriate commands per references/vcs-preamble.md. Message: feat(agentgateway): vendor public root CA bundle for DCR trust [hl-thsf.2]

Task 2: Mount the combined trust bundle and point agentgateway’s TLS trust at it

Files:

Modify: argocd/app-configs/agentgateway/parameters.yaml
Step 1: Pre-flight — confirm the tree is current

Run: wc -l argocd/app-configs/agentgateway/parameters.yaml Expected: 51 lines (before this task’s edit). A different count means the tree is stale — rebase onto main@origin first.

Step 2: Add the projected trust-bundle volume and mount via the deployment.spec overlay

AgentgatewayParameters.spec.deployment.spec is an opaque Strategic-Merge-Patch overlay onto the generated Deployment’s full .spec (i.e. it includes template.spec, not just the Pod-level fields directly) — confirmed against the live CRD schema. Insert this new top-level key into spec:, immediately after the closing of the image: block (i.e. between pullPolicy: IfNotPresent and the existing env: key):

  deployment:
    spec:
      template:
        spec:
          volumes:
            - name: ca-trust
              projected:
                sources:
                  - configMap:
                      name: fzymgc-ica1-ca
                  - configMap:
                      name: agentgateway-public-ca-bundle
          containers:
            - name: agentgateway
              volumeMounts:
                - name: ca-trust
                  mountPath: /etc/agentgateway/ca-trust
                  readOnly: true

fzymgc-ica1-ca is the already-replicated (mittwald replicator) ConfigMap already trusted for agentgateway’s JWKS backend-TLS path — reused unchanged, no new secret-sourcing. agentgateway-public-ca-bundle is the ConfigMap Task 1 created. Both ConfigMaps’ keys (ca.crt and cacert.pem respectively) land as two plain files in the same mounted directory — rustls-native-certs’ SSL_CERT_DIR directory scan (fs::read_dir + parse every plain file, no OpenSSL c_rehash hash-symlink naming required — verified directly against the pinned rustls-native-certs 0.8.4 source and independently re-confirmed via context7) picks up both with no further processing needed.

Step 3: Point SSL_CERT_DIR at the mounted directory

In the existing env: list, append after the OTLP_HEADERS entry:

    - name: SSL_CERT_DIR
      value: "/etc/agentgateway/ca-trust"

Step 4: Render and lint

Run:

kubectl kustomize argocd/app-configs/agentgateway > /dev/null && echo KUSTOMIZE_OK
yamllint -c .yamllint.yaml argocd/app-configs/agentgateway/parameters.yaml

Expected: KUSTOMIZE_OK, no yamllint errors.

Step 5: Confirm the rendered AgentgatewayParameters CR carries the new volume, mount, and env var

kubectl kustomize is pure local YAML templating — it does not invoke the live agentgateway controller, so it never synthesizes an actual Deployment from this CR (that synthesis happens in-cluster, via the controller’s own reconciliation loop, not at render time). Assert against the rendered CR object itself instead:

Run: kubectl kustomize argocd/app-configs/agentgateway | yq 'select(.kind == "AgentgatewayParameters") | {"env": [.spec.env[].name], "volumes": [.spec.deployment.spec.template.spec.volumes[].name], "mounts": [.spec.deployment.spec.template.spec.containers[0].volumeMounts[].name]}'

Expected: env includes SSL_CERT_DIR (alongside TZ, ADMIN_ADDR, OTLP_AUTH_TOKEN, OTLP_HEADERS); volumes and mounts both include ca-trust. The live post-sync check (Live Validation Gate, below) confirms the controller actually applied this overlay to the running pod.

Step 6: Commit

Commit using VCS-appropriate commands per references/vcs-preamble.md. Message: fix(agentgateway): trust ICA1 + public roots for Keycloak DCR TLS [hl-thsf.2]

STOP — Live Validation Gate (before PR2 / Task 3)

Not part of the implementation commits. PR1 (Tasks 1-2) must merge, sync via ArgoCD, and pass every check below before Task 3 begins. Skipping this gate risks silently reintroducing the exact hl-thsf.1 bug (a network deny 403ing agentgateway’s own DCR proxy) — the entire reason this plan is split into two PRs.

AS-metadata discovery still returns 200 with the correctly rewritten registration_endpoint:

curl -s https://mcp-gw.fzymgc.house/.well-known/oauth-authorization-server | jq .registration_endpoint

Primary gate: an actual anonymous POST /register through mcp-gw succeeds end-to-end (RFC 7591 DCR, real client_id returned — not a 403 or a TLS error):
Terminal window
```
curl -si -X POST https://mcp-gw.fzymgc.house/.well-known/oauth-authorization-server/client-registration \
  -H "Content-Type: application/json" \
  -d '{"redirect_uris":["http://localhost:8765/callback"],"token_endpoint_auth_method":"none"}'
```
Expected: 201 Created with a JSON body containing a client_id. A 403 here means the DCR proxy call still isn’t reaching Keycloak (routing regression). A TLS/connection error here means the SSL_CERT_DIR trust bundle isn’t working (Task 2 regression) — check agentgateway pod logs for a TLS handshake failure against keycloak-internal.
Unrelated public-HTTPS agentgateway routes still work — proves the SSL_CERT_DIR bundle didn’t regress public-root trust for other backends (e.g. an LLM backend route through llm-gw.fzymgc.house). Pick any currently-working LLM route and re-confirm it still returns a normal (non-TLS-error) response.

Only proceed to Task 3 once all three pass.

Implementation Tasks — PR2 (re-add the Traefik deny)

Task 3: Restore the Traefik DCR deny (rule + middleware)

Files:

Modify: argocd/app-configs/keycloak/ingress-route.yaml
Step 1: Pre-flight — confirm the tree is current

Run: wc -l argocd/app-configs/keycloak/ingress-route.yaml && rg -c "clients-registrations" argocd/app-configs/keycloak/ingress-route.yaml Expected: 113 lines and 1 match (the explanatory-comment mention, no rule). A different count means the tree is stale — rebase onto main@origin first.

Step 2: Replace the explanatory comment + bare route with the restored deny rule

Replace this exact block (lines 32-41 — the “Anonymous DCR … is NOT denied here” comment and the start of the bare Host(id.fzymgc.house) route through its kind: Rule / priority: 1 lines):

    # Anonymous DCR (/clients-registrations) is NOT denied here. agentgateway's DCR
    # proxy derives its target from the PUBLIC issuer URL (agentgateway v1.3.1
    # mcp/auth.rs client_registration), so it hairpins through this route — a network
    # deny would 403 the gateway's own DCR. Anonymous DCR is instead governed solely by
    # Keycloak's loopback Trusted-Hosts policy (dcr-trusted-hosts-job.yaml). Restoring a
    # cluster-internal-only network posture is tracked in hl-thsf.2. See ADR hl-amad
    # (supersedes hl-xktj). [hl-thsf.1]
    - match: Host(`id.fzymgc.house`)
      kind: Rule
      priority: 1

with:

    # Deny anonymous Dynamic Client Registration on the PUBLIC (LAN-reachable) route.
    # agentgateway's DCR proxy now reaches Keycloak directly, in-cluster, via the
    # keycloak-internal CoreDNS split-horizon rewrite (hl-rs8m) + an ICA1/public-root
    # trust bundle mounted into the agentgateway pod (hl-thsf.2) — it no longer
    # hairpins through this route, so this deny no longer blocks the gateway's own
    # DCR (the bug hl-thsf.1 fixed by removing this same rule). This restores
    # cluster-internal-only DCR per ADR hl-xktj (superseded by hl-amad, both amended
    # by hl-thsf.2's own ADR). Higher priority than the bare Host rule below so it
    # wins for this path.
    - match: Host(`id.fzymgc.house`) && PathPrefix(`/realms/fzymgc/clients-registrations`)
      kind: Rule
      priority: 1000
      middlewares:
        - name: deny-client-registration
      services:
        - name: keycloak-service
          port: 8443
          scheme: https
          serversTransport: keycloak-internal-tls
    - match: Host(`id.fzymgc.house`)
      kind: Rule
      priority: 1

Step 3: Add the deny-client-registration Middleware

After the closing tls: secretName: keycloak-public-tls block and before the admin-allowlist Middleware’s --- separator, insert a new ----separated Middleware document:

---
# Deny-all middleware: ipAllowList of an unreachable range → every real client gets 403.
# Attached only to the /clients-registrations path rule above. (Traefik v3 — ipAllowList.)
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: deny-client-registration
  namespace: keycloak
spec:
  ipAllowList:
    sourceRange:
      - 127.0.0.1/32

Step 4: Render and lint

Run:

kubectl kustomize argocd/app-configs/keycloak > /dev/null && echo KUSTOMIZE_OK
yamllint -c .yamllint.yaml argocd/app-configs/keycloak/ingress-route.yaml

Expected: KUSTOMIZE_OK, no yamllint errors.

Step 5: Confirm the rendered output contains exactly one deny middleware and the priority-1000 rule

Run: kubectl kustomize argocd/app-configs/keycloak | rg -c "name: deny-client-registration" Expected: 2 (one reference in the route’s middlewares: list, one in the Middleware object’s own metadata.name).

Run: kubectl kustomize argocd/app-configs/keycloak | yq 'select(.kind == "IngressRoute") | .spec.routes[] | select(.priority == 1000) | .match' Expected: Host(`id.fzymgc.house`) && PathPrefix(`/realms/fzymgc/clients-registrations`) (backtick-quoted Traefik matcher syntax, rendered as a plain string by yq).

Step 6: Commit

Commit using VCS-appropriate commands per references/vcs-preamble.md. Message: fix(keycloak): restore Traefik DCR deny now that agentgateway's DCR is cluster-internal [hl-thsf.2]

Validation (post-PR2-merge, operator-run)

Not part of the implementation commits. After PR2 merges and ArgoCD syncs the keycloak app:

Re-run the “Primary gate” check from the Live Validation Gate section above — proves the re-added deny does not regress the now-internal DCR path (it should never be on that path at all). Expected: still 201 Created.
Direct LAN POST to id.fzymgc.house/realms/fzymgc/clients-registrations (from a LAN client, not through mcp-gw) is now network-blocked:
Terminal window
```
curl -si -X POST https://id.fzymgc.house/realms/fzymgc/clients-registrations/openid-connect \
  -H "Content-Type: application/json" \
  -d '{"redirect_uris":["http://localhost:8765/callback"]}'
```
Expected: 403 Forbidden (the deny middleware), not a Keycloak response of any kind.

On pass: bd note hl-thsf.2 "Live-validated: PR1 gate + post-PR2 network-block confirmed".

Follow-ups (not in this plan)

New ADR superseding hl-amad, capturing the final A1+trust posture — deferred to the standard capture-adrs step after plan-review, not authored in this plan.
Re-enabling Keycloak-side Trusted-Hosts host-match (dcr-trusted-hosts-job.yaml) — explicitly scoped out of this bead; the Traefik deny is the sole restored control (see spec’s Approaches table).
hl-g3xy — upstream agentgateway DCR/metadata backendRef override, the cleanest long-term fix, unaffected by this bead.