Design: Harden Keycloak DCR to cluster-internal (A1+trust, restore ADR hl-xktj posture)

• Bead: hl-thsf.2 (follow-up to hl-thsf.1, child of hl-thsf)
• Date: 2026-07-02
• Status: Draft (pending design-reviewer gate)
• Author: Sean (with Claude)
• Supersedes/amends: ADR hl-amad (which itself deferred this work; hl-amad superseded hl-xktj)
• Predecessor: hl-thsf.1 / spec 2026-06-29-keycloak-dcr-traefik-deny-fix-design.md (“G now, A1+trust later”)
• Depends on (already merged, unrelated bead): hl-rs8m / PR #1538 — added the id.fzymgc.house CoreDNS split-horizon rewrite this design originally proposed to add itself. See “Revision note” below.

Grounding note — read deployed state, not the local working copy. All manifest and cluster claims below are grounded against origin/main (ArgoCD targetRevision: HEAD) and the live cluster (kubernetes-mcp-server, read-only), not a possibly-stale local checkout. This mattered concretely during this design’s own review — see below.

Revision note (round 1 → round 2). The first draft of this spec proposed adding a CoreDNS rewrite id.fzymgc.house → keycloak-service.keycloak.svc.cluster.local. While that draft was being reviewed, an unrelated bead (hl-rs8m, fixing HCP Terraform Cloud’s access to Keycloak’s master-realm token endpoint) merged the same kind of rewrite for the same hostname — but pointed at a new keycloak-internal Service (443→8443 shim), not keycloak-service directly, because keycloak-service (operator-managed) only exposes 8443/9000 and a bare https://id.fzymgc.house (no explicit port, exactly agentgateway’s DCR call shape) defaults to port 443 and would time out against it. The design-reviewer caught the resulting staleness (this session’s local jj workspace still showed the pre-hl-rs8m file) before implementation started. This revision removes the now-redundant/wrong CoreDNS step entirely and reuses hl-rs8m’s already-merged rewrite as a precondition.

Problem

hl-thsf.1 (Approach G) removed the Traefik deny-client-registration Middleware on id.fzymgc.house to unblock agentgateway’s own DCR proxy, which hairpins through the public issuer URL (https://id.fzymgc.house/realms/fzymgc). That fixed the Phase 1 gate but left anonymous DCR governed only by Keycloak’s loopback Trusted-Hosts policy (host-sending-registration-request-must-match=false) — no network-level control. ADR hl-amad accepted this as an explicit, temporary trade-off and deferred restoring the cluster-internal-only posture to this bead.

Goal: make agentgateway’s DCR proxy call Keycloak directly, in-cluster (A1) and trust its ICA1-issued cert for that call (trust), so the Traefik network deny can be re-added without reintroducing the hl-thsf.1 403. As of hl-rs8m (merged, unrelated bead), the routing half (A1) already exists — this bead’s actual scope is narrower than originally scoped: just the ICA1 trust injection into agentgateway, plus re-adding the Traefik deny.

Grounding (source-verified, not assumed)

• agentgateway v1.3.1 (crates/agentgateway/src/mcp/auth.rs::client_registration, crates/agentgateway/src/http/backendtls.rs, source fetched at the deployed tag): DCR’s proxy call uses .simple_call(ureq), whose TLS trust is SYSTEM_TRUST — a process-lifetime Lazy<BackendTLS> built once from rustls_native_certs::load_native_certs() with no arguments. It does not consult agentgateway’s per-Service backend-TLS policy mechanism (the one already used for the JWKS fetch via keycloak-service-backend-tls).
• rustls-native-certs 0.8.4 (exact version pinned in agentgateway v1.3.1’s Cargo.lock; source read at tag v/0.8.4): load_native_certs() honors SSL_CERT_FILE/SSL_CERT_DIR, but “if either (or both) are set, certificates are only loaded from the locations specified via environment variables and not the platform-native certificate store.” This is a replace, not append. A bundle containing the full public root set and ICA1 is therefore mandatory — setting either env var to an ICA1-only file would silently break trust for every other public-HTTPS call agentgateway’s data plane makes (LLM backends, etc.).
• rustls-native-certs 0.8.4 load_pem_certs_from_dir() (read directly from the pinned-tag source, not context7/deepwiki — this crate isn’t indexed there at this function-body level): SSL_CERT_DIR scans every plain file in the directory (fs::read_dir + metadata.is_file()) and parses all PEM blocks in each — no OpenSSL c_rehash hash-symlink naming required. A directory containing two ordinary ConfigMap-mounted files works. (Mitigation if this reading is somehow wrong: the PR1 live-validation gate below fails fast and loudly rather than silently.)
• AgentgatewayParameters CRD (agentgateway-crds chart 1.3.1, live schema read via resources_get): spec.deployment.spec is an opaque Strategic-Merge-Patch overlay onto the generated Deployment PodSpec (containers/volumes merge by name) — not a narrowly-named extraVolumes field. Arbitrary volumes/volumeMounts are supported through this overlay; the top-level env: field (already used for TZ/ADMIN_ADDR/OTLP vars in parameters.yaml) covers the env-var side.
• agentgateway Dockerfile (v1.3.1): the runtime stage is FROM cgr.dev/chainguard/glibc-dynamic, ENTRYPOINT ["/app/agentgateway"] — no shell, no package manager, no explicit ca-certificates install. Confirmed live: the running pod (agentgateway ns) has readOnlyRootFilesystem: true, runAsNonRoot: true, all capabilities dropped. This rules out an init-container copy-cat-from-the-runtime-image approach (nothing to exec cat/cp with, and no confirmed on-disk bundle to copy from) and motivates vendoring a maintained public CA bundle as a new artifact instead of trying to extract one at runtime.
• Already-merged CoreDNS rewrite (hl-rs8m / PR #1538, origin/main, current): argocd/app-configs/coredns-custom/coredns-custom.yaml already contains rewrite name exact id.fzymgc.house keycloak-internal.keycloak.svc.cluster.local, in the same fzymgc.house server block as the pre-existing llm-gw/mcp-gw rewrites. keycloak-internal (argocd/app-configs/keycloak/internal-service.yaml) is a small, non-operator-managed Service forwarding 443 → 8443 on the same pod selector as keycloak-service — added specifically because a bare https://id.fzymgc.house (no explicit port, exactly how agentgateway’s auth.issuer-derived DCR/AS-metadata URLs are shaped) defaults to port 443, which keycloak-service doesn’t serve. This bead adds no CoreDNS change — the routing precondition already exists and already targets the right port-443 shim.
• Already-merged analogous CA-trust pattern (hl-rs8m / PR #1538): the HCP Terraform Cloud agent pod (argocd/app-configs/hcp-terraform-operator/agent-pool.yaml) solves the identical “trust ICA1 for a call to id.fzymgc.house” problem via an ExternalSecret (ca-trust-secret.yaml, sourcing the same Vault path as this design’s ICA1 needs) plus a same-image initContainer (hashicorp/tfc-agent:1.28.12, which has a shell and update-ca-certificates) that rebuilds /etc/ssl/certs into a shared emptyDir. Not directly reusable here: agentgateway’s runtime image has no shell (see Dockerfile finding above), so there is no in-image tooling to run such an initContainer against. This design instead reuses the already-replicated fzymgc-ica1-ca ConfigMap directly (see next bullet) rather than the TFC pattern’s fresh ExternalSecret — fzymgc-ica1-ca is already proven correct for validating this exact same Keycloak cert from agentgateway specifically (the JWKS backend-TLS path), so no new secret-sourcing is needed.
• Existing ICA1 artifact: fzymgc-ica1-ca (ConfigMap, key ca.crt) is already replicated into every namespace by the mittwald replicator (argocd/app-configs/shared-resources/fzymgc-certificates.yaml) and already consumed by Traefik’s keycloak-internal-tls ServersTransport and agentgateway’s keycloak-service-backend-tls policy. Reused as-is here — no changes to that ConfigMap or its replication.

Approaches considered

	Approach	Verdict
Init-container copy-cat (same-image, à la hl-rs8m’s TFC pattern)	Rebuild /etc/ssl/certs in a same-image initContainer with ICA1 added	Rejected. Proven pattern elsewhere in this repo, but agentgateway’s runtime image (chainguard/glibc-dynamic) has no shell/package manager — there is no in-image tooling to run update-ca-certificates or even cat against. Would need a second, different image, which then can’t see the first image’s actual filesystem without extracting it at build time — a build-time concern, not a runtime one.
SSL_CERT_FILE, single authored bundle	Concatenate a vendored public bundle + ICA1 into one file, commit as a ConfigMap	Workable, but adds a manual “concatenate and keep in sync” step every time either input changes.
SSL_CERT_DIR, two-file directory (chosen)	Mount the existing fzymgc-ica1-ca ConfigMap and a new vendored public-root ConfigMap into the same directory via a projected volume	Chosen. No new maintenance on the ICA1 side (reused unchanged); the public bundle is vendored independently and refreshed on its own cadence; confirmed workable directly from the crate’s directory-scan source (no hash-symlink requirement).
Re-enable Keycloak host-match Trusted-Hosts	Additionally tighten dcr-trusted-hosts-job.yaml so Keycloak itself checks the source host	Out of scope for this bead (explicit scoping decision — see below). Traefik’s re-added network deny is the sole cluster-internal control; Keycloak’s loopback client-URI policy is unchanged.
Single PR (trust bundle + deny together)	Ship both remaining changes in one PR, validate before merge	Rejected. Repeats the exact shape of mistake hl-thsf.1 fixed — the deny must not go back in until the trust bundle is proven live-working post-deploy, not just pre-merge.
Two-PR sequence (chosen)	PR1: ICA1 trust bundle for agentgateway, merge, live-validate. PR2: re-add Traefik deny, merge, re-validate.	Chosen. Matches this repo’s existing live-gate pattern (hl-thsf/hl-thsf.1) and gives a real post-deploy checkpoint between the two changes.

Design

PR1 — ICA1 trust bundle for agentgateway (CoreDNS precondition already met)

No CoreDNS change. The id.fzymgc.house → keycloak-internal rewrite already exists (hl-rs8m, see Grounding) and already resolves agentgateway’s issuer-derived, no-explicit-port DCR/AS-metadata calls to the right port-443 shim in-cluster.

New file argocd/app-configs/agentgateway/public-ca-bundle.yaml — a ConfigMap named agentgateway-public-ca-bundle holding curl’s vendored cacert.pem (the actively-maintained PEM export of Mozilla’s trusted root list), checked into the repo.

argocd/app-configs/agentgateway/parameters.yaml (AgentgatewayParameters CR) — via the spec.deployment.spec Strategic-Merge-Patch overlay:

• A projected volume named ca-trust combining the new agentgateway-public-ca-bundle ConfigMap and the existing fzymgc-ica1-ca ConfigMap into one directory.
• A matching volumeMounts entry on the agentgateway container: ca-trust → /etc/agentgateway/ca-trust.
• env: SSL_CERT_DIR=/etc/agentgateway/ca-trust (appended to the existing env: list alongside TZ/ADMIN_ADDR/OTLP vars).

Data flow after PR1

agentgateway (DCR proxy, SYSTEM_TRUST reads SSL_CERT_DIR)
  → id.fzymgc.house resolves via the existing hl-rs8m CoreDNS rewrite
  → keycloak-internal ClusterIP:443 (shim, hl-rs8m) → pod:8443
  → TLS validated against {public roots + ICA1} → Keycloak clients-registrations
  → 201 + client_id

No Traefik hop. Traefik’s keycloak-internal-tls path (used by browser/LAN traffic to id.fzymgc.house) is untouched by this change. The keycloak-internal Service targets the same pods as keycloak-service, so it presents the identical ICA1-issued cert (SANs already include id.fzymgc.house, per hl-thsf.1’s own grounding) — only the trust anchor is missing on agentgateway’s side, which PR1 supplies.

PR2 — re-add the Traefik deny

argocd/app-configs/keycloak/ingress-route.yaml — restore the exact rule shape hl-thsf.1 removed (recorded in its own spec’s root-cause section): a route rule Host(id.fzymgc.house) && PathPrefix(/realms/fzymgc/clients-registrations) at priority 1000, middleware deny-client-registration with ipAllowList: { sourceRange: [127.0.0.1/32] } (allow-by-enumeration of loopback only → deny-all in practice). Remove the current inline comment (lines ~32-38) documenting why it’s absent — replace with a comment explaining the restored posture and pointing at this spec / the new ADR.

Trust model after PR2 (the control that’s restored)

Cluster-internal-only DCR is enforced by the Traefik network deny alone — LAN and external callers can no longer reach id.fzymgc.house/realms/fzymgc/clients-registrations at the network level, full stop. Keycloak’s app-level Trusted-Hosts policy is unchanged from its current loopback model (host-sending-registration-request-must-match=false, client-uris-must-match=true, trusted-hosts=[localhost,127.0.0.1]) — it remains a secondary, redirect-URI-scoped control, not the primary one. Re-enabling Keycloak-side host-match was considered and explicitly scoped out of this bead (see Approaches table) to keep the change focused; it can be revisited separately if defense-in-depth beyond the network deny is wanted later.

Security trade-off (explicit)

This restores the network-level control ADR hl-xktj originally wanted, closing the gap ADR hl-amad explicitly accepted as temporary. Residual exposure after this bead: none new — DCR is cluster-internal at the network layer again, same as the original hl-xktj intent, achieved via a different mechanism (the hl-rs8m CoreDNS rewrite + ICA1 pod trust instead of a Keycloak-side ClusterIP-allowlist).

Validation plan

After PR1 merge + ArgoCD sync (gate before PR2 may proceed):

1. Sanity-check the (already-existing, hl-rs8m) CoreDNS rewrite is still in place and healthy (CoreDNS logs/metrics — no new debug pods, no behavior change expected here).
2. AS-metadata discovery (GET https://mcp-gw.fzymgc.house/.well-known/oauth-authorization-server) still returns 200 with the correctly rewritten registration_endpoint.
3. Primary gate: an actual anonymous POST /register through mcp-gw succeeds end-to-end — proves the DCR proxy reaches Keycloak directly via keycloak-internal and TLS validates against the new SSL_CERT_DIR bundle, without any Traefik hop.
4. LLM-backend / other public-HTTPS agentgateway routes still work (proves the SSL_CERT_DIR bundle didn’t regress public-root trust for unrelated backends).

After PR2 merge + ArgoCD sync:

5. Re-run gate 3 above — proves the re-added Traefik deny does not regress the now-internal DCR path (it shouldn’t be on that path at all).
6. Direct LAN POST to id.fzymgc.house/realms/fzymgc/clients-registrations is now network-blocked (403 from the Traefik deny, before reaching Keycloak).

Out of scope / follow-ups

• The CoreDNS rewrite and keycloak-internal Service — already delivered by hl-rs8m, not authored by this bead.
• Re-enabling Keycloak-side Trusted-Hosts host-match (dcr-trusted-hosts-job.yaml) — explicitly scoped out; the Traefik deny is the sole restored control.
• hl-g3xy (pre-existing follow-up, from the hl-thsf.1 spec) — upstream agentgateway DCR/metadata backendRef override, the cleanest long-term fix, unaffected by this bead.
• New ADR superseding hl-amad, capturing the final A1+trust posture — deferred to the standard capture-adrs step after plan-review, not authored in this spec.