Skip to content
fzymgc-house Cluster Docs

Design: Keycloak DCR 403 fix — remove the self-defeating Traefik deny (G), harden internal later

  • Bead: hl-thsf.1 (child of hl-thsf — Keycloak Phase 1 real-DCR e2e gate)
  • Date: 2026-06-29
  • Status: Draft (pending design-reviewer gate)
  • Author: Sean (with Claude)
  • Supersedes/amends: ADR hl-xktj (anonymous-DCR network posture)
  • Follow-ups: hl-thsf.2 (restore cluster-internal DCR — A1+trust), hl-g3xy (upstream agentgateway DCR backendRef)

Grounding note — read deployed state, not the local working copy. All manifest claims below are grounded against origin/main (ArgoCD targetRevision: HEAD), the deployed truth. A local jj working copy may predate #1430/#1447 and show an older ingress-route.yaml (no deny rule) and an older ClusterIP-allowlist DCR trust model — that is stale, not current. Verify with git show origin/main:<path>.

Problem

Section titled “Problem”

Anonymous RFC 7591 Dynamic Client Registration (DCR) through the gateway fails:

POST https://mcp-gw.fzymgc.house/.well-known/oauth-authorization-server/client-registration
→ 403 Forbidden  (agentgateway responder, CORS headers)

This blocks the Phase 1 gate (hl-thsf) item (b) “two POST /register via mcp-gw return distinct client_ids” and item (c) engram re-auth. Keycloak direct DCR works (distinct client_ids), so the Keycloak side is ready; only the agentgateway proxy path 403s.

Root cause (grounded in agentgateway v1.3.1 source)

Section titled “Root cause (grounded in agentgateway v1.3.1 source)”

Source read at the deployed tag v1.3.1 (chart appVersion 1.3.1, image cr.agentgateway.dev/agentgateway:v1.3.1), files fetched from raw.githubusercontent.com/agentgateway/agentgateway/v1.3.1/crates/agentgateway/src/: mcp/auth.rs, proxy/httpproxy.rs, http/backendtls.rs. (agentgateway is not in deepwiki/Context7; grounding is direct source-read at the tag — see bead grounding notes.)

  1. mcp/auth.rs::client_registration() builds the DCR target from auth.issuer — https://id.fzymgc.house/realms/fzymgc (the public issuer) — and POSTs to {issuer}/clients-registrations/openid-connect. There is no internal-backend override for DCR (unlike jwks.remote.backendRef in mcp-engram.yaml).
  2. The DCR/AS-metadata calls go via PolicyClient::simple_call → inputs.upstream.simple_call, whose TLS trust is http::backendtls::SYSTEM_TRUST (rustls_native_certs::load_native_certs() — public OS roots only). They do not consult the per-Service backend-TLS policy (the ICA1 trust used by the JWKS backendRef path). This matters for the A1+trust follow-up, not for G — under G the DCR hairpin keeps hitting the public ACME cert via Traefik, the same id.fzymgc.house host whose AS-metadata fetch already succeeds today over SYSTEM_TRUST (live-confirmed: AS discovery returns 200), so the system-roots TLS path is proven for this hop.
  3. In-cluster, id.fzymgc.house has no CoreDNS rewrite (coredns-custom.yaml rewrites only mcp-gw/llm-gw), so it resolves via the Firewalla to the Traefik LB VIP 192.168.20.145 — the request hairpins out.
  4. Traefik matches Host(id.fzymgc.house) && PathPrefix(/realms/fzymgc/clients-registrations) (priority 1000, ingress-route.yaml) → middleware deny-client-registration → ipAllowList: { sourceRange: [127.0.0.1/32] }. ipAllowList is allow-by-enumeration: loopback is the only permitted source, so every real caller — including the gateway — gets 403. agentgateway then wraps that 403 with CORS headers (auth.rs ~L354). This is the exact observed symptom.

Disproven en route (by code, not opinion)

Section titled “Disproven en route (by code, not opinion)”
  • mode: Strict is not the cause. is_well_known_endpoint() makes apply_token_validation() skip /.well-known/oauth-authorization-server/*.
  • Host-shared path matching works. handle_mcp_request() matches path.ends_with("client-registration") first; its error path returns 500, not 403.
  • The “DCR is internal (pod→keycloak-service)” belief is false. That came from the direct JWKS backendRef path (Keycloak saw pod IP 10.245.2.149), not the hairpinned DCR path. The Traefik deny was built on this false premise and therefore blocks the legitimate gateway DCR it was meant to permit.

Approaches considered

Section titled “Approaches considered”
ApproachVerdict
ESource-IP allow-list on the deny (allow pod CIDR 10.245.0.0/16)Not pursued (uncertain). Whether Traefik sees the gateway pod IP or a masqueraded node IP for the hairpin is not determinable from config: Cilium runs kubeProxyReplacement=true + bpf.masquerade=true (tunnel/VXLAN). eBPF LB may preserve the pod source IP to a recognized MetalLB service VIP (→ E viable) or masquerade it to a node IP (→ E unworkable, indistinguishable from LAN). We did not confirm empirically (ClickStack timed out; no live DCR to sample). G is preferred because it does not depend on source-IP visibility.
A1+trustCoreDNS rewrite id.fzymgc.house→keycloak-service + inject ICA1 into the gateway pod (AgentgatewayParameters env/volumes)Feasible, deferred to hl-thsf.2. Keeps the network deny; makes DCR truly internal. Cost: combined system+ICA1 CA bundle, verify SSL_CERT_FILE/SSL_CERT_DIR honored, re-validate AS-metadata.
G (chosen now)Remove the Traefik deny rule + middlewareSimplest path to real DCR via the existing public-cert route.
CMock DCR via static clientIdRejected — not real DCR (same client_id), fails the gate’s intent.

Decision: “G now, A1+trust later.”

Design — Approach G

Section titled “Design — Approach G”

Change

Section titled “Change”

In argocd/app-configs/keycloak/ingress-route.yaml (the origin/main 78-line version):

  1. Delete the Host(id.fzymgc.house) && PathPrefix(/realms/fzymgc/clients-registrations) route rule (priority 1000) and the deny-client-registration Middleware. /clients-registrations then falls through to the bare Host(id.fzymgc.house) route → keycloak-service:8443 (existing keycloak-internal-tls ServersTransport, ICA1-trusted).
  2. Correct the false comments in ingress-route.yaml (the “agentgateway proxies DCR to keycloak-service INTERNALLY (pod→pod, bypassing Traefik)” block, lines ~12-19) and the related note in dcr-trusted-hosts-job.yaml:13 (“under Cilium (no SNAT)” — which contradicts the confirmed bpf_masquerade=true config and the root-cause analysis).

No agentgateway, CoreDNS, or TLS-trust change. agentgateway’s DCR hairpin uses the already-working public-ACME-cert path (keycloak-public-tls, validated by system roots); Traefik→Keycloak uses the existing ICA1 ServersTransport.

Data flow after G

Section titled “Data flow after G”
MCP client → mcp-gw (agentgateway) → [proxy] → id.fzymgc.house (public ACME cert)
           → Traefik (no deny) → keycloak-service:8443 (ICA1) → Keycloak DCR → 201 + client_id

Trust model after G (the control that remains)

Section titled “Trust model after G (the control that remains)”

DCR is governed solely by Keycloak’s realm Trusted-Hosts client-registration policy. The deployed policy on origin/main (dcr-trusted-hosts-job.yaml kcadm args; the Job runs on every ArgoCD sync via BeforeHookCreation) is the loopback model:

  • host-sending-registration-request-must-match=false (source-host check off)
  • client-uris-must-match=true
  • trusted-hosts=["localhost","127.0.0.1"]

Effect: because the source-host check is off, the request’s source IP is irrelevant — so G works regardless of whether the hairpin is SNAT’d. A DCR request is accepted only if its redirect URIs are loopback (http://localhost:*, http://127.0.0.1:*) — which MCP PKCE clients use. Non-loopback registrations are rejected by Keycloak.

ADR hl-xktj is already partially stale. #1447 switched this Job from the ClusterIP-allowlist model the ADR ratified to the loopback model above. G removes the Traefik network control too. Both deltas must be folded into the ADR amendment (scheduled post-impl-plan-review via /capture-adrs).

Security trade-off (explicit)

Section titled “Security trade-off (explicit)”

G removes the network-level control that kept anonymous DCR cluster-internal (ADR hl-xktj). After G, LAN-direct DCR against id.fzymgc.house/clients-registrations is no longer network-blocked — it is governed only by the application-level loopback Trusted-Hosts policy above. Marginal added exposure is modest (the primary DCR surface, mcp-gw, is already the intended public entry, and Trusted-Hosts confines registered clients to loopback redirect URIs), but it is a real reduction in defense-in-depth. hl-thsf.2 restores the network-internal posture; this spec amends ADR hl-xktj accordingly.

Validation plan (Phase 1 gate, revised)

Section titled “Validation plan (Phase 1 gate, revised)”

After merge + ArgoCD sync:

  1. (b) Two POST /register via mcp-gw → distinct client_ids (real DCR). ✅ expected.
  2. (c) Full MCP client (Claude Code / engram) DCR + PKCE + list_memory green against id.fzymgc.house.
  3. (a) PRM discovery …/.well-known/oauth-protected-resource/mcp/engram resolves to the Keycloak realm; AS-metadata …/oauth-authorization-server returns 200 (unchanged path).
  4. (d) Unauth /mcp/engram → 401 + WWW-Authenticate.
  5. REVISED expectation: LAN-direct DCR with a loopback redirect URI now succeeds (201) instead of 403; LAN-direct DCR with a non-loopback redirect URI is rejected by Keycloak. (The old gate asserted “LAN-direct → 403”; that assertion is retired by G.)

Out of scope / follow-ups

Section titled “Out of scope / follow-ups”
  • hl-thsf.2 — A1+trust: restore cluster-internal DCR and re-add the network deny.
  • hl-g3xy — upstream agentgateway: DCR/metadata backendRef override (the clean long-term fix).