Remove Traefik DCR deny; loopback Trusted-Hosts as sole DCR control
Date: 2026-06-29 Status: Superseded by hl-ll5p Decision: hl-amad Deciders: Sean Brandt
Context
Section titled “Context”ADR hl-xktj ratified a Keycloak Trusted-Hosts client-registration policy keyed on the agentgateway Service ClusterIP (host-match enabled), plus a Traefik deny-client-registration Middleware (ipAllowList 127.0.0.1/32) to keep anonymous DCR cluster-internal. That rested on a false premise: agentgateway v1.3.1 (mcp/auth.rs::client_registration) derives its DCR target from auth.issuer — the PUBLIC issuer URL — so the gateway’s DCR hairpins out through Traefik rather than reaching keycloak-service pod-to-pod. PR #1447 corrected the Keycloak side to a loopback model (host-sending-registration-request-must-match=false, client-uris-must-match=true, trusted-hosts=[localhost,127.0.0.1]) but left the Traefik deny in place — which, under Cilium bpf.masquerade=true, 403s the gateway’s own hairpinned DCR, breaking the Phase 1 real-DCR gate (hl-thsf).
Decision
Section titled “Decision”Delete the Host(id.fzymgc.house) && PathPrefix(/realms/fzymgc/clients-registrations) IngressRoute rule (priority 1000) and the deny-client-registration Middleware from argocd/app-configs/keycloak/ingress-route.yaml. The Keycloak loopback Trusted-Hosts policy — host-sending=false, client-uris-must-match=true, trusted-hosts=[localhost,127.0.0.1] — becomes the SOLE anonymous-DCR control. Restoring a cluster-internal-only network posture (CoreDNS rewrite + ICA1 trust injection into the gateway pod) is deferred to bead hl-thsf.2.
Rationale
Section titled “Rationale”- The Traefik deny was built on a false pod-to-pod premise and is self-defeating: it blocks the legitimate gateway DCR hairpin it was meant to permit.
- Source-IP allowlisting the deny (Approach E) is not viable: Cilium kubeProxyReplacement + bpf.masquerade make the hairpinned source IP indeterminate at Traefik (pod IP vs masqueraded node IP), and a node IP is indistinguishable from LAN.
- Because the loopback model has
host-sending=false, the source IP is irrelevant — G works regardless of SNAT; client-uris-must-match=true confines registered clients to loopback redirect URIs, an adequate application-level control for the homelab threat model. - A1+trust (CoreDNS rewrite + ICA1 bundle in the gateway pod) is the correct long-term fix but needs non-trivial TLS-trust work; deferring it to hl-thsf.2 avoids blocking Phase 1.
Alternatives Considered
Section titled “Alternatives Considered”- Remove the Traefik deny (Approach G, chosen): immediately unblocks DCR via the existing public-ACME-cert hairpin path; no agentgateway/CoreDNS/TLS change; loopback redirect-URI constraint remains as the control.
- Source-IP allowlist on the deny (Approach E, rejected): Cilium masquerade makes the hairpin source IP indeterminate/indistinguishable from LAN — deny stays broken or over-permits.
- CoreDNS rewrite + ICA1 trust injection (A1+trust, deferred to hl-thsf.2): makes DCR truly internal and keeps the network deny, but requires ICA1 CA-bundle injection and AS-metadata re-validation.
- Mock DCR via static clientId (Approach C, rejected): not real DCR (same client_id), fails the gate’s distinct-client_ids requirement.
Consequences
Section titled “Consequences”- Positive: agentgateway-proxied RFC 7591 DCR works (two POST /register via mcp-gw return distinct client_ids); minimal blast radius (no agentgateway/CoreDNS/TLS change).
- Negative: defense-in-depth reduced — LAN-direct DCR against id.fzymgc.house/clients-registrations is now governed only by Keycloak’s loopback application-level policy, not a network-level deny; registered clients are constrained to loopback redirect URIs (not zero surface).
- Neutral: the loopback Trusted-Hosts model was already deployed (#1447); G makes it the effective sole gate. Restoring cluster-internal-only DCR is tracked in hl-thsf.2.
References
Section titled “References”- Supersedes: hl-xktj
- Superseded by: hl-ll5p