Design: Fix Keycloak admin console access — restore internal reachability (SNAT) + close public tunnel-bypass exposure

• Bead: hl-ba3a (reused as design bead per Rule 6; related hl-5g05.14)
• Date: 2026-07-02
• Status: Accepted (design-reviewer READY, round 1)
• Author: Sean (with Claude)
• Related: PR #1525 (hl-5g05.14, merged) added the admin-allowlist Traefik Middleware this spec builds on and partially supersedes

Problem

Two independent, unrelated bugs both manifest as “the Keycloak admin console (id.fzymgc.house/admin, /realms/master) is not correctly gated”:

Bug A — legitimate internal access is wrongly denied. The operator, on their internal network, gets a plain-text Forbidden page (Traefik’s own ipAllowList default-deny response — confirmed via a live screenshot, not a Cloudflare error page) when visiting /admin.

Bug B — public/internet access is not denied at all (pre-existing, hl-ba3a). The Cloudflare tunnel’s id.fzymgc.house ingress rule (tf/cloudflare/tunnel.tf) forwards directly to keycloak-service.keycloak.svc.cluster.local:8443, bypassing Traefik entirely. Internet clients can reach /admin and /realms/master completely unrestricted — the exact exposure PR #1525 (hl-5g05.14) was opened to close, still open because that PR only added a Traefik-side control, and Traefik never sees tunnel-relayed traffic.

These look like one problem (“access control on /admin is wrong”) but are opposite failures in different parts of the request path, and neither fix addresses the other.

Root cause

Bug A: externalTrafficPolicy unset on Traefik’s Service → SNAT

Traefik’s Service (argocd/cluster-app/templates/traefik.yaml, service: block) is type: LoadBalancer with no externalTrafficPolicy set, so it defaults to Cluster. Under Cilium’s kube-proxy replacement, cross-node external traffic to a Cluster-policy LoadBalancer Service gets SNAT’d to an internal address before reaching the backend — so the admin-allowlist Middleware’s ipAllowList (default RemoteAddrStrategy, reads the raw TCP peer, not X-Forwarded-For) evaluates the post-SNAT address, not the real client IP. That address falls outside both allowed ranges (192.168.20.0/24 LAN, 100.64.0.0/10 Tailscale CGNAT), so genuine LAN/Tailscale clients are denied.

This is not a new failure mode in this codebase — argocd/app-configs/dolt/network-policies.yaml:12-17 already documents the identical pattern for a different LoadBalancer service:

“LoadBalancer service dolt (192.168.20.147) with externalTrafficPolicy=Cluster SNATs external traffic to the local node IP, so dolt sees sources as host/remote-node entities rather than the original client CIDR.”

Live evidence (ClickHouse default.otel_logs, Traefik access logs): the /admin request matching the operator’s screenshot logged client_ip = 10.245.2.25 — a cluster pod-range address, not a LAN or Tailscale CIDR — on the admin-allowlist-gated router, status 403.

Bug B: tunnel bypasses Traefik (recap, hl-ba3a original finding)

tf/cloudflare/tunnel.tf’s id.fzymgc.house ingress targets keycloak-service:8443 directly (pre-existing, unchanged by PR #1525). The admin-allowlist Middleware PR #1525 added only governs traffic that actually transits Traefik — Cloudflare-tunnel-relayed requests never do. curl https://id.fzymgc.house/admin/ from any internet host reaches Keycloak directly, ungated.

Also confirmed: tf/cloudflare/access.tf already defines a Keycloak cloudflare_zero_trust_access_identity_provider used as the primary IdP for Cloudflare Access elsewhere in this account (docs, uptime_kuma applications). Gating Keycloak’s own admin console with a Cloudflare Access Application whose allowed_idps includes Keycloak would be circular (you’d need to log into Keycloak to reach the page that lets you log into Keycloak) — any fix for Bug B must use a different IdP.

Approaches considered

Bug A

	Approach	Verdict
A1	Set externalTrafficPolicy: Local on Traefik’s Service	Chosen. Stops the SNAT at the source; matches the exact remedy the dolt comment already names for this pattern. MetalLB’s L2 mode natively supports Local policy — it only announces the VIP (ARP) from nodes with a Ready backend pod, so with Traefik’s 2 replicas this degrades gracefully rather than black-holing traffic. Trade-off: this is a Service-wide setting — it changes traffic handling for every app behind Traefik, not just Keycloak.
A2	Recover the real client IP via X-Forwarded-For / PROXY protocol instead of avoiding the SNAT	Rejected. Cilium’s SNAT operates at L3/L4; it does not add an L7 X-Forwarded-For header, so there is nothing for ipStrategy to read. Enabling PROXY protocol between MetalLB/Cilium and Traefik is unconfigured today and a materially larger change for the same outcome as A1.
A3	Drop IP-based gating for /admin entirely; rely solely on the Bug B fix (Cloudflare Access) for both paths	Rejected. Cloudflare Access only sees traffic that transits Cloudflare’s edge. LAN-direct traffic to the internal MetalLB VIP never does, so this would leave the LAN-direct path completely ungated (any device on the LAN, unauthenticated at the network layer) — a regression from today’s defense-in-depth intent.

Bug B

	Approach	Verdict
B1	Cloudflare Access Application scoped to id.fzymgc.house /admin* + /realms/master*, gated by GitHub + Email-OTP IdPs (excludes Keycloak — avoids circularity)	Chosen. Narrow blast radius — new tf/cloudflare/access.tf resources only, no change to the tunnel or the OIDC back-channel (/realms/fzymgc/*) other apps depend on. Mirrors the existing docs and uptime_kuma Access Applications almost exactly (same file, same IdPs). Also gives the operator a real, network-independent path to /admin (log in via GitHub/OTP) instead of depending on LAN/Tailscale network topology.
B2	Repoint the tunnel through Traefik (tf/cloudflare/tunnel.tf target Traefik’s websecure Service instead of keycloak-service directly)	Rejected for now. Would make the Traefik admin-allowlist govern tunnel traffic too, but tunnel-relayed requests always present the in-cluster cloudflared pod’s IP as RemoteAddr — so it would block all tunnel traffic to /admin, including legitimate remote access, without actually authenticating anyone. Also the highest blast radius of any option: touches the OIDC back-channel Cloudflare Access itself depends on, and re-treads the hl-7una Traefik→Keycloak TLS-hop territory (already stabilized; native rootCAs field, do not revert per the ServersTransport comment in ingress-route.yaml).
B3	NetworkPolicy	Rejected (already ruled out in the original hl-ba3a bead notes — operates on IP/port only, cannot do path-based gating, and cloudflared legitimately needs L3/L4 access to keycloak-service:8443 for the OIDC back-channel regardless).

Design

Fix A — externalTrafficPolicy: Local

Change (argocd/cluster-app/templates/traefik.yaml, Helm valuesObject.service.spec):

service:
  single: true
  type: LoadBalancer
  spec:
    loadBalancerIP: 192.168.20.145
    externalTrafficPolicy: Local
  additionalServices:
    internal:
      type: ClusterIP
      labels:
        traefik-service-label: internal

No change to argocd/app-configs/keycloak/ingress-route.yaml — the admin-allowlist Middleware and its CIDR ranges are correct as merged; they were only ever evaluating the wrong (SNAT’d) address.

Verified against the traefik chart 40.3.0 default values.yaml: service.spec is documented as passed through verbatim to the Kubernetes ServiceSpec (“Cannot contains type, selector or ports entries” — type is set separately at service.type, not relevant here), so externalTrafficPolicy is a valid key at this path. The exact service.spec.externalTrafficPolicy: Local snippet also appears verbatim in the chart’s own EXAMPLES.md.

Data flow after A:

LAN/Tailscale client → MetalLB VIP 192.168.20.145 (node w/ ready Traefik pod)
  → Traefik (RemoteAddr = real client IP, no SNAT)
  → admin-allowlist Middleware evaluates the real IP → allow/deny correctly

Fix B — Cloudflare Access Application

Change (new resources in tf/cloudflare/access.tf, following the existing docs/uptime_kuma pattern):

resource "cloudflare_zero_trust_access_application" "keycloak_admin" {
  account_id       = var.cloudflare_account_id
  name             = "Keycloak Admin Console"
  domain           = "id.fzymgc.house/admin"
  type             = "self_hosted"
  session_duration = "4h"

  destinations = [
    { type = "public", uri = "id.fzymgc.house/admin" },
    { type = "public", uri = "id.fzymgc.house/realms/master" },
  ]

  # Excludes the Keycloak IdP deliberately — gating Keycloak's own admin
  # console with Keycloak-as-IdP is circular.
  allowed_idps = concat(
    cloudflare_zero_trust_access_identity_provider.github[*].id,
    [cloudflare_zero_trust_access_identity_provider.otp.id],
  )

  policies = [{
    id         = cloudflare_zero_trust_access_policy.docs_allow.id
    precedence = 1
  }]

  auto_redirect_to_identity = false
  app_launcher_visible      = false
}

Notes on the fields above (grounded via context7, /cloudflare/terraform-provider-cloudflare):

• destinations is the current mechanism for path-scoped self-hosted apps; self_hosted_domains is deprecated (unsupported as of 2025-11-21) and MUST NOT be used.
• domain is set for App Launcher / UI display purposes; destinations is what Access actually enforces against.
• Reuses the existing docs_allow policy (approved email domains) rather than introducing a new policy — consistent with existing convention. This is a deliberate scope decision: Access here is an outer identity checkpoint, not the sole gate (Keycloak’s own login still follows), so the broader 4-domain policy is acceptable for this repo’s 3-user trust model. Tightening to specific emails is a low-risk follow-up if wanted later.
• session_duration = "4h", shorter than the 24h used for docs and uptime_kuma — this app gates a higher-privilege surface. Adjustable.
• app_launcher_visible = false — this isn’t a normal app users browse to, it’s a security gate on an existing admin path.

Data flow after B:

Internet client → Cloudflare edge → Access check on /admin*, /realms/master*
  (GitHub or Email-OTP login required)
  → allowed → Cloudflare tunnel → keycloak-service:8443 (unchanged, direct)
  denied → Access block page, never reaches keycloak-service

Internet client → /realms/fzymgc/* (OIDC back-channel, all other apps + CF
  Access's own Keycloak IdP calls) → unaffected, no Access gate, unchanged

Fixes A and B are independent and additive

• LAN-direct traffic to the internal VIP never transits Cloudflare, so B cannot gate it — A remains the only control for that path.
• Tunnel-relayed traffic never transits Traefik, so A cannot gate it — B remains the only control for that path.
• Both fixes ship together in this round because they were diagnosed together, but they touch unrelated subsystems (Kubernetes/Helm vs. Cloudflare Terraform) and can land as two separate PRs / two separate HCP Terraform Cloud apply cycles (main-cluster-bootstrap’s ArgoCD sync for A; main-cluster-cloudflare for B).

Validation plan

Fix A:

1. After ArgoCD syncs the Traefik Application, confirm the traefik Service shows externalTrafficPolicy: Local (kubectl get svc -n traefik traefik -o yaml).
2. From a LAN client: curl -I https://id.fzymgc.house/admin/ → expect a Keycloak response (redirect to login), not Forbidden.
3. Spot-check 2-3 other apps behind Traefik still resolve correctly from both LAN and Tailscale clients post-rollout (global setting — confirm no regression from nodes without a ready Traefik pod).
4. Re-run the ClickHouse query from this investigation (RequestHost = 'id.fzymgc.house' AND RequestPath = '/admin') and confirm ClientHost now shows a real LAN/Tailscale CIDR, not a pod-range address.

Fix B:

1. From an external/non-LAN, non-Tailscale vantage point: curl -I https://id.fzymgc.house/admin/ → expect a Cloudflare Access redirect (302 to the Access login page), not a direct Keycloak response.
2. Same vantage point: curl -I https://id.fzymgc.house/realms/fzymgc/.well-known/openid-configuration → expect 200, unaffected by the new Access Application.
3. Manually complete the GitHub or OTP login flow once to confirm the Access Application actually admits an authorized user through to Keycloak’s own login page (not a redirect loop).

Docs: update the admin-allowlist row in docs/reference/network.md (currently states it “fails closed for the entire Cloudflare-tunnel path” — true pre-fix, no longer true once Fix B adds an Access gate on that path instead of a blanket deny) once both fixes are live.

Out of scope / follow-ups

• hl-thsf.2 — restoring cluster-internal-only DCR posture — unrelated subsystem, untouched by this spec.
• Tightening the Fix B Access policy from domain-wide to specific admin email(s), if desired after living with the broader policy.
• Auditing whether any other Traefik-fronted service relies on externalTrafficPolicy: Cluster behavior (e.g. expects to see a node IP, not the real client) before Fix A rolls out — not identified during this investigation, but Fix A is a Service-wide change and worth a quick sweep.