Keycloak Admin Access Fixes — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use dev-flow:subagent-driven-development (recommended) or dev-flow:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Fix two independent bugs in Keycloak admin-console access control: legitimate LAN/Tailscale clients are wrongly denied (SNAT breaks the IP allowlist), and public internet traffic is completely ungated (the Cloudflare tunnel bypasses the allowlist entirely).

Architecture: Two unrelated, additive config changes in two different subsystems — a one-line Kubernetes Service traffic-policy change (Helm values, synced via ArgoCD) and a new Cloudflare Access Application (Terraform, applied via HCP Terraform Cloud). Neither depends on the other; they can merge and deploy in either order.

Tech Stack: Kubernetes (Traefik Helm chart 40.3.0, MetalLB L2), Terraform (cloudflare provider, workspace main-cluster-cloudflare)

Spec: docs/engineering/specs/2026-07-02-keycloak-admin-access-design.md Design bead: hl-ba3a

Model intent (for plan-to-beads): Tasks 1 and 3 are mechanical, single-file edits with a fully-specified diff (Helm value addition, doc-row replacement) → model:haiku. Task 2 adds a new security-gating resource (Cloudflare Access Application scoping a privileged admin console, choosing which IdPs to trust) with a self-flagged schema discrepancy to reason through → model:sonnet.

File Structure

File	Responsibility
`argocd/cluster-app/templates/traefik.yaml`	ArgoCD `Application` for Traefik; `valuesObject.service.spec` is passed through verbatim to the rendered `Service`. Task 1 adds `externalTrafficPolicy: Local` here.
`tf/cloudflare/access.tf`	Cloudflare Access identity providers, policies, and Applications. Task 2 adds one new `cloudflare_zero_trust_access_application` resource, reusing the existing `github`/`otp` IdPs and `docs_allow` policy already defined in this file.
`docs/reference/network.md`	Middleware/network reference table. Task 3 updates the `admin-allowlist` row description, which currently states the Middleware “fails closed for the entire Cloudflare-tunnel path” — no longer accurate once Task 2 adds an Access gate on that path instead of a blanket deny.

No new files. Both config changes are single-resource additions to existing files, following patterns already established in each.

Task 1: Fix SNAT — `externalTrafficPolicy: Local` on Traefik’s Service

Files:

Modify: argocd/cluster-app/templates/traefik.yaml:66-75
Step 1: Add externalTrafficPolicy: Local to the service spec

Current (lines 66-75):

          service:
            single: true
            type: LoadBalancer
            spec:
              loadBalancerIP: 192.168.20.145
            additionalServices:
              internal:
                type: ClusterIP
                labels:
                  traefik-service-label: internal

Change to:

          service:
            single: true
            type: LoadBalancer
            spec:
              loadBalancerIP: 192.168.20.145
              # Preserve real client source IPs (no SNAT). Required for the
              # Keycloak admin-allowlist Middleware's ipAllowList to see
              # actual LAN/Tailscale client IPs instead of a SNAT'd address
              # — see docs/engineering/specs/2026-07-02-keycloak-admin-access-design.md.
              # MetalLB L2 mode natively supports Local: it only ARP-announces
              # the VIP from nodes with a Ready backend pod, so this degrades
              # gracefully rather than black-holing traffic.
              externalTrafficPolicy: Local
            additionalServices:
              internal:
                type: ClusterIP
                labels:
                  traefik-service-label: internal

Step 2: Verify the chart renders cleanly with this value

Extract just the changed service block into a standalone values file and render it against the pinned chart version to confirm no schema errors and that externalTrafficPolicy: Local appears on the rendered Service:

cat > /tmp/traefik-service-check.yaml <<'EOF'
service:
  single: true
  type: LoadBalancer
  spec:
    loadBalancerIP: 192.168.20.145
    externalTrafficPolicy: Local
  additionalServices:
    internal:
      type: ClusterIP
      labels:
        traefik-service-label: internal
EOF

helm repo add traefik https://traefik.github.io/charts 2>/dev/null
helm repo update traefik
helm template traefik traefik/traefik --version 40.3.0 \
  -f /tmp/traefik-service-check.yaml -n traefik \
  --show-only templates/service.yaml

Expected: a Service manifest with type: LoadBalancer, loadBalancerIP: 192.168.20.145, and externalTrafficPolicy: Local in spec:. No template errors.

Step 3: yamllint the modified file

yamllint argocd/cluster-app/templates/traefik.yaml

Expected: no errors (this repo’s .yamllint.yaml config applies).

Step 4: Commit

Commit using VCS-appropriate commands per references/vcs-preamble.md (jj-first in this repo). Suggested message:

fix(traefik): preserve client source IPs (externalTrafficPolicy: Local)

Traefik's LoadBalancer Service had no externalTrafficPolicy set, so it
defaulted to Cluster and Cilium SNAT'd cross-node external traffic before
it reached any Middleware. This broke the Keycloak admin-allowlist
Middleware (hl-5g05.14 / PR #1525): legitimate LAN/Tailscale clients were
denied because the Middleware evaluated a SNAT'd address, not the real
client IP. [hl-ba3a]

Task 2: Close the public tunnel-bypass — Cloudflare Access Application

Files:

Modify: tf/cloudflare/access.tf (append after line 182, the end of the uptime_kuma resource)
Step 1: Add the Access Application resource

Append to tf/cloudflare/access.tf:

# Keycloak Admin Console Application (hl-ba3a)
# Protects id.fzymgc.house/admin and /realms/master from the Cloudflare
# tunnel path (tf/cloudflare/tunnel.tf routes id.fzymgc.house directly to
# keycloak-service, bypassing Traefik's admin-allowlist Middleware entirely
# for internet-originated traffic). Deliberately excludes the Keycloak IdP
# from allowed_idps — gating Keycloak's own admin console with Keycloak as
# the identity provider is circular.
resource "cloudflare_zero_trust_access_application" "keycloak_admin" {
  account_id       = var.cloudflare_account_id
  name             = "Keycloak Admin Console"
  domain           = "id.fzymgc.house/admin"
  type             = "self_hosted"
  session_duration = "4h"

  destinations = [
    { type = "public", uri = "id.fzymgc.house/admin" },
    { type = "public", uri = "id.fzymgc.house/realms/master" },
  ]

  allowed_idps = concat(
    cloudflare_zero_trust_access_identity_provider.github[*].id,
    [cloudflare_zero_trust_access_identity_provider.otp.id],
  )

  policies = [{
    id         = cloudflare_zero_trust_access_policy.docs_allow.id
    precedence = 1
  }]

  auto_redirect_to_identity = false
  app_launcher_visible      = false
}

Grounding note: the upstream cloudflare_zero_trust_access_application docs (context7, /cloudflare/terraform-provider-cloudflare) list zone_id as required and don’t mention account_id in that resource’s schema excerpt. This conflicts with the existing, already-applied docs and uptime_kuma resources in this same file, which use account_id only (no zone_id) — the working, deployed pattern in this repo. Followed the repo’s proven pattern (account_id) rather than the possibly version-drifted doc excerpt; terraform validate in Step 2 will catch it immediately if this repo’s provider version actually requires zone_id.

Step 2: Format and validate

cd tf/cloudflare
terraform fmt -check
terraform init -backend=false
terraform validate

Expected: terraform fmt -check exits 0 (no diff); terraform validate reports Success! The configuration is valid.

Do not commit the generated .terraform.lock.hcl if terraform init creates or modifies one (repo convention — see tf/CLAUDE.md).

Step 3: Commit

fix(cloudflare): gate Keycloak admin console behind Access (github/otp)

tf/cloudflare/tunnel.tf routes id.fzymgc.house directly to
keycloak-service, bypassing Traefik's admin-allowlist Middleware for all
internet traffic — /admin and /realms/master were completely ungated.
Adds a path-scoped Access Application using GitHub + Email-OTP identity
providers (not Keycloak — circular) to close that gap and give a working,
network-independent path to the admin console. [hl-ba3a]

Task 3: Update network reference docs

Files:

Modify: docs/reference/network.md:161
Step 1: Update the admin-allowlist row

Current row (line 161):

| admin-allowlist | keycloak | IPAllowList | Restricts `id.fzymgc.house` `/admin` + `/realms/master` to `192.168.20.0/24` (LAN) + `100.64.0.0/10` (Tailscale CGNAT, best-effort). Uses default `RemoteAddrStrategy`, so it fails closed for the entire Cloudflare-tunnel path regardless of claimed client origin — see `argocd/app-configs/keycloak/ingress-route.yaml` |

Replace with:

| admin-allowlist | keycloak | IPAllowList | Restricts `id.fzymgc.house` `/admin` + `/realms/master` to `192.168.20.0/24` (LAN) + `100.64.0.0/10` (Tailscale CGNAT) for traffic that reaches Traefik directly. Uses default `RemoteAddrStrategy`; Traefik's Service runs `externalTrafficPolicy: Local` so this Middleware sees real client IPs, not a SNAT'd address (hl-ba3a). The Cloudflare-tunnel path never reaches this Middleware at all — it's gated separately by a Cloudflare Access Application (`tf/cloudflare/access.tf`, `keycloak_admin`) — see `argocd/app-configs/keycloak/ingress-route.yaml` |

Step 2: Lint the doc

rumdl check docs/reference/network.md

Expected: no new errors.

Step 3: Commit

docs(network): update admin-allowlist row for hl-ba3a fixes

Traefik's Service now runs externalTrafficPolicy: Local (real client IPs,
no SNAT) and a Cloudflare Access Application gates the tunnel path
separately — the old row's "fails closed for the entire tunnel path"
description is no longer accurate. [hl-ba3a]

Task Dependencies

Tasks 1 and 2 are fully independent (different subsystems, different HCP Terraform Cloud workspaces — main-cluster-bootstrap’s ArgoCD sync for Task 1, main-cluster-cloudflare for Task 2) and can be implemented, reviewed, and merged in either order or in parallel. Task 3 references the end state of both and should land last (after 1 and 2 are merged and deployed), though the doc edit itself has no code dependency and can be drafted anytime.

Post-merge verification (not automatable in an isolated worktree — requires live cluster/DNS access)

See the spec’s Validation plan section (docs/engineering/specs/2026-07-02-keycloak-admin-access-design.md) for the full manual verification steps once both fixes are deployed: kubectl get svc check, LAN curl to /admin, external-vantage-point curl to confirm the Access redirect, and a one-time manual GitHub/OTP login to confirm the Access Application doesn’t redirect-loop.

Keycloak Admin Access Fixes — Implementation Plan

File Structure

Task 1: Fix SNAT — externalTrafficPolicy: Local on Traefik’s Service

Task 2: Close the public tunnel-bypass — Cloudflare Access Application

Task 3: Update network reference docs

Task Dependencies

Post-merge verification (not automatable in an isolated worktree — requires live cluster/DNS access)

Task 1: Fix SNAT — `externalTrafficPolicy: Local` on Traefik’s Service