Keycloak Migration — Phase 0+1 Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use dev-flow:subagent-driven-development (recommended) or dev-flow:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Stand up Keycloak (Operator + CNPG) at id.fzymgc.house and cut the agentgateway MCP/engram path over to it, delivering real RFC 7591 Dynamic Client Registration — the first shippable increment of the Authentik→Keycloak migration.

Architecture: Keycloak via the official Operator (raw keycloak-k8s-resources manifests, k8s.keycloak.org/v2) on a dedicated CNPG Postgres cluster, ArgoCD-managed, realm config in a new tf/keycloak Terraform module (HCP Terraform). Phase 1 repoints the existing mcp-engram-oauth AgentgatewayPolicy and engram’s issuer from Authentik to the Keycloak realm, and enables the agentgateway-proxied anonymous DCR via Keycloak’s Trusted Hosts client-registration policy. Authentik stays fully live throughout — this is purely additive until the Phase 1 cutover, which is a single-policy revert if needed.

Tech Stack: Keycloak Operator (k8s.keycloak.org/v2alpha1), CloudNativePG, ArgoCD, Traefik IngressRoute, cert-manager, External Secrets Operator, Vault, HCP Terraform + keycloak/terraform-provider-keycloak, agentgateway (agentgateway.dev/v1alpha1), Gateway API.

Spec of record: docs/engineering/specs/2026-06-28-keycloak-migration-design.md (design bead hl-jb6q).

GitOps & Validation Conventions (read first)

This repo is GitOps — you do not kubectl apply manifests or run terraform apply locally (CLAUDE.md). The apply boundary:

Kubernetes manifests under argocd/ are synced by ArgoCD after merge to main.
tf/keycloak is planned/applied by HCP Terraform on PR merge (workspace main-cluster-keycloak).

Therefore each task splits into pre-merge work (author + local validation + commit) and post-merge verification (after ArgoCD/HCP sync). Local validation tools used throughout:

# kustomize build (catches structural errors before ArgoCD sees them)
kubectl --context fzymgc-house kustomize argocd/app-configs/<dir>
# client-side dry-run (read-only context is fine; never use --dry-run=server writes)
kubectl --context fzymgc-house apply --dry-run=client -k argocd/app-configs/<dir>
# terraform local checks (no apply)
cd tf/keycloak && terraform fmt -check && terraform validate

The Kubernetes MCP is read-only — use it for post-sync state verification (pods_list, resources_get), never mutation. Commit with the commit-commands:commit skill; one logical change per commit; AI authorship byline.

File structure created by this plan:

Path	Responsibility
`argocd/app-configs/cnpg/db-keycloak.yaml`	Keycloak DB credential ExternalSecret + CNPG `Database`
`argocd/app-configs/cnpg/pooler-keycloak.yaml`	CNPG `Pooler` (rw) for Keycloak
`argocd/cluster-app/templates/keycloak-operator.yaml`	ArgoCD Application: Keycloak Operator
`argocd/app-configs/keycloak-operator/kustomization.yaml`	Operator CRDs + deployment (remote `keycloak-k8s-resources`) + ns patch
`argocd/cluster-app/templates/keycloak.yaml`	ArgoCD Application: Keycloak instance + config
`argocd/app-configs/keycloak/{namespace,kustomization,secrets,certificate,ingress-route,keycloak-cr,keycloak-jwks-referencegrant}.yaml`	Keycloak instance, secrets, TLS, exposure, cross-ns JWKS grant
`tf/keycloak/{versions,terraform,variables,data-sources,main,groups,outputs}.tf` + `README.md`	Realm + provider/backend scaffolding
`tf/keycloak/mcp_public.tf`	`mcp-public` PKCE client + audience mapper
`tf/vault/policy-keycloak.tf`, `tf/vault/jwt-hcp-terraform.tf` (modify)	Vault policy + TFC JWT role for the new workspace
Modify: `argocd/app-configs/agentgateway/mcp-engram.yaml`, `argocd/cluster-app/templates/agent-memory.yaml`	Phase 1 cutover (mcp-engram.yaml is already in the agentgateway kustomization — no kustomization edit needed)

Phase 0 — Foundation

Task 1: Vault secrets + HCP Terraform workspace prerequisites

Files:

Create: tf/vault/policy-keycloak.tf
Modify: tf/vault/jwt-hcp-terraform.tf (add the tfc-keycloak run role)

These are the out-of-band prerequisites: Vault KV material for the DB role, the Keycloak admin bootstrap, and the Terraform provider token; plus the HCP workspace and its Vault JWT role. Vault KV values are written by an operator (not committed); the policy and JWT role are Terraform.

Step 1: Confirm the Vault paths don’t exist yet

Run (via the NAS/cluster Vault access pattern — see docs/operations/vault.md):

vault kv get secret/fzymgc-house/cluster/postgres/users/main-keycloak ; echo "rc=$?"
vault kv get secret/fzymgc-house/cluster/keycloak ; echo "rc=$?"

Expected: both error No value found (rc≠0) — confirms clean slate.

Step 2: Write the Vault KV material (operator action; values are secret, not committed)

# DB role credentials (consumed by CNPG ExternalSecret + the keycloak provider DB)
vault kv put secret/fzymgc-house/cluster/postgres/users/main-keycloak \
  username=keycloak password="$(openssl rand -base64 24)"
# Keycloak bootstrap admin + the terraform admin-cli password the keycloak provider uses
vault kv put secret/fzymgc-house/cluster/keycloak \
  admin_username=kc-admin admin_password="$(openssl rand -base64 24)" \
  terraform_username=tf-admin terraform_password="$(openssl rand -base64 24)"

Expected: Success! Data written for both.

Step 3: Write the Vault policy granting read on the new paths

tf/vault/policy-keycloak.tf:

# Read policy for the Keycloak HCP Terraform workspace + in-cluster ExternalSecrets.
# Name follows the repo convention `terraform-<svc>-admin` (cf. terraform-authentik-admin).
resource "vault_policy" "keycloak" {
  name   = "terraform-keycloak-admin"
  policy = <<-EOT
    path "secret/data/fzymgc-house/cluster/keycloak" {
      capabilities = ["read"]
    }
    path "secret/data/fzymgc-house/cluster/postgres/users/main-keycloak" {
      capabilities = ["read"]
    }
  EOT
}

Step 4: Add the TFC JWT run role for the keycloak workspace

In tf/vault/jwt-hcp-terraform.tf, mirror the existing tfc_authentik role block exactly (verified against the file — bound_claims_type = "glob", user_claim = "terraform_workspace_name", project:main-cluster, token_ttl = 3600, token_max_ttl = 7200, and the terraform-<svc>-admin policy name are all load-bearing):

resource "vault_jwt_auth_backend_role" "tfc_keycloak" {
  backend   = vault_jwt_auth_backend.hcp_terraform.path
  role_name = "tfc-keycloak"

  bound_audiences = ["vault.workload.identity"]
  bound_claims = {
    sub = "organization:fzymgc-house:project:main-cluster:workspace:main-cluster-keycloak:run_phase:*"
  }
  bound_claims_type = "glob" # Enable wildcard matching for run_phase:*

  user_claim     = "terraform_workspace_name"
  role_type      = "jwt"
  token_ttl      = 3600 # 1 hour for long-running applies
  token_max_ttl  = 7200 # 2 hour max for complex infrastructure changes
  token_policies = ["terraform-keycloak-admin"]
}

Step 5: Validate the Terraform

Run:

cd tf/vault && terraform fmt -check && terraform validate

Expected: Success! The configuration is valid.

Step 6: Create the HCP Terraform workspace main-cluster-keycloak

Per docs/operations/hcp-terraform.md: VCS-backed workspace, working directory tf/keycloak, execution mode Agent, env vars TFC_VAULT_PROVIDER_AUTH=true, TFC_VAULT_ADDR, TFC_VAULT_AUTH_PATH=jwt-hcp-terraform, TFC_VAULT_RUN_ROLE=tfc-keycloak, tags main-cluster,keycloak. (Console/API action — record completion in the bead.)

Step 7: Commit

Commit tf/vault/* via commit-commands:commit. Message: feat(keycloak): Vault policy + TFC JWT role for keycloak workspace [hl-jb6q].

Task 2: CNPG database + pooler for Keycloak

Files:

Create: argocd/app-configs/cnpg/db-keycloak.yaml
Create: argocd/app-configs/cnpg/pooler-keycloak.yaml
Modify: argocd/app-configs/cnpg/kustomization.yaml (add the two files to resources)

Mirrors db-authentik.yaml / pooler-authentik.yaml against the existing CNPG Cluster named main in namespace postgres.

Step 1: Confirm the DB doesn’t exist (read-only MCP)

Use Kubernetes MCP resources_get for postgresql.cnpg.io/v1 Database keycloak in ns postgres. Expected: NotFound.

Step 2: Author the credential ExternalSecret + Database

argocd/app-configs/cnpg/db-keycloak.yaml:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: main-keycloak-credentials
  namespace: postgres
spec:
  refreshPolicy: Periodic
  refreshInterval: 5m
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: main-keycloak-credentials
    creationPolicy: Owner
    deletionPolicy: Delete
    template:
      metadata:
        labels:
          cnpg.io/reload: "true"
      type: kubernetes.io/basic-auth
      data:
        username: "{{ .username }}"
        password: "{{ .password }}"
  data:
    - secretKey: username
      remoteRef:
        key: fzymgc-house/cluster/postgres/users/main-keycloak
        property: username
    - secretKey: password
      remoteRef:
        key: fzymgc-house/cluster/postgres/users/main-keycloak
        property: password
---
apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata:
  name: keycloak
  namespace: postgres
spec:
  cluster:
    name: main
  ensure: present
  name: keycloak
  owner: keycloak

Step 3: Author the rw Pooler

argocd/app-configs/cnpg/pooler-keycloak.yaml (mirror pooler-authentik.yaml, session mode, client_tls_sslmode: require):

apiVersion: postgresql.cnpg.io/v1
kind: Pooler
metadata:
  name: keycloak-pooler-rw
  namespace: postgres
spec:
  cluster:
    name: main
  instances: 2
  type: rw
  pgbouncer:
    poolMode: session
    parameters:
      max_client_conn: "200"
      default_pool_size: "25"
      client_tls_sslmode: "require"

Step 4: Add both files to the cnpg kustomization

Append to the resources: list in argocd/app-configs/cnpg/kustomization.yaml:

  - db-keycloak.yaml
  - pooler-keycloak.yaml

Step 5: Validate locally

Run:

kubectl --context fzymgc-house kustomize argocd/app-configs/cnpg | \
  grep -E 'name: (keycloak|main-keycloak-credentials|keycloak-pooler-rw)'

Expected: the three names print — the kustomization includes them and builds cleanly.

Step 6: Commit

feat(keycloak): CNPG database + rw pooler [hl-jb6q].

Step 7: Post-merge verification

After PR merge + ArgoCD sync, via Kubernetes MCP: Database keycloak shows status.applied: true; Pooler service keycloak-pooler-rw.postgres.svc.cluster.local exists; secret main-keycloak-credentials populated.

Task 3: Install the Keycloak Operator

Files:

Create: argocd/cluster-app/templates/keycloak-operator.yaml
Create: argocd/app-configs/keycloak-operator/kustomization.yaml

The Keycloak Operator ships as raw manifests (no Helm chart). Kustomize pulls them as remote resources so ArgoCD can sync them. Pin KC_VERSION to the latest GA (verify before committing — see step 1).

Step 1: Determine the latest GA Keycloak version

Run:

curl -fsS https://api.github.com/repos/keycloak/keycloak-k8s-resources/releases/latest | \
  grep -m1 '"tag_name"'

Expected: a tag like "tag_name": "26.x.y". Use that value as <KC_VERSION> below.

Step 2: Author the operator kustomization (remote CRDs + deployment + ns patch)

argocd/app-configs/keycloak-operator/kustomization.yaml (replace <KC_VERSION>):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: keycloak
resources:
  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/<KC_VERSION>/kubernetes/keycloaks.k8s.keycloak.org-v1.yml
  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/<KC_VERSION>/kubernetes/keycloakrealmimports.k8s.keycloak.org-v1.yml
  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/<KC_VERSION>/kubernetes/kubernetes.yml
patches:
  # The upstream ClusterRoleBinding hard-codes the 'keycloak' subject namespace;
  # we deploy into 'keycloak' so this is a no-op assertion, but pin it explicitly
  # so a future namespace change can't silently break operator RBAC.
  - target:
      kind: ClusterRoleBinding
      name: keycloak-operator-clusterrole-binding
    patch: |
      - op: replace
        path: /subjects/0/namespace
        value: keycloak

Step 3: Author the ArgoCD Application

argocd/cluster-app/templates/keycloak-operator.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: keycloak-operator
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  project: auth
  source:
    repoURL: https://github.com/fzymgc-house/selfhosted-cluster
    targetRevision: HEAD
    path: argocd/app-configs/keycloak-operator
  destination:
    server: https://kubernetes.default.svc
    namespace: keycloak
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
      - CreateNamespace=true

(Confirm project: auth exists — it’s the project Authentik uses; reuse it.)

Step 4: Validate locally

Run:

kubectl --context fzymgc-house kustomize argocd/app-configs/keycloak-operator | \
  grep -E 'kind: (CustomResourceDefinition|Deployment|ClusterRoleBinding)' | sort -u

Expected: all three kinds present (CRDs + operator Deployment + RBAC) — remote resources resolve.

Step 5: Commit

feat(keycloak): install Keycloak Operator via ArgoCD [hl-jb6q].

Step 6: Post-merge verification

After sync: kubectl get crd keycloaks.k8s.keycloak.org keycloakrealmimports.k8s.keycloak.org both exist (via read-only MCP resources_get); operator Deployment in ns keycloak is Available.

Task 4: Keycloak instance — namespace, secrets, TLS, ingress, CR

Files:

Create: argocd/cluster-app/templates/keycloak.yaml (ArgoCD Application)
Create: argocd/app-configs/keycloak/namespace.yaml
Create: argocd/app-configs/keycloak/secrets.yaml
Create: argocd/app-configs/keycloak/certificate.yaml
Create: argocd/app-configs/keycloak/ingress-route.yaml
Create: argocd/app-configs/keycloak/keycloak-cr.yaml
Create: argocd/app-configs/keycloak/kustomization.yaml
Step 1: Author namespace + kustomization

argocd/app-configs/keycloak/namespace.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: keycloak

argocd/app-configs/keycloak/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: keycloak
resources:
  - namespace.yaml
  - secrets.yaml
  - certificate.yaml
  - ingress-route.yaml
  - keycloak-cr.yaml

Step 2: Author the ExternalSecrets

argocd/app-configs/keycloak/secrets.yaml — two secrets: the DB creds the Keycloak CR references, and the bootstrap admin. The CR’s db.usernameSecret/passwordSecret need a basic-auth-shaped secret; the admin uses the operator’s KC_BOOTSTRAP_ADMIN_* convention via a secret.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: keycloak-db-secret
  namespace: keycloak
spec:
  refreshPolicy: Periodic
  refreshInterval: 5m
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: keycloak-db-secret
    creationPolicy: Owner
    template:
      data:
        username: "{{ .username }}"
        password: "{{ .password }}"
  data:
    - secretKey: username
      remoteRef:
        key: fzymgc-house/cluster/postgres/users/main-keycloak
        property: username
    - secretKey: password
      remoteRef:
        key: fzymgc-house/cluster/postgres/users/main-keycloak
        property: password
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: keycloak-bootstrap-admin
  namespace: keycloak
spec:
  refreshPolicy: Periodic
  refreshInterval: 5m
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: keycloak-bootstrap-admin
    creationPolicy: Owner
    template:
      data:
        username: "{{ .username }}"
        password: "{{ .password }}"
  data:
    - secretKey: username
      remoteRef:
        key: fzymgc-house/cluster/keycloak
        property: admin_username
    - secretKey: password
      remoteRef:
        key: fzymgc-house/cluster/keycloak
        property: admin_password

Step 3: Author the Certificates (internal + public)

argocd/app-configs/keycloak/certificate.yaml:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: keycloak-tls
  namespace: keycloak
spec:
  secretName: keycloak-tls
  issuerRef:
    name: vault-issuer
    kind: ClusterIssuer
  dnsNames:
    - id.fzymgc.house
    - keycloak-service.keycloak.svc.cluster.local
    - keycloak-service.keycloak.svc
  usages:
    - server auth
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: keycloak-public-tls
  namespace: keycloak
spec:
  secretName: keycloak-public-tls
  issuerRef:
    name: cloudflare-acme-issuer
    kind: ClusterIssuer
  dnsNames:
    - id.fzymgc.house
  usages:
    - server auth

Step 4: Author the IngressRoute

argocd/app-configs/keycloak/ingress-route.yaml (the operator’s Service is keycloak-service:8443 for an HTTPS-enabled CR; route at the websecure entrypoint):

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: keycloak
  namespace: keycloak
  annotations:
    router-hosts.fzymgc.house/enabled: "true"
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`id.fzymgc.house`)
      kind: Rule
      services:
        - name: keycloak-service
          port: 8443
          scheme: https
          serversTransport: keycloak-internal-tls
  tls:
    secretName: keycloak-public-tls
---
# Trust the internal (vault-issuer) cert on the Traefik→Keycloak hop.
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
  name: keycloak-internal-tls
  namespace: keycloak
spec:
  serverName: keycloak-service.keycloak.svc.cluster.local
  rootCAsSecrets:
    - keycloak-tls

If the Task 4 Step 9 exit-gate curl fails TLS verification on the Traefik→Keycloak hop, rootCAsSecrets needs the vault-issuer CA (a secret carrying ca.crt), not the leaf keycloak-tls secret. Reference the vault-issuer CA secret instead. Do not set insecureSkipVerify — skipping TLS verification is prohibited without explicit approval (root CLAUDE.md).

Step 5: Author the Keycloak CR

argocd/app-configs/keycloak/keycloak-cr.yaml (points at the CNPG pooler; proxy.headers: xforwarded because Traefik terminates the public cert).

Before authoring, confirm the CR schema for the pinned <KC_VERSION> — the Keycloak CRD ships k8s.keycloak.org/v2alpha1 (verified against keycloak-k8s-resources 26.x; there is no v2beta1), and the spec field shapes (hostname, bootstrapAdmin, proxy, db) have evolved across operator releases. After Task 3 installs the operator, run kubectl --context fzymgc-house explain keycloak.spec --api-version=k8s.keycloak.org/v2alpha1 and adjust any field below that the pinned CRD names differently:

apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
metadata:
  name: keycloak
  namespace: keycloak
spec:
  instances: 2
  db:
    vendor: postgres
    host: keycloak-pooler-rw.postgres.svc.cluster.local
    port: 5432
    database: keycloak
    usernameSecret:
      name: keycloak-db-secret
      key: username
    passwordSecret:
      name: keycloak-db-secret
      key: password
  http:
    tlsSecret: keycloak-tls
  hostname:
    hostname: https://id.fzymgc.house
  proxy:
    headers: xforwarded
  bootstrapAdmin:
    user:
      secret: keycloak-bootstrap-admin
  additionalOptions:
    - name: log-console-output
      value: json
    - name: metrics-enabled
      value: "true"

Step 6: Author the ArgoCD Application

argocd/cluster-app/templates/keycloak.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: keycloak
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  project: auth
  source:
    repoURL: https://github.com/fzymgc-house/selfhosted-cluster
    targetRevision: HEAD
    path: argocd/app-configs/keycloak
  destination:
    server: https://kubernetes.default.svc
    namespace: keycloak
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
      - CreateNamespace=true
  ignoreDifferences:
    - kind: ExternalSecret
      group: external-secrets.io
      jqPathExpressions:
        - .spec.data[].remoteRef.conversionStrategy
        - .spec.data[].remoteRef.decodingStrategy
        - .spec.data[].remoteRef.metadataPolicy

(Sync-wave "1" so it follows the operator at wave "0".)

Step 7: Validate locally

Run:

kubectl --context fzymgc-house kustomize argocd/app-configs/keycloak >/dev/null && echo OK

Expected: OK (builds; the Keycloak CRD is cluster-side so client dry-run of the CR is skipped — build validation is sufficient pre-merge).

Step 8: Commit

feat(keycloak): Keycloak instance, TLS, ingress at id.fzymgc.house [hl-jb6q].

Step 9: Post-merge verification (the Phase 0 exit gate)

After sync:

curl -fsS https://id.fzymgc.house/realms/master/.well-known/openid-configuration | \
  grep -o '"issuer":"[^"]*"'

Expected: "issuer":"https://id.fzymgc.house/realms/master" — Keycloak is up, TLS valid, hostname correct.

Task 5: `tf/keycloak` module skeleton + `fzymgc` realm

Files:

Create: tf/keycloak/versions.tf, terraform.tf, vault.tf, variables.tf, data-sources.tf, main.tf, groups.tf, outputs.tf, README.md
Step 1: Author versions.tf

terraform {
  required_version = "~> 1.15"
  required_providers {
    keycloak = {
      source  = "keycloak/keycloak"
      version = "~> 5.0"
    }
    vault = {
      source  = "hashicorp/vault"
      version = "~> 5.9"
    }
  }
}

(Verify the latest GA keycloak/keycloak provider major before pinning — terraform init will report; adjust ~> 5.0 if needed.)

Step 2: Author variables.tf

Copy tf/authentik/variables.tf verbatim (the vault_addr + tfc_vault_dynamic_credentials object) — it is provider-agnostic.

Step 3: Author terraform.tf (providers + backend)

provider "vault" {
  address          = var.tfc_vault_dynamic_credentials != null ? var.tfc_vault_dynamic_credentials.default.address : var.vault_addr
  skip_child_token = var.tfc_vault_dynamic_credentials != null
  dynamic "auth_login_token_file" {
    for_each = var.tfc_vault_dynamic_credentials != null ? [1] : []
    content {
      filename = var.tfc_vault_dynamic_credentials.default.token_filename
    }
  }
}

# data.vault_kv_secret_v2.keycloak is defined in vault.tf (mirrors tf/authentik's
# vault.tf split); Terraform resolves the cross-file reference below.
provider "keycloak" {
  url       = "https://id.fzymgc.house"
  client_id = "admin-cli"
  username  = data.vault_kv_secret_v2.keycloak.data["terraform_username"]
  password  = data.vault_kv_secret_v2.keycloak.data["terraform_password"]
}

terraform {
  cloud {
    organization = "fzymgc-house"
    workspaces {
      tags = ["main-cluster", "keycloak"]
    }
  }
}

Step 4: Author main.tf (the realm)

resource "keycloak_realm" "fzymgc" {
  realm   = "fzymgc"
  enabled = true

  display_name = "fzymgc.house"

  # Token lifespans mirror the Authentik mcp-public posture (access 1h).
  access_token_lifespan = "1h"

  # Registration of NEW end-users stays off here (Phase 4 owns the
  # registration flow). This realm flag is unrelated to OAuth client DCR.
  registration_allowed = false
}

Step 5: Author vault.tf, data-sources.tf, groups.tf, outputs.tf, README.md

vault.tf (the data source the keycloak provider reads its admin creds from — split out to mirror tf/authentik/vault.tf):

data "vault_kv_secret_v2" "keycloak" {
  mount = "secret"
  name  = "fzymgc-house/cluster/keycloak"
}

outputs.tf:

output "realm_issuer" {
  value = "https://id.fzymgc.house/realms/${keycloak_realm.fzymgc.realm}"
}

data-sources.tf and groups.tf: create as empty marker files with a header comment (groups arrive with the per-app migrations in Phase 2). README.md: one paragraph pointing at the spec + the HCP workspace name.

Step 6: Validate locally

Run:

cd tf/keycloak && terraform fmt -check && terraform init -backend=false && terraform validate

Expected: Success! The configuration is valid.

Step 7: Commit

feat(keycloak): tf/keycloak module skeleton + fzymgc realm [hl-jb6q].

Step 8: Post-merge verification (HCP apply)

After PR merge, HCP TF plans/applies. Then:

curl -fsS https://id.fzymgc.house/realms/fzymgc/.well-known/openid-configuration | \
  grep -o '"issuer":"[^"]*"'

Expected: "issuer":"https://id.fzymgc.house/realms/fzymgc" — the realm exists.

Phase 1 — MCP authorization-server slice (real DCR) ⭐

Task 6: `mcp-public` PKCE client in Keycloak

Files:

Create: tf/keycloak/mcp_public.tf

Reproduces the Authentik mcp_public_oauth.tf semantics: a public client, PKCE S256, loopback redirect URIs, and an audience mapper so issued tokens carry aud: mcp-public (the AgentgatewayPolicy validates that audience).

Step 1: Author the client + audience mapper

resource "keycloak_openid_client" "mcp_public" {
  realm_id  = keycloak_realm.fzymgc.id
  client_id = "mcp-public"
  name      = "mcp-public"
  enabled   = true

  access_type                  = "PUBLIC"
  standard_flow_enabled        = true
  pkce_code_challenge_method   = "S256"

  # Loopback redirect for local MCP clients (Claude Code, agentgateway).
  valid_redirect_uris = [
    "http://127.0.0.1:*/*",
    "http://localhost:*/*",
  ]

  # DCR-registered clients are created dynamically; this static client is the
  # discovery anchor / fallback. Full DCR is exercised in Task 10.
}

# Force the audience claim to "mcp-public" so the agentgateway jwtAuthentication
# (audiences: [mcp-public]) and engram (audience: mcp-public) both accept the token.
resource "keycloak_openid_audience_protocol_mapper" "mcp_public_aud" {
  realm_id                  = keycloak_realm.fzymgc.id
  client_id                 = keycloak_openid_client.mcp_public.id
  name                      = "mcp-public-audience"
  included_client_audience  = "mcp-public"
  add_to_id_token           = false
  add_to_access_token       = true
}

(Confirm the resource name keycloak_openid_audience_protocol_mapper and arg names via terraform validate — adjust to the provider’s exact schema if it differs.)

Step 2: Validate locally

Run: cd tf/keycloak && terraform fmt -check && terraform validate Expected: valid.

Step 3: Commit

feat(keycloak): mcp-public PKCE client + audience mapper [hl-jb6q].

Step 4: Post-merge verification

After HCP apply:

curl -fsS "https://id.fzymgc.house/realms/fzymgc/.well-known/openid-configuration" | \
  grep -o '"registration_endpoint":"[^"]*"'

Expected: the registration_endpoint is advertised (it is realm-level and always present) — confirms DCR discovery surface exists. Client presence is verified end-to-end in Task 10.

Task 7: Enable agentgateway-proxied anonymous DCR (Trusted Hosts policy) ⚠️ RISKIEST

Files:

Create: tf/keycloak/client_registration.tf (if the provider supports it) or argocd/app-configs/keycloak/dcr-trusted-hosts-job.yaml (REST bootstrap fallback)

Context — why this is the risk: Keycloak disables anonymous DCR by default via the realm’s Trusted Hosts client-registration policy (whitelists nothing). agentgateway proxies the anonymous /register (no initial-access-token), so the policy must trust the agentgateway source. The keycloak/keycloak Terraform provider has no confirmed resource for the Trusted Hosts client-registration policy (it is a realm component). This task tries Terraform first and falls back to a one-shot REST bootstrap Job.

Step 1: Determine agentgateway’s source identity Keycloak will see

The registration request reaches Keycloak from the agentgateway pod via its Service. Resolve the stable Service ClusterIP (not pod IP):

# read-only MCP resources_get: Service 'agentgateway' (or the gateway data-plane svc) in ns 'agentgateway'
# record the spec.clusterIP

Record the ClusterIP as <AGW_CLUSTERIP>. (If Keycloak sees the request source as the in-cluster Service, that IP is the trusted host; if via an egress, use that. Confirm against the first live 401 in Task 10 and adjust.)

Step 2: Probe whether the TF provider exposes a trusted-hosts / component resource

Run:

cd tf/keycloak && terraform providers schema -json 2>/dev/null | \
  grep -oE 'keycloak_(realm_component|generic_component|openid_client_registration[a-z_]*|component)' | sort -u

Expected: prints any matching resource name, or nothing. If a usable resource exists, prefer it (declarative). If nothing, use the REST fallback (steps 3–4).

Step 3 (fallback): Author a one-shot REST bootstrap Job

argocd/app-configs/keycloak/dcr-trusted-hosts-job.yaml — a Job that authenticates with the bootstrap admin and updates the realm’s trusted-hosts client-registration policy component to whitelist <AGW_CLUSTERIP> as a trusted host (enabling host-sending-registration-request-must-match so only that source may anonymously register, and relaxing client-uris-must-match since DCR-registered loopback redirect URIs won’t match a fixed host list):

apiVersion: batch/v1
kind: Job
metadata:
  name: keycloak-dcr-trusted-hosts
  namespace: keycloak
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  backoffLimit: 5
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: configure-trusted-hosts
          image: quay.io/keycloak/keycloak:<KC_VERSION>
          env:
            - name: KC_ADMIN
              valueFrom: { secretKeyRef: { name: keycloak-bootstrap-admin, key: username } }
            - name: KC_PASS
              valueFrom: { secretKeyRef: { name: keycloak-bootstrap-admin, key: password } }
            - name: AGW_CLUSTERIP
              value: "<AGW_CLUSTERIP>"
          command:
            - /bin/bash
            - -ec
            - |
              kcadm=/opt/keycloak/bin/kcadm.sh
              $kcadm config credentials --server https://keycloak-service:8443 \
                --realm master --user "$KC_ADMIN" --password "$KC_PASS" \
                --truststore /dev/null --trustpass "" 2>/dev/null || \
              $kcadm config credentials --server https://keycloak-service:8443 \
                --realm master --user "$KC_ADMIN" --password "$KC_PASS" --client admin-cli
              # Find the Trusted Hosts component for the fzymgc realm and update it.
              cid=$($kcadm get components -r fzymgc \
                -q name="Trusted Hosts" --fields id --format csv --noquotes | head -1)
              $kcadm update components/$cid -r fzymgc \
                -s 'config."trusted-hosts"=["'"$AGW_CLUSTERIP"'"]' \
                -s 'config."host-sending-registration-request-must-match"=["true"]' \
                -s 'config."client-uris-must-match"=["false"]'

(The truststore flags are best-effort; if the internal cert chain blocks kcadm, mount the keycloak-tls CA and pass --truststore. This is the documented rough edge — validate against the live cluster.)

Step 4: Validate locally

Run: kubectl --context fzymgc-house apply --dry-run=client -f argocd/app-configs/keycloak/dcr-trusted-hosts-job.yaml and add the file to the keycloak kustomization.yaml resources. Expected: dry-run OK.

Step 5: Commit

feat(keycloak): enable agentgateway-proxied anonymous DCR via Trusted Hosts [hl-jb6q].

Step 6: Post-merge verification — config applied, NOT an external curl

Verify the configuration was written, not the end-to-end registration. Do not test with an external curl to the registration endpoint: Keycloak’s Trusted Hosts policy checks the real TCP source IP (getRemoteAddr()), not X-Forwarded-For, so an external request arrives with Traefik’s IP — not <AGW_CLUSTERIP> — and is correctly rejected with 403. That 403 is the policy working as designed; “fixing” it by loosening the host match would be a security regression. End-to-end anonymous DCR is verified only via the agentgateway-proxied path in Task 10 Step 2 (the request then originates from the agentgateway pod).

Here, confirm only:

# read-only MCP: Job 'keycloak-dcr-trusted-hosts' in ns keycloak shows status.succeeded: 1
# then confirm the component carries the trusted host (run inside a keycloak pod):
kubectl --context fzymgc-house exec -n keycloak deploy/keycloak -- \
  /opt/keycloak/bin/kcadm.sh get components -r fzymgc -q name="Trusted Hosts" \
  --fields 'config' 2>/dev/null | grep -o '<AGW_CLUSTERIP>'

Expected: the Job succeeded and <AGW_CLUSTERIP> appears in the Trusted Hosts component config. (This kubectl exec is a read-only inspection, not a GitOps mutation.)

Task 8: Repoint the agentgateway `mcp-engram-oauth` policy to Keycloak

Files:

Modify: argocd/app-configs/agentgateway/mcp-engram.yaml
Create: argocd/app-configs/keycloak/keycloak-jwks-referencegrant.yaml
Modify: argocd/app-configs/keycloak/kustomization.yaml (add the ReferenceGrant)
Step 1: Author the cross-namespace JWKS ReferenceGrant

Mirror argocd/app-configs/authentik/authentik-jwks.yaml for the keycloak namespace. argocd/app-configs/keycloak/keycloak-jwks-referencegrant.yaml:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: agentgateway-to-keycloak-jwks
  namespace: keycloak
spec:
  from:
    - group: agentgateway.dev
      kind: AgentgatewayPolicy
      namespace: agentgateway
  to:
    - group: ""
      kind: Service
      name: keycloak-service

(Confirm the from group/kind against the existing authentik ReferenceGrant — match it exactly, including apiVersion.)

Step 2: Rewrite the mcp-engram-oauth policy block

In argocd/app-configs/agentgateway/mcp-engram.yaml, replace the jwtAuthentication provider + mcp block (currently issuer https://auth.fzymgc.house/application/o/mcp-public/, jwksPath /application/o/mcp-public/jwks/, backendRef authentik-server ns authentik, and mcp.clientId: "mcp-public") with:

  traffic:
    jwtAuthentication:
      mode: Strict
      providers:
        - issuer: "https://id.fzymgc.house/realms/fzymgc"
          audiences:
            - mcp-public
          jwks:
            remote:
              jwksPath: /realms/fzymgc/protocol/openid-connect/certs
              cacheDuration: 5m
              backendRef:
                group: ""
                kind: Service
                name: keycloak-service
                namespace: keycloak
                port: 8443
      mcp:
        # Real RFC 7591 DCR-proxy: agentgateway proxies anonymous /register to
        # Keycloak's clients-registrations endpoint. No clientId short-circuit.
        provider: Keycloak
        resourceMetadata:
          resource: "https://mcp-gw.fzymgc.house/mcp/engram"
          bearerMethodsSupported:
            - header

Leave the AgentgatewayBackend, HTTPRoute, and mcp-engram-auth-passthrough policy unchanged.

Step 3: Validate locally

Run:

kubectl --context fzymgc-house kustomize argocd/app-configs/agentgateway | \
  grep -A2 'issuer:' | grep id.fzymgc.house
kubectl --context fzymgc-house kustomize argocd/app-configs/keycloak | grep agentgateway-to-keycloak-jwks

Expected: the new issuer line and the ReferenceGrant both appear; no remaining clientId: under the engram policy (grep -c 'clientId' argocd/app-configs/agentgateway/mcp-engram.yaml → 0).

Step 4: Commit

feat(keycloak): repoint agentgateway mcp-engram-oauth to Keycloak realm [hl-jb6q].

Task 9: Repoint engram’s issuer to the Keycloak realm

Files:

Modify: argocd/cluster-app/templates/agent-memory.yaml (the oidc.issuer at the mcp-public validation block, ~line 152)

engram re-validates the forwarded JWT against the same issuer agentgateway enforces (the passthrough policy re-attaches the token). It must move in lockstep with Task 8 or engram 401s every forwarded request.

Step 1: Change the issuer (keep audience)

In the oidc: block that currently reads issuer: "https://auth.fzymgc.house/application/o/mcp-public/" with audience: mcp-public, set:

            oidc:
              issuer: "https://id.fzymgc.house/realms/fzymgc"
              audience: mcp-public

Leave the ui.issuer (engram-ui, application/o/engram-ui/) unchanged — that confidential UI client is a Phase 2 app migration, not part of this slice.

Step 2: Validate locally

Run:

grep -n 'id.fzymgc.house/realms/fzymgc' argocd/cluster-app/templates/agent-memory.yaml
grep -n 'application/o/mcp-public' argocd/cluster-app/templates/agent-memory.yaml

Expected: the first matches the new issuer; the second prints nothing (the mcp-public Authentik issuer is gone). The engram-ui issuer line remains.

Step 3: Commit

feat(keycloak): repoint engram MCP issuer to Keycloak realm [hl-jb6q].

Task 10: End-to-end real-DCR validation (Phase 1 exit gate)

Files: none (verification only)

This is the acceptance gate that proves the forcing function — the property mock-DCR and static-client both fail: two registrations return distinct client_ids, and a full MCP client completes DCR + PKCE + a tool call.

Step 1: PRM discovery is served and points at Keycloak

Run:

curl -fsS https://mcp-gw.fzymgc.house/.well-known/oauth-protected-resource/mcp/engram | \
  python3 -m json.tool

Expected: a valid RFC 9728 document whose authorization_servers (or resource/metadata) resolves to the Keycloak realm https://id.fzymgc.house/realms/fzymgc.

Step 2: Two DCR registrations yield DISTINCT client_ids

Run:

ep=https://mcp-gw.fzymgc.house/.well-known/oauth-authorization-server/mcp/engram/client-registration
id1=$(curl -fsS -X POST "$ep" -H 'content-type: application/json' \
  -d '{"redirect_uris":["http://127.0.0.1:8976/callback"]}' | python3 -c 'import sys,json;print(json.load(sys.stdin)["client_id"])')
id2=$(curl -fsS -X POST "$ep" -H 'content-type: application/json' \
  -d '{"redirect_uris":["http://127.0.0.1:8977/callback"]}' | python3 -c 'import sys,json;print(json.load(sys.stdin)["client_id"])')
echo "id1=$id1 id2=$id2"; [ "$id1" != "$id2" ] && echo "REAL DCR ✅" || echo "NOT REAL DCR ❌"

Expected: REAL DCR ✅ — two different client_ids. (Under the old mock-DCR/static model both would be the literal mcp-public.)

Step 3: Full MCP client DCR + PKCE + tool call

Configure a real MCP client (Claude Code) against https://mcp-gw.fzymgc.house/mcp/engram with OAuth (PKCE) discovery, complete the browser auth against Keycloak, and invoke one engram tool (e.g. list_memory). Expected: the client registers dynamically, authenticates against id.fzymgc.house, and the tool call returns data — engram accepted the Keycloak-issued, agentgateway-validated, passthrough-forwarded JWT.

Step 4: Negative check — Strict JWT still enforced

Run:

curl -s -o /dev/null -w '%{http_code}\n' https://mcp-gw.fzymgc.house/mcp/engram \
  -H 'content-type: application/json' -d '{"jsonrpc":"2.0","method":"tools/list","id":1}'

Expected: 401 with a WWW-Authenticate header pointing at the protected-resource metadata — unauthenticated calls are still rejected.

Step 5: Record the gate result on the bead

bd note hl-jb6q "Phase 1 exit gate: real DCR verified (distinct client_ids), MCP client e2e green, Strict JWT 401 on unauth. Keycloak is the MCP AS; hl-23m mock-DCR superseded in practice."

Rollback

Phase 1 is a single coordinated revert: restore the issuer/jwks/clientId block in argocd/app-configs/agentgateway/mcp-engram.yaml (Task 8) and the engram oidc.issuer (Task 9) to the Authentik values, and merge. That returns the cluster to today’s state (static mcp-public against Authentik) within one ArgoCD sync. Phase 0 infrastructure (Keycloak, CNPG, realm) is additive and can remain stood up regardless.

Spec coverage check

Spec §6 phase item	Task(s)
Phase 0: CNPG, operator, Keycloak CR, realm, tf/keycloak skeleton, id.fzymgc.house ingress	2, 3, 4, 5 (+ Vault/workspace prereqs in 1)
Phase 1: mcp-public client, repoint agentgateway, Trusted-Hosts DCR, engram re-validate, real-DCR assertion	6, 7, 8, 9, 10
§4 anonymous-DCR Trusted-Hosts security trade-off	7 (Service ClusterIP, not pod IP)
§7 Phase-1 fallback = static mcp-public against Authentik	Rollback section

Phases 2–5 (13 OIDC apps, forward-auth via oauth2-proxy, parity, decommission) are out of scope for this plan and get their own plans once this slice proves out.

Keycloak Migration — Phase 0+1 Implementation Plan

GitOps & Validation Conventions (read first)

Phase 0 — Foundation

Task 1: Vault secrets + HCP Terraform workspace prerequisites

Task 2: CNPG database + pooler for Keycloak

Task 3: Install the Keycloak Operator

Task 4: Keycloak instance — namespace, secrets, TLS, ingress, CR

Task 5: tf/keycloak module skeleton + fzymgc realm

Phase 1 — MCP authorization-server slice (real DCR) ⭐

Task 6: mcp-public PKCE client in Keycloak

Task 7: Enable agentgateway-proxied anonymous DCR (Trusted Hosts policy) ⚠️ RISKIEST

Task 8: Repoint the agentgateway mcp-engram-oauth policy to Keycloak

Task 9: Repoint engram’s issuer to the Keycloak realm

Task 10: End-to-end real-DCR validation (Phase 1 exit gate)

Rollback

Spec coverage check

Task 5: `tf/keycloak` module skeleton + `fzymgc` realm

Task 6: `mcp-public` PKCE client in Keycloak

Task 8: Repoint the agentgateway `mcp-engram-oauth` policy to Keycloak