Skip to content
fzymgc-house Cluster Docs

Plan: per-resource MCP audiences — Phase A (hosted k8s MCP)

Per-Resource MCP Audiences — Phase A Implementation Plan

Section titled “Per-Resource MCP Audiences — Phase A Implementation Plan”

For agentic workers: REQUIRED SUB-SKILL: Use dev-flow:subagent-driven-development (recommended) or dev-flow:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Bead: hl-c1ub (Phase A). Spec: docs/engineering/specs/2026-07-03-mcp-per-resource-audiences-design.md. Follow-ons: hl-u18e (Phase B, engram), hl-05xz (Phase C, remove blanket).

Goal: Make the hosted Kubernetes MCP token carry a resource-specific audience (aud=https://mcp-gw.fzymgc.house/mcp/kubernetes) so /mcp/kubernetes stops returning 401 InvalidAudience, without letting any generic realm token authenticate to the k8s API.

Architecture: Keycloak 26 cannot map RFC 8707 resource= to aud, so per-resource audiences are delivered via an optional client scope carrying an Audience mapper, made a realm optional scope (DCR clients inherit realm default+optional scopes), and advertised in the route’s RFC 9728 scopesSupported — the MCP client forwards it, the mapper fires, the audience appears only for k8s connections. The audience value is the resource URL (RFC 8707 alignment), which the gateway route and k3s_oidc_audiences are updated to trust. Fails closed.

Tech Stack: Terraform keycloak/keycloak ~> 5.8 (HCP TFC workspace main-cluster-keycloak, Agent exec, auto-apply on merge); agentgateway AgentgatewayPolicy (ArgoCD); Ansible k3s-common role (structured AuthenticationConfiguration).

Task 1: Keycloak — per-resource audience scope, mapper, realm registration, and fallback repoint

Section titled “Task 1: Keycloak — per-resource audience scope, mapper, realm registration, and fallback repoint”

Files:

  • Modify: tf/keycloak/mcp_kubernetes.tf
  • Modify: tf/keycloak/client_scopes.tf

All changes land in one main-cluster-keycloak PR (they are interdependent within the module).

  • Step 0: Confirm the realm’s CURRENT optional client scopes (read-only pre-flight)

keycloak_realm_optional_client_scopes is authoritative — it overwrites whatever optional scopes the realm has. A TFC speculative plan on the PR will not reveal what it clobbers (a to-be-created resource has no prior state to diff), so verify the live list first. This is a read-only admin-API GET (not a TFC run — GitOps-safe).

Run (admin creds: vault kv get secret/fzymgc-house/cluster/keycloak → admin_username / admin_password):

Terminal window
KC=https://id.fzymgc.house
TOKEN=$(curl -sf -d client_id=admin-cli -d "username=$KC_ADMIN_USER" -d "password=$KC_ADMIN_PW" \
  -d grant_type=password "$KC/realms/master/protocol/openid-connect/token" | jq -r .access_token)
curl -sf -H "Authorization: Bearer $TOKEN" \
  "$KC/admin/realms/fzymgc/default-optional-client-scopes" | jq -r '.[].name' | sort

Expected output (Keycloak built-in defaults; tf/keycloak manages no realm scopes today):

address
microprofile-jwt
offline_access
phone

If the live list contains additional names, add each of them to optional_scopes in Step 3 so the authoritative resource does not drop them.

  • Step 1: Add the mcp-kubernetes optional client scope + its Audience mapper

In tf/keycloak/mcp_kubernetes.tf, append:

# Per-resource audience for the hosted Kubernetes MCP (hl-c1ub). This OPTIONAL
# client scope is advertised on the k8s route (RFC 9728 scopesSupported) and
# requested by DCR clients; its Audience mapper stamps the resource URL into the
# access token ONLY when the scope is requested => per-resource aud, fails closed.
# (Keycloak 26 does not map the RFC 8707 resource= param to aud; this is the
# vendor-documented workaround — keycloak docs/guides/securing-apps/mcp-authz-server.adoc.)
resource "keycloak_openid_client_scope" "mcp_kubernetes" {
  realm_id               = keycloak_realm.fzymgc.id
  name                   = "mcp-kubernetes"
  description            = "Per-resource audience for the hosted Kubernetes MCP (hl-c1ub)"
  include_in_token_scope = true
}


resource "keycloak_openid_audience_protocol_mapper" "mcp_kubernetes_scope_aud" {
  realm_id        = keycloak_realm.fzymgc.id
  client_scope_id = keycloak_openid_client_scope.mcp_kubernetes.id
  name            = "mcp-kubernetes-resource-audience"


  included_custom_audience = "https://mcp-gw.fzymgc.house/mcp/kubernetes"
  add_to_id_token          = false
  add_to_access_token      = true
}
  • Step 2: Repoint the static fallback client’s audience mapper to the URL

The static mcp_kubernetes client (fallback for a non-DCR clientId=mcp-kubernetes config) currently stamps its own client id as the audience. Change it to the resource URL so the fallback path mints the same audience the DCR path does. In tf/keycloak/mcp_kubernetes.tf, in resource "keycloak_openid_audience_protocol_mapper" "mcp_kubernetes_aud", replace the line:

  included_client_audience = "mcp-kubernetes"

with:

  included_custom_audience = "https://mcp-gw.fzymgc.house/mcp/kubernetes"

(included_client_audience and included_custom_audience are mutually exclusive; this swaps which one is set.)

Also update the now-stale comments in the same file so they no longer claim the audience is the literal mcp-kubernetes (comment hygiene — no functional effect). In the header comment (line ~4), change MUST carry aud=mcp-kubernetes to MUST carry aud=https://mcp-gw.fzymgc.house/mcp/kubernetes. In the mapper preamble (lines ~23-25) — # Access-token aud=mcp-kubernetes so both agentgateway jwtAuthentication / # (audiences: [mcp-kubernetes]) and the k3s API server (k3s_oidc_audiences) — replace both mcp-kubernetes occurrences with https://mcp-gw.fzymgc.house/mcp/kubernetes.

  • Step 3: Register the realm optional client scopes (authoritative)

In tf/keycloak/client_scopes.tf, append. Include any extra names Step 0 surfaced:

# Realm OPTIONAL client scopes. AUTHORITATIVE: this resource overwrites Keycloak's
# defaults and any manual adjustments (provider docs). The first four are Keycloak's
# built-in realm optional scopes (verified live, Task 1 Step 0). groups + mcp-kubernetes
# are added so DCR-registered MCP clients can REQUEST them (DCR clients inherit realm
# default+optional scopes) and thereby receive the RBAC groups claim / per-resource
# audience ONLY when connecting to the k8s MCP. [hl-c1ub]
resource "keycloak_realm_optional_client_scopes" "fzymgc" {
  realm_id = keycloak_realm.fzymgc.id


  optional_scopes = [
    "address",
    "phone",
    "offline_access",
    "microprofile-jwt",
    keycloak_openid_client_scope.groups.name,
    keycloak_openid_client_scope.mcp_kubernetes.name,
  ]
}
  • Step 4: Format and validate

Run:

Terminal window
cd tf/keycloak && terraform fmt -recursive && terraform validate

Expected: terraform fmt reports the two edited files (or nothing if already formatted); terraform validate → Success! The configuration is valid. (Requires terraform init first if the module is not initialized locally.)

  • Step 5: Commit
Terminal window
jj commit -m "feat(keycloak): per-resource aud for hosted k8s MCP via optional client scope [hl-c1ub]" \
  -m "Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"

(In this jj worktree, run jj commit with the Bash sandbox disabled — the shared store lives in the main repo’s .jj/.)

Merge-time expectation (TFC main-cluster-keycloak, auto-apply on merge): the plan should show 3 to add (mcp_kubernetes scope, its audience mapper, realm_optional_client_scopes.fzymgc) and 1 to change (the repointed mcp_kubernetes_aud mapper), above the standing keycloak provider-replacement baseline (memory hl-je6h: a 9/9 provider-replacement cascade is normal). Confirm realm_optional_client_scopes shows no removals of existing scope names.

Task 2: agentgateway — advertise the scopes and move the validated audience to the URL

Section titled “Task 2: agentgateway — advertise the scopes and move the validated audience to the URL”

Files:

  • Modify: argocd/app-configs/agentgateway/mcp-kubernetes.yaml

  • Step 1: Advertise mcp-kubernetes + groups in scopesSupported

In the mcp-kubernetes-oauth AgentgatewayPolicy, under traffic.jwtAuthentication.providers[].mcp.resourceMetadata, replace:

          scopesSupported:
            - offline_access

with:

          scopesSupported:
            - mcp-kubernetes
            - groups
            - offline_access
  • Step 2: Move the validated audience to the resource URL

In the same policy, under traffic.jwtAuthentication.providers[], replace:

          audiences:
            - mcp-kubernetes

with:

          audiences:
            - https://mcp-gw.fzymgc.house/mcp/kubernetes
  • Step 3: Render and lint

Run:

Terminal window
kubectl kustomize argocd/app-configs/agentgateway | rg -A3 'scopesSupported|audiences:'
yamllint argocd/app-configs/agentgateway/mcp-kubernetes.yaml

Expected: the kustomize render succeeds and shows the new scopesSupported list (mcp-kubernetes, groups, offline_access) and audiences: [https://mcp-gw.fzymgc.house/mcp/kubernetes]; yamllint prints no errors.

  • Step 4: Commit
Terminal window
jj commit -m "feat(agentgateway): validate k8s MCP resource-URL aud + advertise mcp-kubernetes/groups scopes [hl-c1ub]" \
  -m "Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"

Task 3: k3s — trust the resource-URL audience

Section titled “Task 3: k3s — trust the resource-URL audience”

Files:

  • Modify: ansible/roles/k3s-common/defaults/main.yml:59-65

  • Step 1: Swap the audience value and fix the comment

In ansible/roles/k3s-common/defaults/main.yml, replace the audiences block and its comment. Old:

# audiences replaces the legacy required-claim aud=kubernetes; each must equal a
# Keycloak client_id. Multi-element => default audienceMatchPolicy MatchAny, so a
# token bearing EITHER audience is accepted (kubectl=kubernetes; hosted MCP
# identity passthrough=mcp-kubernetes, hl-qffr.2 / design #1561).
k3s_oidc_audiences:
  - "kubernetes"
  - "mcp-kubernetes"

New:

# audiences replaces the legacy required-claim aud=kubernetes. Each entry is an
# accepted aud value (NOT necessarily a Keycloak client_id): kubectl uses the
# "kubernetes" client (aud=client_id by OIDC spec); the hosted k8s MCP passthrough
# token carries aud=<resource URL>, stamped by a per-resource Keycloak optional
# client scope (RFC 8707 alignment, hl-c1ub). Multi-element => audienceMatchPolicy
# MatchAny, so a token bearing EITHER audience is accepted.
k3s_oidc_audiences:
  - "kubernetes"
  - "https://mcp-gw.fzymgc.house/mcp/kubernetes"
  • Step 2: Lint and syntax-check

Run:

Terminal window
yamllint ansible/roles/k3s-common/defaults/main.yml
ansible-playbook -i ansible/inventory/hosts.yml ansible/k3s-playbook.yml --syntax-check

Expected: yamllint clean; --syntax-check reports playbook: ansible/k3s-playbook.yml with no errors.

  • Step 3: Commit
Terminal window
jj commit -m "feat(k3s): trust hosted k8s MCP resource-URL aud in k3s_oidc_audiences [hl-c1ub]" \
  -m "Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
  • Step 4: Apply the k3s config (post-merge operator action — serial, health-gated)

The list stays 2-element, so authentication-config.yaml.j2 still emits audienceMatchPolicy: MatchAny (length > 1). The k3s-config task templates /etc/rancher/k3s/authentication-config.yaml and notifies Restart k3s; the play is serial(1) and OIDC-health-gated, so a bad restart stops at the first control plane.

Dry-run, then apply:

Terminal window
ansible-playbook -i ansible/inventory/hosts.yml ansible/k3s-playbook.yml \
  --tags k3s-config,k3s-oidc-verify --check --diff
ansible-playbook -i ansible/inventory/hosts.yml ansible/k3s-playbook.yml \
  --tags k3s-config,k3s-oidc-verify

Expected: --check --diff shows the authentication-config.yaml diff adding the URL audience on each control plane; the real run restarts k3s serially and the k3s-oidc-verify tasks pass on each node.

Recovery if an apiserver fails to start (memory hl-qffr.2 incident): on the affected control plane, read health and patch via the node-local X.509 kubeconfig (OIDC-independent): /etc/rancher/k3s/k3s.yaml → edit /etc/rancher/k3s/authentication-config.yaml → systemctl restart k3s.

Task 4: End-to-end interactive PKCE smoke test (human gate)

Section titled “Task 4: End-to-end interactive PKCE smoke test (human gate)”

Files: none (verification only).

This is the definitive confirmation that the DCR client forwards the custom mcp-kubernetes scope in practice. Run after Tasks 1–3 are applied (Keycloak auto-applied on merge, gateway synced by ArgoCD, k3s play run).

  • Step 1: Connect Claude Code to the hosted k8s MCP via DCR (no hardcoded clientId)

Point an MCP client at https://mcp-gw.fzymgc.house/mcp/kubernetes with no oauth.clientId, and complete the interactive Keycloak PKCE login.

  • Step 2: Decode the access token and assert the claims

Decode the JWT (e.g. paste the access token into a local decoder, or cut -d. -f2 | base64 -d | jq). Assert:

  • aud includes https://mcp-gw.fzymgc.house/mcp/kubernetes
  • the groups claim is present (the user’s k8s groups, e.g. k8s-admins)
  • email / email_verified present (still supplied by the blanket mapper in Phase A)

If aud/groups are missing: the DCR client may be a pre-change registration (registered before the realm optional scopes existed, so it did not inherit them). Clear the client’s cached OAuth registration so Claude Code re-runs DCR and inherits the updated realm optional scopes, then retry.

  • Step 3: Assert the route now works

tools/list against /mcp/kubernetes returns 200 (previously 401). In the agentgateway namespace, the pod logs no longer show route:mcp-kubernetes … Error(InvalidAudience) for this user.

  • Step 4: Confirm no engram regression

The same user’s engram MCP calls (/mcp/engram) still return 200 — the blanket mcp-public mapper is untouched in Phase A, so engram is unaffected.

  • Step 5: Close the bead

On green: bd close hl-c1ub --reason="k8s MCP per-resource audience live; /mcp/kubernetes authenticates with aud=<resource URL> + groups; hl-qffr.4 unblocked". This unblocks hl-qffr.4; hl-u18e (Phase B) becomes the next ready follow-on.

Rollback

Section titled “Rollback”

Each task is an isolated revert:

  • Keycloak (Task 1): revert the two .tf files. Removing realm_optional_client_scopes.fzymgc returns the realm to Keycloak-managed optional scopes (the four defaults). DCR clients stop being able to request mcp-kubernetes/groups; tokens lose the resource audience → /mcp/kubernetes returns to 401 (fail-closed, no exposure).
  • Gateway (Task 2): revert mcp-kubernetes.yaml; the route goes back to validating [mcp-kubernetes].
  • k3s (Task 3): revert defaults/main.yml and re-run the k3s-config,k3s-oidc-verify play; k3s trusts mcp-kubernetes again.

Because every combination is fail-closed, a partial rollback never accepts an unintended token.