Design: per-resource MCP audiences via DCR + optional client scopes (Phase A — k8s)

Bead: hl-c1ub (Phase A). Follow-on: hl-u18e (Phase B, engram), hl-05xz (Phase C, remove blanket)
Date: 2026-07-03
Status: Draft (pending design-reviewer gate)
Author: Sean (with Claude)
Relates to: epic hl-qffr (hosted k8s + TF MCP), hl-qffr.4 (blocked k8s route), ADR hl-bc94 (k8s identity passthrough), ADR hl-2jrh (resource-specific well-known paths), ADR hl-idws (k3s Keycloak OIDC), hl-thsf.4 (blanket mcp-public mapper), tf/keycloak/, argocd/app-configs/agentgateway/

Problem

The hosted Kubernetes MCP route /mcp/kubernetes returns 401 Error(InvalidAudience). The route (and the k3s API it forwards to) require aud=mcp-kubernetes, but every fzymgc realm token carries a blanket aud=mcp-public — injected by an audience mapper on the built-in basic realm-default client scope (tf/keycloak/mcp_audience.tf, hl-thsf.4). Because that scope is applied to every client (including RFC 7591 dynamically-registered ones), every token gets mcp-public and none get mcp-kubernetes.

The tempting shortcut — add mcp-public to the k3s API’s trusted audiences — is rejected: the k8s access token is passed through to the k3s API server, so trusting mcp-public there would let any realm user’s token authenticate to Kubernetes. That is a least-privilege violation; the k8s API is higher-stakes than engram precisely because the bearer reaches a real cluster.

Goal: make each MCP token’s aud reflect the specific resource it was requested for, so k8s trusts only k8s-scoped tokens.

Feasibility finding (grounded)

The bead’s original hypothesis was “make Keycloak 26 honor the RFC 8707 resource= parameter so aud = the requested resource.” Grounding disproved this:

Keycloak 26 does not map resource= to aud. It validates the RFC 8707 resource parameter (via ResourceIndicatorValidation, behind the non-default RESOURCE_INDICATORS feature; an unrecognized resource yields INVALID_TARGET), but the aud claim is populated only by Audience protocol mappers and the Audience Resolve mapper — never from resource=. (deepwiki keycloak/keycloak.) Enabling the feature would add validation friction without ever stamping the audience.
Keycloak does stamp per-resource audiences via optional client scopes. An Audience mapper attached to an optional client scope adds its audience to the access token only when that scope is explicitly requested via the scope parameter (TokenManager.getRequestedClientScopes). Dynamically-registered (DCR) clients inherit the realm’s default and optional client scopes, so a realm optional scope is requestable by DCR clients without per-client configuration. (deepwiki keycloak/keycloak.)
MCP clients forward advertised scopes. The MCP authorization spec requires the client to include all scopes from the resource’s RFC 9728 scopes_supported (or the WWW-Authenticate scope parameter, which takes priority) in its authorization request, with no distinction between well-known and custom scope strings. (deepwiki modelcontextprotocol/modelcontextprotocol.) This is already observed live for offline_access on the engram route (hl-o5w7).

Conclusion: native RFC 8707 audience-stamping is not available, but its intent — an audience scoped to the requested resource — is achievable by having the resource server advertise a resource-specific optional scope that the IdP binds to an audience. This is not an improvisation: it is the pattern Keycloak’s own MCP guidance documents (docs/guides/securing-apps/mcp-authz-server.adoc — an optional client scope carrying an Audience mapper, gated by the scope request). This is the design.

Mechanism

Route (mcp-kubernetes.yaml)                    Keycloak (fzymgc realm)
  resourceMetadata.scopesSupported:              optional client scope "mcp-kubernetes"
    - mcp-kubernetes  ───────────────┐             └─ Audience mapper → aud=https://mcp-gw.fzymgc.house/mcp/kubernetes
    - groups                         │           optional client scope "groups" (realm-optional)
    - offline_access                 │             └─ group-membership mapper → groups claim
                                     ▼
  DCR client (Claude Code — no hardcoded clientId)
    reads scopesSupported → authz request scope="mcp-kubernetes groups offline_access"
                                     ▼
  Keycloak access token: aud=https://mcp-gw.fzymgc.house/mcp/kubernetes + groups (+ email from blanket, Phase A)
                                     ▼
  agentgateway /mcp/kubernetes validates audiences:[https://mcp-gw.fzymgc.house/mcp/kubernetes]  (CHANGED from mcp-kubernetes)
  k3s k3s_oidc_audiences [kubernetes, https://mcp-gw.fzymgc.house/mcp/kubernetes] (MatchAny)      (CHANGED: replaces mcp-kubernetes)

A token minted for any other purpose never requests mcp-kubernetes, so it never carries that audience, so k3s rejects it. Per-resource least privilege — and it fails closed (a missing scope yields no audience → 401, never fail-open).

Scope of Phase A (this bead)

Phase A delivers the k8s leg only and unblocks hl-qffr.4. Engram and blanket-mapper removal are separate, dependency-ordered beads.

Changes:

The canonical audience is the resource URL https://mcp-gw.fzymgc.house/mcp/kubernetes (RFC 8707 alignment — it equals the value the client sends as resource= and the route’s already-declared resourceMetadata.resource). The client scope is still named mcp-kubernetes (that is the string advertised in scopesSupported and requested by the client); the scope name and the audience value are independent.

tf/keycloak — new optional client scope mcp-kubernetes with a keycloak_openid_audience_protocol_mapper (client_scope_id, included_custom_audience = "https://mcp-gw.fzymgc.house/mcp/kubernetes", access token only). This is the DCR-reachable audience mechanism.
tf/keycloak — promote groups to a realm optional scope. The groups client scope (client_scopes.tf) is today only a per-client default on the static clients (kubernetes.tf, mcp_kubernetes.tf), so DCR tokens carry no groups claim — which k8s RBAC needs. Adding it to the realm optional-scope list makes it requestable by DCR clients. Static clients that already carry it as a per-client default are unaffected (per-client and realm assignments are independent). (Contrast the email gap in mcp_email_claims.tf / hl-9az6, which was solved with a blanket mapper on basic. Here we deliver the claim narrowly via an MCP-advertised optional scope instead — because the MCP spec obligates clients to request advertised scopesSupported, a lever the email fix predated; the blanket email mapper is re-homed the same way in Phase C.)
tf/keycloak — realm optional-scope registration. Add mcp-kubernetes (and groups) to the realm’s optional client scopes. keycloak_realm_optional_client_scopes is authoritative (overwrites Keycloak defaults + manual adjustments), so the resource must enumerate the existing optional scopes too — a plan step reads the live realm optional-scope list first so nothing is dropped.
tf/keycloak — repoint the static fallback client’s audience to the URL. Change mcp_kubernetes.tf’s mapper from included_client_audience = "mcp-kubernetes" to included_custom_audience = "https://mcp-gw.fzymgc.house/mcp/kubernetes" so the fallback path mints the same URL audience. The client is kept (not deleted) as a fail-closed fallback, mirroring the mcp_public.tf “discovery anchor / fallback” precedent: if the live smoke test ever shows a client not forwarding the custom scope, the operator sets oauth.clientId = "mcp-kubernetes" in the MCP client config. Its fate is revisited in Phase C once DCR is proven.
argocd/app-configs/agentgateway/mcp-kubernetes.yaml — advertise the scopes AND move the validated audience to the URL. Add mcp-kubernetes and groups to traffic.jwtAuthentication.providers[].mcp.resourceMetadata.scopesSupported (alongside the existing offline_access), and change providers[].audiences from [mcp-kubernetes] to [https://mcp-gw.fzymgc.house/mcp/kubernetes].
ansible/roles/k3s-common/defaults/main.yml — trust the URL audience. Replace mcp-kubernetes with https://mcp-gw.fzymgc.house/mcp/kubernetes in k3s_oidc_audiences (the list stays multi-element, so audienceMatchPolicy: MatchAny is still emitted, per #1578). Also update the adjacent comment (lines 59–60) (“each must equal a Keycloak client_id”) — that is no longer true once an audience is a resource URL rather than a client id. Apply via the serial, health-gated k3s-common play (tags k3s-config, k3s-oidc-verify).

Downstream change is larger than the short-token option (the rejected alternative: keep aud=mcp-kubernetes as a plain opaque string — no gateway/k3s change, since both already validate that literal, but the token’s aud then does not name the resource). The URL choice pulls in the gateway route audience (item 5) and a k3s control-plane reroll (item 6). This is the accepted cost of RFC-8707-faithful audiences, and item 6 is the same class of change as hl-qffr.2 (which added mcp-kubernetes to k3s_oidc_audiences).

Apply order among the three systems is inconsequential. The Keycloak TF apply, the agentgateway ArgoCD sync, and the k3s Ansible play are triggered independently; because the design fails closed, every intermediate ordering leaves /mcp/kubernetes either still-401 or fully working — never a state where an unintended token is accepted. (k3s trusting the URL before any token carries it is inert; a token carrying the URL before the gateway/k3s trust it simply 401s, as today.)

Key decisions

Audience value = the resource URL https://mcp-gw.fzymgc.house/mcp/kubernetes (RFC 8707 alignment; user decision). It equals the resource= the client already sends and the route’s declared resourceMetadata.resource, so the token’s aud literally names the resource it is bound to. Trade-off: this requires changing the gateway route audience and rerolling k3s_oidc_audiences (a serial, health-gated k3s play) — a larger blast radius than the short token, accepted for spec fidelity and a self-descriptive token. The least-privilege property still comes from the scoping (audience present only when the resource scope is requested); the URL value additionally makes the binding self-descriptive.
groups bundled into the k8s advertisement — solves the DCR RBAC gap in the same mechanism, delivering groups only on k8s connections.
Blanket mcp-public + email mappers left untouched in Phase A. They still supply email/email_verified (engram fails closed without email_verified, memory hl-9az6) during Phase A; re-homing and removal is Phase C’s job, gated behind Phase B.

End-to-end verification (human gate)

The authenticated leg needs an interactive Keycloak PKCE login, so it is a human smoke test, not an automated check:

Pre-merge: terraform validate / fmt; kustomize build of the agentgateway app renders clean; yamllint.
Post-apply (operator, interactive): connect Claude Code to https://mcp-gw.fzymgc.house/mcp/kubernetes via the DCR flow (no hardcoded clientId), complete the Keycloak PKCE login, and confirm:
- the access token’s aud includes https://mcp-gw.fzymgc.house/mcp/kubernetes and the groups claim is present (decode the JWT);
- /mcp/kubernetes tools/list returns 200 (was 401);
- agentgateway pod logs (ns agentgateway) no longer show route:mcp-kubernetes … Error(InvalidAudience).
Confirm no regression: the same user’s engram calls still return 200 (blanket mapper untouched).

This test is also the definitive confirmation that Claude Code forwards the custom mcp-kubernetes scope in practice (the one residual the grounding could not fully close).

Risks & mitigations

Risk	Likelihood	Mitigation
Claude Code does not forward the custom `mcp-kubernetes` scope	Low (MCP spec + `offline_access` precedent)	Fails closed (k8s stays 401). Static `mcp_kubernetes.tf` client retained as documented `clientId` fallback.
`keycloak_realm_optional_client_scopes` authoritative resource drops an existing optional scope	Medium if list not enumerated	Plan step reads live realm optional scopes before writing; codified list = Keycloak defaults + `mcp-kubernetes` + `groups`.
k8s tool calls (not `tools/list`) fail RBAC without `groups`	Addressed	`groups` is bundled into the advertised scopes; smoke test asserts the claim is present.
DCR clients in this realm inherit fewer scopes than assumed	Low	Verified: DCR inherits realm default + optional scopes (deepwiki); plan Step 0 reads live realm scope config to confirm.
k3s control-plane reroll (adding the URL audience) breaks an apiserver at startup	Low–Medium	Serial, health-gated `k3s-common` play (`k3s-config`, `k3s-oidc-verify`); multi-element list keeps `audienceMatchPolicy: MatchAny` (#1578); a broken CP is recoverable via its node-local X.509 kubeconfig (`/etc/rancher/k3s/k3s.yaml`, OIDC-independent) — cf. the `hl-qffr.2` incident.

Out of scope (Phase A)

Engram migration to its own per-resource optional scope → hl-u18e (Phase B).
Removing the blanket mcp-public + email mappers from the basic scope → hl-05xz (Phase C, gated on Phase B).
The k8s MCP server / deployment itself — deployed and correct (hl-qffr.3/.5); only the token audience was wrong.
The Terraform MCP track (hl-qffr.6–.10) — independent, staged separately.