Hosted Kubernetes MCP uses Keycloak-OIDC identity passthrough, not a shared ServiceAccount

Date: 2026-07-03 Status: Accepted Decision: hl-bc94 Deciders: Sean

Context

The cluster hosts MCP servers centrally behind agentgateway (ADR hl-nhi, per-server path-routed gateway). A hosted Kubernetes MCP server needs an upstream credential to reach the k8s API. The firewalla MCP (ADR hl-nmix) deliberately uses a single static shared upstream bearer — NOT per-user — because its upstream (Firewalla MSP) has no per-user identity. Kubernetes is different: every action can and should be bounded to the caller’s own identity, and k3s already trusts Keycloak OIDC (ADR hl-idws).

Decision

The hosted kubernetes-mcp-server runs with cluster_auth_mode = passthrough + require_oauth. agentgateway validates the caller’s Keycloak JWT (jwtAuthentication.mcp) and re-attaches it to the upstream (backend.auth.passthrough); the server builds an ephemeral k8s client from that token; the k3s API server enforces the caller’s own oidc: RBAC. No shared ServiceAccount token lives in the pod. A dedicated mcp-kubernetes Keycloak client is minted and its aud added to k3s_oidc_audiences.

Rationale

• k3s already trusts Keycloak OIDC (hl-idws), so the forwarded access token is a first-class k8s credential — per-user RBAC comes for free.
• The API server is an INDEPENDENT authorization backstop: a gateway-policy mistake cannot exceed what the caller’s own ClusterRoleBindings allow.
• No static credential at rest in the pod — nothing to steal.
• Deliberate contrast to hl-nmix: a shared bearer is correct for firewalla (no per-user upstream identity); passthrough is correct here because k8s HAS per-user identity.

Alternatives Considered

• Shared ServiceAccount + gateway CEL only (rejected as target): simpler and needs no audience wiring, but authz lives solely at the gateway, all writers share one identity in the audit trail, and it discards the free API-server backstop k3s OIDC already provides.
• Keep a local kubeconfig (rejected): cannot serve headless/cloud/cron sessions, and laptop kubeconfigs are often cluster-admin — more privileged than a scoped passthrough.

Consequences

• Positive: per-user RBAC + per-user audit; no pod-resident credential; the hosted path can be LESS privileged than a laptop kubeconfig.
• Negative: requires a dedicated mcp-kubernetes Keycloak client whose aud is added to k3s_oidc_audiences; token-audience alignment is a hard prerequisite before writes.
• Neutral: read-only in Phase 1; operational day-2 writes (Phase 2) add an McpAuthorization CEL group gate on top of the RBAC backstop.