Design: central hosted Terraform + Kubernetes MCP servers (read + scoped writes)

• Bead: hl-qffr
• Date: 2026-07-03
• Status: Draft (pending design-reviewer gate — round 2)
• Author: Sean (with Claude)
• Relates to: ADR hl-idws (k3s Keycloak OIDC), hl-m3c/hl-hpbv (agentgateway k8s-native), docs/operations/hcp-terraform.md, tf/CLAUDE.md (HCP Terraform Workflow), argocd/app-configs/agentgateway/

Question

Should the Terraform and Kubernetes MCP servers be hosted centrally in-cluster behind agentgateway (mcp-gw.fzymgc.house) instead of run locally per-workstation — and can that be done securely given the cluster’s GitOps and IaC invariants? Each server is evaluated separately. Drivers: consistency across agents, headless/shared access, less local setup friction.

Verdict

Yes to both — and “central vs local” is a false binary. The decisive driver is headless/shared access: a laptop-hosted MCP server cannot serve a cloud, cron, or teammate session because there is no laptop in the loop. Local hosting therefore cannot satisfy the requirement, independent of security. The cluster already carries every primitive to host both well:

Primitive	Already present	Source
Central MCP gateway w/ per-server routes	agentgateway @ mcp-gw.fzymgc.house	argocd/app-configs/agentgateway/
Read-only hosted-MCP precedent	firewalla-mcp, clickhouse-ro	argocd/app-configs/firewalla-mcp/, .../agentgateway/mcp-clickhouse.yaml
RO/RW capability-split precedent	clickhouse-ro + clickhouse-rw (separate pods, users, routes)	.../agentgateway/mcp-clickhouse.yaml
OAuth/JWT validation + identity passthrough	engram mcp-engram-oauth (jwtAuthentication.mcp, Keycloak issuer)	argocd/app-configs/agentgateway/mcp-engram.yaml
Per-tool authorization (allow/deny by identity)	McpAuthorization CEL (mcp.tool.name, jwt.groups)	agentgateway
k8s API trusts Keycloak	structured AuthenticationConfiguration, issuer id.fzymgc.house/realms/fzymgc, oidc:k8s-* CRBs	ansible/roles/k3s-common/ (ADR hl-idws)

Security posture is set by what upstream credential each server holds and what the caller’s identity is allowed to do — and the two servers answer that differently: the Kubernetes server can pass the caller’s identity through to a real per-user RBAC backstop; the Terraform server cannot, so it is scoped down to strict read-only (which the repo already mandates).

Decisions (capability posture)

Server	Posture	Enforcement (defense in depth)
Kubernetes	Read + operational day-2 writes: delete pod, exec, scale, run debug pod. Structurally excludes arbitrary declarative apply/patch/delete of managed manifests.	(1) Per-user k8s RBAC via identity passthrough (real backstop). (2) Gateway CEL group gate on write tools. (3) enabled_tools allowlist omits resources_create_or_update/resources_delete. (4) denied_resources: [Secret].
Terraform	Read-only: registry docs + observe runs/workspaces (list_runs, get_run_details, get_workspace_details, list_workspaces). No run creation of any kind (no plan_only, no apply).	(1) ENABLE_TF_OPERATIONS unset → run-creation tools never registered. (2) Dedicated TFC read team with state_versions=none. (3) Gateway CEL restricts route to authenticated callers.

Why these are correct, not just conservative:

• Kubernetes — the allowlist excludes exactly the two tools (resources_create_or_update, resources_delete) that could perform the declarative apply/patch/delete argocd/CLAUDE.md forbids (“MUST NOT apply kubectl changes directly”). The remaining writes are non-declarative, transient day-2 actions; if one (e.g. resources_scale) drifts a managed resource from Git, ArgoCD self-heal reverts it — the GitOps invariant functioning, not a violation.
• Terraform — tf/CLAUDE.md already states “MUST NOT manually trigger TFC runs via API or MCP tools — GitOps only”. create_run (even plan_only) triggers a run, so any write tier violates that invariant. Independently, HashiCorp documents that TFC “plan” permission “is equivalent to ‘write’ permission from a security standpoint … intended to prevent accidental runs, not malicious access” — so a plan-scoped token is not a confused-deputy boundary. Read-only is the only posture consistent with both facts. (Revises this design’s earlier “read + plan_only” pick, made before the tf/CLAUDE.md invariant was surfaced.)

Architecture

Both servers reuse the established per-server pattern — nothing novel in the control plane:

Client (Claude Code / cloud / cron)
  │  Authorization: Bearer <Keycloak JWT>          (LAN / Tailscale)
  ▼
agentgateway  (mcp-gw.fzymgc.house, MetalLB LB, LAN-scoped)
  ├─ HTTPRoute  /mcp/<server>            → AgentgatewayBackend (StreamableHTTP, path /mcp)
  ├─ jwtAuthentication.mcp              → validate iss/aud/jwks (Keycloak)
  ├─ McpAuthorization (CEL)             → filter tools/list, reject tools/call by jwt.groups
  └─ (k8s only) forward JWT upstream    → per-user RBAC at the API server
  ▼
MCP server Deployment (own namespace, hardened pod, NetworkPolicy)

Each server contributes: a namespace, a hardened Deployment (non-root, readOnlyRootFilesystem, drop ALL caps — copy clickhouse-mcp/firewalla-mcp), a Service, a NetworkPolicy (ingress only from agentgateway ns), an AgentgatewayBackend + HTTPRoute + policies, and (TF only) an ExternalSecret for the read token. New stateless namespaces are added to Velero excludedNamespaces (argocd/app-configs/velero/backup-schedule.yaml).

Component: Kubernetes MCP (containers/kubernetes-mcp-server)

Go server; StreamableHTTP + SSE; per-request Authorization Bearer; runs stateless (LB-friendly). Single deployment across phases — capability is separated by identity (passthrough RBAC + CEL), so no ro/rw pod split is needed (unlike clickhouse, whose split exists only because it uses one shared DB credential).

Identity passthrough (the security core):

• require_oauth: true, authorization_url: https://id.fzymgc.house/realms/fzymgc.
• Gateway jwtAuthentication.mcp validates the caller’s Keycloak JWT, then forwards the same bearer upstream. Manager.Derived() builds an ephemeral k8s client from the caller’s token, omitting all other credentials → the API server enforces the caller’s own oidc: RBAC. A gateway-policy mistake cannot exceed what that human’s ClusterRoleBindings allow. No static upstream credential lives in the pod.
• Audience: the forwarded token’s aud must be accepted by k3s (k3s_oidc_audiences). Decision: mint a dedicated public Keycloak client mcp-kubernetes in tf/keycloak (distinct from the local-CLI mcp-k8s alias in docs/getting-started/ai-tooling.md), and add mcp-kubernetes to k3s_oidc_audiences (ansible/roles/k3s-common/defaults/main.yml). Clients use mcp-kubernetes for the k8s route’s PKCE flow (engram-style).

Tool scoping (phase-dependent config on the one deployment):

• Phase 1: --read-only (only readOnlyHint=true tools).
• Phase 2: drop --read-only; set an explicit enabled_tools allowlist = read tools plus the operational write set pods_delete, pods_exec, resources_scale, pods_run. Excluded: resources_create_or_update and resources_delete (arbitrary declarative mutation — the “kubectl apply” class). --disable-destructive is not used (it carries destructiveHint=true and would disable pods_delete/pods_exec/resources_scale — the tools we want).
• denied_resources: [Secret] (deny Secret reads by GVK regardless of RBAC).

Gateway CEL (McpAuthorization): read tools → any authenticated sub; the four operational write tools → require jwt.groups.exists(g, g in ["k8s-admins", "k8s-developers"]). The groups claim is bare names (full_path=false, tf/keycloak/client_scopes.tf), not the k3s-internal oidc:-prefixed form — the prefix is added server-side by k3s claimMappings, so CEL matches the unprefixed name.

Component: Terraform MCP (hashicorp/terraform-mcp-server, pin ≥ 0.5.2 by digest)

Serves StreamableHTTP (TRANSPORT_MODE=streamable-http, /mcp, /health). No identity passthrough — one token. Split into two deployments (clickhouse ro/rw precedent):

1. terraform-registry — --toolsets=registry, no token. Zero-credential provider/module docs lookup. Route /mcp/terraform-registry, open to any authenticated caller. (This becomes the terraform skill’s docs-lookup path.)
2. terraform-ro — --toolsets=registry,terraform, ENABLE_TF_OPERATIONS unset, TFE_ADDRESS=https://app.terraform.io, TFE_TOKEN from Vault. With ENABLE_TF_OPERATIONS off, create_run/apply/discard tools are never registered — only list_runs/get_run_details/get_workspace_details/list_workspaces exist. Route /mcp/terraform-ro.

Token scope (closes the “plan grants full state read” gap): the terraform-ro TFE_TOKEN is a dedicated read team token, defined as code in tf/hcp-terraform via tfe_team + tfe_team_access with custom access runs = "read", state_versions = "none", variables = "none", for_each over local.all_workspaces (the source of truth in tf/hcp-terraform/workspaces.tf — the doc tables in tf/CLAUDE.md and docs/operations/hcp-terraform.md are both stale). The token can observe run status/logs but cannot download state versions or read sensitive variables. (First tfe_team/tfe_team_access in the repo; confirm provider permission-dial names at implementation.)

Security model — why “central” beats “local” here

Concern	Local	Central (this design)
Serves headless/cloud/cron	✗ impossible	✓ single authenticated endpoint
k8s privilege	often cluster-admin kubeconfig	per-user RBAC via passthrough — can be less than a laptop
Credential at rest	token/kubeconfig on every laptop	k8s: none (passthrough); TF: one Vault read token, never on client
Blast radius	distributed	k8s: bounded by per-user RBAC + CEL + tool allowlist; TF: read-only, no state read
Confused-deputy	n/a	k8s: eliminated (passthrough); TF: shrunk to read observation, state_versions=none
Audit	per-machine, ad hoc	centralized at gateway (jwt.sub) + k8s audit log
Attack surface	laptop	LAN/Tailscale-scoped LB (not public)

Phasing

• Phase 1 — read-only both. kubernetes-mcp (--read-only, passthrough + mcp-k8s client + aud), terraform-registry + terraform-ro. Ships fast, matches firewalla/clickhouse-ro, unblocks the consistency + headless + friction drivers for the read-heavy majority.
• Phase 2 — k8s operational writes. Swap --read-only for the enabled_tools allowlist + CEL group gate. No Terraform Phase 2 (read-only is the end state).

Network, secrets, exposure

• Exposure: mcp-gw is a MetalLB LAN LoadBalancer (HostMapping DNS); not a public Cloudflare tunnel. Headless/cloud access is via Tailscale (mind the documented dual-home EHOSTUNREACH route fix).
• CA trust (k8s server): validating the forwarded token against in-cluster id.fzymgc.house (split-horizon ICA1) requires the combined ICA1 + Mozilla CA bundle; failures are restart-latent. Mount the existing combined bundle.
• Vault: TF read token at secret/fzymgc-house/cluster/terraform-mcp (new; add the Vault policy grant) → ExternalSecret. k8s server needs no Vault secret (passthrough).
• NetworkPolicy: ingress only from agentgateway ns; egress k8s API (k8s), Keycloak (token validation), app.terraform.io (TF read).

TF apply path (corrected)

The Terraform apply invariant to preserve is HCP Terraform’s own auto-apply-on-merge, not a Temporal step: docs/operations/hcp-terraform.md documents GitHub PR → HCP Terraform → Agent Pod → Dynamic Credentials → Apply (auto_apply = true), and “HCP Terraform replaced Windmill for Terraform execution (December 2025).” (argocd/CLAUDE.md’s “runs through Temporal workflows” line is stale re Terraform and should be corrected in a follow-up.) The read-only TF server does not touch this path at all.

Error handling / failure modes

• Gateway auth fail → 401 + WWW-Authenticate (engram-style); client re-auths.
• k8s RBAC denies a passthrough call → API server 403, surfaced as a tool error — the backstop working as intended.
• CEL misconfig (fail-open) → mitigated by the independent k8s per-user RBAC layer; authz never relies on CEL alone.
• Audience mismatch (k8s) → API server rejects the forwarded token; caught in Phase-1 verification (below) before any write posture.
• TF token over-scoped → prevented at source: runs=read, state_versions=none.

Testing / verification

• Phase 1: tools/list per route returns only read tools; a mutating tools/call is absent/denied. curl smoke test per docs/operations/mcp-gateway-clients.md.
• k8s audience: a real Keycloak-authenticated kubectl-equivalent call through the server succeeds (proves mcp-kubernetes aud is accepted by k3s) — gate Phase 2 on this.
• k8s passthrough (Phase 2): a k8s-viewers-only user gets 403 on pods_delete; a k8s-admins user succeeds — proving the API-server backstop, not just CEL.
• k8s Secrets deny: denied_resources blocks a Secret read regardless of RBAC.
• k8s no-apply: confirm resources_create_or_update/resources_delete are absent from tools/list.
• TF read-only: confirm no create_run tool exists; confirm the token cannot download a state version (state_versions=none).

Alternatives considered

• CEL-only, no k8s passthrough (shared ServiceAccount): simpler, no audience work, but writers share one SA and authz lives only at the gateway — discards the free API-server backstop k3s OIDC already provides. Rejected; passthrough is strictly better here.
• TF read + plan_only: rejected — violates tf/CLAUDE.md “MUST NOT trigger runs via MCP,” and TFC plan-permission is not a security boundary (HashiCorp). Would require amending a documented invariant.
• k8s full write (expose resources_create_or_update/delete): rejected — enables the declarative apply/patch argocd/CLAUDE.md forbids; ArgoCD would fight it via self-heal anyway.
• Keep local-only: rejected — cannot serve headless/shared access at all.

Non-goals

• Replacing ArgoCD (k8s managed state) or HCP Terraform auto-apply (TF managed state) as the mutation path.
• Public-internet exposure of mcp-gw.
• Per-user Terraform identity (the server does not support it; read-only makes it moot).
• Migrating other MCP servers, or refreshing the stale mcp-gateway-clients.md / argocd/CLAUDE.md / RBAC-comment references (tracked as follow-up housekeeping).