Plan: Central hosted Terraform + Kubernetes MCP servers (Phase 1)

Central Terraform + Kubernetes MCP Servers — Phase 1 Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Host the Kubernetes and Terraform MCP servers in-cluster behind agentgateway (mcp-gw.fzymgc.house), read-only, so any agent (local, cloud, cron) reaches them at a single authenticated endpoint.

Architecture: Reuse the existing per-server agentgateway pattern (AgentgatewayBackend + HTTPRoute + AgentgatewayPolicy). The k8s server uses Keycloak-OIDC identity passthrough (cluster_auth_mode = passthrough) so the k3s API server enforces each caller’s own RBAC. The Terraform servers use a single read-scoped TFC team token (registry docs need none). This is Phase 1 (read-only); Phase 2 (k8s operational writes) is a separate follow-up plan.

Tech Stack: Kustomize/ArgoCD (GitOps), agentgateway CRDs (agentgateway.dev/v1alpha1), Keycloak (Terraform hashicorp/keycloak), HCP Terraform (hashicorp/tfe ~> 0.78), Vault + External Secrets, Ansible (k3s), CiliumNetworkPolicy.

Scope boundary (out of this plan): Phase 2 k8s writes (enabled_tools allowlist, McpAuthorization CEL group-gate); refreshing stale docs (argocd/CLAUDE.md “Temporal” line, mcp-gateway-clients.md Authentik mentions). Tracked as follow-ups.

Model intent (for plan-to-beads): Tasks 5 and 9 (ArgoCD Application clones) and Task 7 (Terraform-MCP manifests following the firewalla/clickhouse skeleton) are mechanical, fully-specified diffs → model:haiku. Tasks 1, 3, 4, 6, 8 (HCL / agentgateway-policy / passthrough correctness), Task 2 (control-plane rolling restart), and Task 10 (operator apply + functional verification) require judgment → model:sonnet.

File structure

Path	Responsibility	New/Modify
`tf/keycloak/mcp_kubernetes.tf`	`mcp-kubernetes` public PKCE client + audience mapper + group/email scopes	New
`ansible/roles/k3s-common/defaults/main.yml`	add `mcp-kubernetes` to `k3s_oidc_audiences`	Modify
`argocd/app-configs/kubernetes-mcp/`	k8s MCP namespace, config, deployment, service, netpol	New (dir)
`argocd/app-configs/agentgateway/mcp-kubernetes.yaml`	backend + route + jwt/passthrough policies for `/mcp/kubernetes`	New
`argocd/app-configs/agentgateway/mcp-terraform.yaml`	2 backends + 2 routes + jwt policies for `/mcp/terraform-registry`, `/mcp/terraform-ro`	New
`argocd/app-configs/agentgateway/kustomization.yaml`	register the two new agentgateway files	Modify
`argocd/cluster-app/templates/kubernetes-mcp.yaml`	ArgoCD Application for the k8s MCP app	New
`argocd/cluster-app/templates/terraform-mcp.yaml`	ArgoCD Application for the TF MCP app	New
`tf/hcp-terraform/terraform_mcp_team.tf`	read-only `tfe_team` + `tfe_team_access` + `tfe_team_token`	New
`argocd/app-configs/terraform-mcp/`	TF registry + ro deployments, services, ESO, netpol	New (dir)
`docs/reference/services.md`, `docs/operations/mcp-gateway-clients.md`, `docs/getting-started/ai-tooling.md`	document the two new endpoints	Modify
`argocd/app-configs/velero/backup-schedule.yaml`	exclude the two stateless namespaces	Modify

Dependency order: Task 1→2 (client before k3s audience). Tasks 1–5 (k8s track) and 6–9 (TF track) are independent. Task 6 (TFC token in Vault) must complete before Task 7’s ESO resolves. Task 10 (docs/velero) last.

Task 1: Keycloak `mcp-kubernetes` client

Files:

Create: tf/keycloak/mcp_kubernetes.tf

Clones the mcp_public.tf pattern (public PKCE + audience mapper — required because the access token is forwarded to the k8s API and only carries aud via a mapper), plus the kubernetes.tf default-scopes list so the token carries groups (for RBAC) and email (username claim).

Step 1: Create the client, audience mapper, and scopes

# PUBLIC PKCE client for the hosted Kubernetes MCP server (agentgateway
# identity passthrough). The access token is forwarded to the k3s API, so it
# MUST carry aud=mcp-kubernetes (audience mapper, cf. mcp_public.tf) and the
# groups+email claims (default scopes, cf. kubernetes.tf) for RBAC. hl-qffr

resource "keycloak_openid_client" "mcp_kubernetes" {
  realm_id  = keycloak_realm.fzymgc.id
  client_id = "mcp-kubernetes"
  name      = "mcp-kubernetes"
  enabled   = true

  access_type                = "PUBLIC"
  standard_flow_enabled      = true
  pkce_code_challenge_method = "S256"

  valid_redirect_uris = [
    "http://127.0.0.1:*/*",
    "http://localhost:*/*",
  ]
}

# Access-token aud=mcp-kubernetes so both agentgateway jwtAuthentication
# (audiences: [mcp-kubernetes]) and the k3s API server (k3s_oidc_audiences)
# accept the forwarded bearer.
resource "keycloak_openid_audience_protocol_mapper" "mcp_kubernetes_aud" {
  realm_id                 = keycloak_realm.fzymgc.id
  client_id                = keycloak_openid_client.mcp_kubernetes.id
  name                     = "mcp-kubernetes-audience"
  included_client_audience = "mcp-kubernetes"
  add_to_id_token          = false
  add_to_access_token      = true
}

# groups (RBAC) + email (username claim) on the access token.
resource "keycloak_openid_client_default_scopes" "mcp_kubernetes" {
  realm_id       = keycloak_realm.fzymgc.id
  client_id      = keycloak_openid_client.mcp_kubernetes.id
  default_scopes = ["profile", "email", "roles", "web-origins", "acr", "basic", keycloak_openid_client_scope.groups.name]
}

Step 2: Validate

Run: terraform -chdir=tf/keycloak fmt -check && terraform -chdir=tf/keycloak validate Expected: Success! The configuration is valid.

Step 3: Commit

Commit per references/vcs-preamble.md: feat(keycloak): add mcp-kubernetes public client for hosted k8s MCP passthrough [hl-qffr]

Apply note: main-cluster-keycloak auto-applies on merge to main. Verify aud in the plan: expect 1 to add (client) + 2 to add (mapper, scopes) above the standing keycloak baseline (see hl-je6h memory: 9/9 provider-replacement cascade is normal).

Task 2: Add `mcp-kubernetes` to k3s OIDC audiences

Files:

Modify: ansible/roles/k3s-common/defaults/main.yml:61-62

The k3s API server rejects a forwarded token whose aud is not in k3s_oidc_audiences. Add mcp-kubernetes (multi-element list keeps the default MatchAny policy).

Step 1: Edit the audiences list

k3s_oidc_audiences:
  - "kubernetes"
  - "mcp-kubernetes"

Step 2: Syntax check

Run: ansible-playbook -i ansible/inventory/hosts.yml ansible/k3s-playbook.yml --syntax-check Expected: no errors.

Step 3: Dry-run the k3s-config change

Run: ansible-playbook -i ansible/inventory/hosts.yml ansible/k3s-playbook.yml --tags k3s-config --check --diff Expected diff: authentication-config.yaml gains the mcp-kubernetes audience entry on all 3 control-plane nodes; no other changes.

Step 4: Commit

feat(k3s): trust mcp-kubernetes audience for hosted MCP passthrough [hl-qffr]

Step 5: Operator apply (serial, health-gated)

Run: ansible-playbook -i ansible/inventory/hosts.yml ansible/k3s-playbook.yml --tags k3s-config,k3s-oidc-verify This uses the existing serial rolling-restart + OIDC health-gate handler (ansible/roles/k3s-common/tasks/main.yml, from ADR hl-idws). Expected: all 3 apiservers restart one at a time, OIDC smoke test green. Verify: the play’s own OIDC health-gate handler (ansible/roles/k3s-common/tasks/main.yml) passes — it hits /readyz with the node-local admin kubeconfig (anonymous /readyz returns 401) and checks stdout == 'ok'. A green --tags k3s-config,k3s-oidc-verify run is the gate; do not expect anonymous ?verbose to return readyz check passed.

Task 3: Kubernetes MCP app-config (deployment)

Files:

Create: argocd/app-configs/kubernetes-mcp/namespace.yaml
Create: argocd/app-configs/kubernetes-mcp/config.yaml (ConfigMap: TOML)
Create: argocd/app-configs/kubernetes-mcp/deployment.yaml
Create: argocd/app-configs/kubernetes-mcp/service.yaml
Create: argocd/app-configs/kubernetes-mcp/networkpolicy.yaml
Create: argocd/app-configs/kubernetes-mcp/kustomization.yaml
Step 1: Resolve and pin the server image

Run: crane digest ghcr.io/containers/kubernetes-mcp-server:latest (or read the digest from the repo’s GitHub Packages release page). Record the sha256:... for the deployment image: line below. Do not ship a floating tag.

Step 2: Namespace

apiVersion: v1
kind: Namespace
metadata:
  name: kubernetes-mcp

Step 3: TOML config (identity passthrough, read-only)

apiVersion: v1
kind: ConfigMap
metadata:
  name: kubernetes-mcp-config
  namespace: kubernetes-mcp
data:
  config.toml: |
    log_level = 2
    port = "8080"
    read_only = true

    # OAuth: require a bearer and pass it through so the k3s API server enforces
    # the caller's own RBAC. skip_jwt_verification=true means the server does NO
    # signature/aud/expiry check of its own (oauth_audience would be IGNORED, so
    # it is omitted). The gateway (jwtAuthentication Strict) and the k3s API server
    # both cryptographically validate the token — incl. aud against k3s_oidc_audiences.
    require_oauth = true
    skip_jwt_verification = true
    cluster_auth_mode = "passthrough"

    # Belt-and-suspenders: never expose Secrets, even to a token whose RBAC allows it.
    [[denied_resources]]
    group = ""
    version = "v1"
    kind = "Secret"

Step 4: Deployment

With skip_jwt_verification = true the server does not fetch Keycloak’s JWKS, so it needs no CA bundle and no egress to Keycloak — it only talks to the in-cluster k8s API using the forwarded caller token.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-mcp
  namespace: kubernetes-mcp
  labels:
    app: kubernetes-mcp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubernetes-mcp
  template:
    metadata:
      labels:
        app: kubernetes-mcp
    spec:
      automountServiceAccountToken: false   # passthrough uses the caller token, not the pod SA
      containers:
        - name: mcp-server
          image: ghcr.io/containers/kubernetes-mcp-server:<TAG>@sha256:<DIGEST>   # from Step 1
          args: ["--config=/config/config.toml"]
          ports:
            - containerPort: 8080
              name: http
          volumeMounts:
            - name: config
              mountPath: /config
              readOnly: true
          resources:
            requests: { cpu: 50m, memory: 64Mi }
            limits: { cpu: 200m, memory: 256Mi }
          securityContext:
            allowPrivilegeEscalation: false
            capabilities: { drop: ["ALL"] }
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1000
          readinessProbe:
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 15
            periodSeconds: 30
      volumes:
        - name: config
          configMap: { name: kubernetes-mcp-config }

Verify at implementation: the readiness/liveness path (/healthz vs /health) and the config flag (--config) against the pinned image’s --help. Adjust if the image differs.

Step 5: Service

apiVersion: v1
kind: Service
metadata:
  name: kubernetes-mcp
  namespace: kubernetes-mcp
spec:
  selector:
    app: kubernetes-mcp
  ports:
    - name: http
      port: 8080
      targetPort: 8080

Step 6: CiliumNetworkPolicy (ingress only from agentgateway; egress to k8s API + Keycloak)

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: kubernetes-mcp-default
  namespace: kubernetes-mcp
spec:
  endpointSelector: {}
  ingress:
    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: agentgateway
      toPorts:
        - ports: [{ port: "8080", protocol: TCP }]
  egress:
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports: [{ port: "53", protocol: UDP }]
    # k8s API server only — the server talks to the in-cluster kube-apiserver
    # using the forwarded caller token; no Keycloak egress (skip_jwt_verification).
    - toEntities: ["kube-apiserver"]

Verify: confirm the kube-apiserver entity selector against the cluster’s Cilium version (cilium.io/v2).

Step 7: Kustomization

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: kubernetes-mcp
resources:
  - namespace.yaml
  - config.yaml
  - deployment.yaml
  - service.yaml
  - networkpolicy.yaml

Step 8: Validate the render

Run: kubectl kustomize argocd/app-configs/kubernetes-mcp | yamllint -c .yamllint.yaml - Expected: clean render, no yamllint errors. Run: kubectl --context fzymgc-house apply --dry-run=server -k argocd/app-configs/kubernetes-mcp Expected: all resources created (server dry run), no schema errors.

Step 9: Commit

feat(kubernetes-mcp): read-only hosted k8s MCP server with OIDC passthrough [hl-qffr]

Task 4: agentgateway route for the k8s MCP server

Files:

Create: argocd/app-configs/agentgateway/mcp-kubernetes.yaml
Modify: argocd/app-configs/agentgateway/kustomization.yaml

Clones mcp-engram.yaml: backend + route (incl. the /.well-known/oauth-protected-resource match) + jwtAuthentication.mcp (audiences [mcp-kubernetes]) + a backend.auth.passthrough: {} policy that re-attaches the JWT the gateway strips after validation.

Step 1: Create the CRs

---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: mcp-kubernetes
  namespace: agentgateway
spec:
  mcp:
    targets:
      - name: kubernetes
        static:
          host: kubernetes-mcp.kubernetes-mcp.svc.cluster.local
          port: 8080
          path: /mcp
          protocol: StreamableHTTP
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: mcp-kubernetes
  namespace: agentgateway
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: agentgateway
      sectionName: mcp-gw
  hostnames:
    - mcp-gw.fzymgc.house
  rules:
    - matches:
        - path: { type: PathPrefix, value: /mcp/kubernetes }
        # Resource-specific well-known path (RFC 9728). NOTE: this deviates from
        # engram's bare /.well-known/oauth-protected-resource match on purpose — with
        # >1 OAuth route on one host, bare-prefix matches collide under Gateway API
        # precedence and a client discovering /mcp/kubernetes could be served engram's
        # metadata. The per-resource suffix keeps each route's discovery correct.
        - path: { type: PathPrefix, value: /.well-known/oauth-protected-resource/mcp/kubernetes }
      backendRefs:
        - group: agentgateway.dev
          kind: AgentgatewayBackend
          weight: 1
          name: mcp-kubernetes
---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: mcp-kubernetes-oauth
  namespace: agentgateway
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: mcp-kubernetes
  traffic:
    jwtAuthentication:
      mode: Strict
      providers:
        - issuer: "https://id.fzymgc.house/realms/fzymgc"
          audiences:
            - mcp-kubernetes
          jwks:
            remote:
              jwksPath: /realms/fzymgc/protocol/openid-connect/certs
              cacheDuration: 5m
              backendRef:
                group: ""
                kind: Service
                name: keycloak-service
                namespace: keycloak
                port: 8443
      mcp:
        provider: Keycloak
        resourceMetadata:
          resource: "https://mcp-gw.fzymgc.house/mcp/kubernetes"
          bearerMethodsSupported:
            - header
          scopesSupported:
            - offline_access
---
# Re-attach the validated JWT so the upstream (cluster_auth_mode=passthrough)
# receives the bearer; jwtAuthentication strips it after validation.
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: mcp-kubernetes-auth-passthrough
  namespace: agentgateway
spec:
  targetRefs:
    - group: agentgateway.dev
      kind: AgentgatewayBackend
      name: mcp-kubernetes
  backend:
    auth:
      passthrough: {}

Step 2: Register in the agentgateway kustomization

Modify argocd/app-configs/agentgateway/kustomization.yaml — add under resources: after mcp-firewalla.yaml:

  - mcp-kubernetes.yaml
  - mcp-terraform.yaml

(Adds both this task’s file and Task 8’s in one edit.)

Step 3: Validate

Run: kubectl kustomize argocd/app-configs/agentgateway | yamllint -c .yamllint.yaml - Expected: clean (both new files render; mcp-terraform.yaml must exist by Task 8 before this passes — if validating now, temporarily omit that line).

Step 4: Commit

feat(agentgateway): route /mcp/kubernetes with Keycloak JWT passthrough [hl-qffr]

Task 5: ArgoCD Application for the k8s MCP app

Files:

Create: argocd/cluster-app/templates/kubernetes-mcp.yaml

Clones firewalla-mcp.yaml (sync-wave 6, automated selfHeal+prune, CreateNamespace, ServerSideApply). No ExternalSecret here (passthrough → no upstream secret), so drop the ignoreDifferences block.

Step 1: Create the Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kubernetes-mcp
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "6"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/fzymgc-house/selfhosted-cluster
    targetRevision: HEAD
    path: argocd/app-configs/kubernetes-mcp
  destination:
    server: https://kubernetes.default.svc
    namespace: kubernetes-mcp
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

Step 2: Validate (Helm template of cluster-app)

Run: kubectl kustomize argocd/cluster-app 2>/dev/null || helm template argocd/cluster-app | yamllint -c .yamllint.yaml - Expected: the kubernetes-mcp Application renders. (Use whichever tool cluster-app is — check Chart.yaml vs kustomization.yaml.)

Step 3: Commit

feat(argocd): register kubernetes-mcp Application [hl-qffr]

Task 6: HCP Terraform read-only team + token → Vault

Files:

Create: tf/hcp-terraform/terraform_mcp_team.tf

A dedicated tfe_team with custom access runs=read, state_versions=none, variables=none across every workspace in local.all_workspaces, plus a tfe_team_token. The token is output sensitively and seeded into Vault by the operator (mirrors the manually-seeded cf_email_token/firewalla pattern — avoids the token landing in TF state via a vault_kv_secret_v2 write).

Step 1: Create the team, access, and token

# Read-only team for the hosted Terraform MCP server (terraform-ro). Custom
# access grants run OBSERVATION only — NOT state or variable read (closes the
# "plan permission grants full state read" gap) and NOT run creation (tf/CLAUDE.md:
# "MUST NOT manually trigger TFC runs via API or MCP tools"). hl-qffr

resource "tfe_team" "mcp_readonly" {
  name         = "mcp-readonly"
  organization = var.organization
  organization_access {
    read_workspaces = true
  }
}

resource "tfe_team_access" "mcp_readonly" {
  for_each     = tfe_workspace.this           # every workspace in local.all_workspaces
  team_id      = tfe_team.mcp_readonly.id
  workspace_id = each.value.id
  permissions {
    runs              = "read"
    variables         = "none"
    state_versions    = "none"
    sentinel_mocks    = "none"
    workspace_locking = false
    run_tasks         = false
  }
}

resource "tfe_team_token" "mcp_readonly" {
  team_id = tfe_team.mcp_readonly.id
}

output "mcp_readonly_team_token" {
  value     = tfe_team_token.mcp_readonly.token
  sensitive = true
}

Grounded: var.organization is the module’s existing org variable (tf/hcp-terraform/variables.tf, default fzymgc-house) — the same one tfe_workspace.this uses. The permissions {} argument names are verified against the hashicorp/tfe ~> 0.78 provider schema.

Step 2: Validate

Run: terraform -chdir=tf/hcp-terraform fmt -check && terraform -chdir=tf/hcp-terraform validate Expected: valid.

Step 3: Commit

feat(hcp-terraform): read-only mcp team + token for hosted TF MCP [hl-qffr]

Step 4: Operator apply + seed Vault

tf/hcp-terraform is Local execution — the operator runs it:

terraform -chdir=tf/hcp-terraform apply    # with Vault creds per hcp-terraform.md
vault kv put secret/fzymgc-house/cluster/terraform-mcp \
  tfe_token="$(terraform -chdir=tf/hcp-terraform output -raw mcp_readonly_team_token)"

Verify: vault kv get -field=tfe_token secret/fzymgc-house/cluster/terraform-mcp returns a token. No new Vault policy is needed: the external-secrets Vault auth role already grants a secret/data/* wildcard (tf/vault/policy-external-secrets-operator.tf), which covers the new cluster/terraform-mcp path.

Task 7: Terraform MCP app-config (registry + ro deployments)

Files:

Create: argocd/app-configs/terraform-mcp/namespace.yaml
Create: argocd/app-configs/terraform-mcp/external-secret.yaml
Create: argocd/app-configs/terraform-mcp/registry-deployment.yaml
Create: argocd/app-configs/terraform-mcp/ro-deployment.yaml
Create: argocd/app-configs/terraform-mcp/services.yaml
Create: argocd/app-configs/terraform-mcp/networkpolicy.yaml
Create: argocd/app-configs/terraform-mcp/kustomization.yaml
Step 1: Pin the image

Run: crane digest hashicorp/terraform-mcp-server:0.5.2 Record the digest for both deployments’ image: lines.

Step 2: Namespace + ExternalSecret

apiVersion: v1
kind: Namespace
metadata:
  name: terraform-mcp
---
# argocd/app-configs/terraform-mcp/external-secret.yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: terraform-mcp-secrets
  namespace: terraform-mcp
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: terraform-mcp-secrets
    creationPolicy: Owner
  data:
    - secretKey: tfe_token
      remoteRef:
        key: fzymgc-house/cluster/terraform-mcp
        property: tfe_token

Step 3: Registry deployment (no credentials)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: terraform-registry
  namespace: terraform-mcp
  labels: { app: terraform-registry }
spec:
  replicas: 1
  selector: { matchLabels: { app: terraform-registry } }
  template:
    metadata: { labels: { app: terraform-registry } }
    spec:
      automountServiceAccountToken: false
      containers:
        - name: mcp-server
          image: hashicorp/terraform-mcp-server:0.5.2@sha256:<DIGEST>
          args: ["--toolsets=registry"]
          ports: [{ containerPort: 8080, name: http }]
          env:
            - { name: TRANSPORT_MODE, value: streamable-http }
            - { name: TRANSPORT_HOST, value: "0.0.0.0" }
            - { name: TRANSPORT_PORT, value: "8080" }
          resources:
            requests: { cpu: 25m, memory: 64Mi }
            limits: { cpu: 200m, memory: 256Mi }
          securityContext:
            allowPrivilegeEscalation: false
            capabilities: { drop: ["ALL"] }
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1000
          readinessProbe: { httpGet: { path: /health, port: 8080 }, initialDelaySeconds: 5, periodSeconds: 10 }
          livenessProbe:  { httpGet: { path: /health, port: 8080 }, initialDelaySeconds: 15, periodSeconds: 30 }

Step 4: Read-only ops deployment (TFE token, no write ops)

ENABLE_TF_OPERATIONS is intentionally unset → create_run/apply tools are never registered.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: terraform-ro
  namespace: terraform-mcp
  labels: { app: terraform-ro }
spec:
  replicas: 1
  selector: { matchLabels: { app: terraform-ro } }
  template:
    metadata: { labels: { app: terraform-ro } }
    spec:
      automountServiceAccountToken: false
      containers:
        - name: mcp-server
          image: hashicorp/terraform-mcp-server:0.5.2@sha256:<DIGEST>
          args: ["--toolsets=registry,terraform"]
          ports: [{ containerPort: 8080, name: http }]
          env:
            - { name: TRANSPORT_MODE, value: streamable-http }
            - { name: TRANSPORT_HOST, value: "0.0.0.0" }
            - { name: TRANSPORT_PORT, value: "8080" }
            - { name: TFE_ADDRESS, value: "https://app.terraform.io" }
            - name: TFE_TOKEN
              valueFrom:
                secretKeyRef: { name: terraform-mcp-secrets, key: tfe_token }
          resources:
            requests: { cpu: 25m, memory: 64Mi }
            limits: { cpu: 200m, memory: 256Mi }
          securityContext:
            allowPrivilegeEscalation: false
            capabilities: { drop: ["ALL"] }
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1000
          readinessProbe: { httpGet: { path: /health, port: 8080 }, initialDelaySeconds: 5, periodSeconds: 10 }
          livenessProbe:  { httpGet: { path: /health, port: 8080 }, initialDelaySeconds: 15, periodSeconds: 30 }

Step 5: Services

apiVersion: v1
kind: Service
metadata: { name: terraform-registry, namespace: terraform-mcp }
spec:
  selector: { app: terraform-registry }
  ports: [{ name: http, port: 8080, targetPort: 8080 }]
---
apiVersion: v1
kind: Service
metadata: { name: terraform-ro, namespace: terraform-mcp }
spec:
  selector: { app: terraform-ro }
  ports: [{ name: http, port: 8080, targetPort: 8080 }]

Step 6: CiliumNetworkPolicy (ingress agentgateway; egress DNS + TFC over world:443)

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: terraform-mcp-default
  namespace: terraform-mcp
spec:
  endpointSelector: {}
  ingress:
    - fromEndpoints:
        - matchLabels: { io.kubernetes.pod.namespace: agentgateway }
      toPorts:
        - ports: [{ port: "8080", protocol: TCP }]
  egress:
    - toEndpoints:
        - matchLabels: { io.kubernetes.pod.namespace: kube-system, k8s-app: kube-dns }
      toPorts:
        - ports: [{ port: "53", protocol: UDP }]
    - toEntities: ["world"]      # app.terraform.io + registry.terraform.io over HTTPS
      toPorts:
        - ports: [{ port: "443", protocol: TCP }]

Step 7: Kustomization

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: terraform-mcp
resources:
  - namespace.yaml
  - external-secret.yaml
  - registry-deployment.yaml
  - ro-deployment.yaml
  - services.yaml
  - networkpolicy.yaml

Step 8: Validate

Run: kubectl kustomize argocd/app-configs/terraform-mcp | yamllint -c .yamllint.yaml - Then: kubectl --context fzymgc-house apply --dry-run=server -k argocd/app-configs/terraform-mcp Expected: clean render + created (server dry run).

Step 9: Commit

feat(terraform-mcp): read-only hosted Terraform MCP (registry + observe) [hl-qffr]

Task 8: agentgateway routes for the Terraform servers

Files:

Create: argocd/app-configs/agentgateway/mcp-terraform.yaml
(kustomization already updated in Task 4 Step 2)

Two backends + two routes + two jwtAuthentication policies. No passthrough (no OIDC-aware upstream) — gate on Keycloak JWT with audiences: [mcp-public] (the shared MCP client), so callers reuse their existing engram credential.

Step 1: Create the CRs

---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata: { name: mcp-terraform-registry, namespace: agentgateway }
spec:
  mcp:
    targets:
      - name: terraform-registry
        static:
          host: terraform-registry.terraform-mcp.svc.cluster.local
          port: 8080
          path: /mcp
          protocol: StreamableHTTP
---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata: { name: mcp-terraform-ro, namespace: agentgateway }
spec:
  mcp:
    targets:
      - name: terraform-ro
        static:
          host: terraform-ro.terraform-mcp.svc.cluster.local
          port: 8080
          path: /mcp
          protocol: StreamableHTTP
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: mcp-terraform-registry, namespace: agentgateway }
spec:
  parentRefs:
    - { group: gateway.networking.k8s.io, kind: Gateway, name: agentgateway, sectionName: mcp-gw }
  hostnames: ["mcp-gw.fzymgc.house"]
  rules:
    - matches:
        - path: { type: PathPrefix, value: /mcp/terraform-registry }
        # Resource-specific well-known path (RFC 9728) — required for OAuth
        # discovery; per-resource suffix avoids collision with the other routes.
        - path: { type: PathPrefix, value: /.well-known/oauth-protected-resource/mcp/terraform-registry }
      backendRefs:
        - { group: agentgateway.dev, kind: AgentgatewayBackend, weight: 1, name: mcp-terraform-registry }
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: mcp-terraform-ro, namespace: agentgateway }
spec:
  parentRefs:
    - { group: gateway.networking.k8s.io, kind: Gateway, name: agentgateway, sectionName: mcp-gw }
  hostnames: ["mcp-gw.fzymgc.house"]
  rules:
    - matches:
        - path: { type: PathPrefix, value: /mcp/terraform-ro }
        # Resource-specific well-known path (RFC 9728) — required for OAuth discovery.
        - path: { type: PathPrefix, value: /.well-known/oauth-protected-resource/mcp/terraform-ro }
      backendRefs:
        - { group: agentgateway.dev, kind: AgentgatewayBackend, weight: 1, name: mcp-terraform-ro }
---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata: { name: mcp-terraform-registry-oauth, namespace: agentgateway }
spec:
  targetRefs:
    - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: mcp-terraform-registry }
  traffic:
    jwtAuthentication:
      mode: Strict
      providers:
        - issuer: "https://id.fzymgc.house/realms/fzymgc"
          audiences: [mcp-public]
          jwks:
            remote:
              jwksPath: /realms/fzymgc/protocol/openid-connect/certs
              cacheDuration: 5m
              backendRef: { group: "", kind: Service, name: keycloak-service, namespace: keycloak, port: 8443 }
      mcp:
        provider: Keycloak
        resourceMetadata:
          resource: "https://mcp-gw.fzymgc.house/mcp/terraform-registry"
          bearerMethodsSupported: [header]
          scopesSupported: [offline_access]
---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata: { name: mcp-terraform-ro-oauth, namespace: agentgateway }
spec:
  targetRefs:
    - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: mcp-terraform-ro }
  traffic:
    jwtAuthentication:
      mode: Strict
      providers:
        - issuer: "https://id.fzymgc.house/realms/fzymgc"
          audiences: [mcp-public]
          jwks:
            remote:
              jwksPath: /realms/fzymgc/protocol/openid-connect/certs
              cacheDuration: 5m
              backendRef: { group: "", kind: Service, name: keycloak-service, namespace: keycloak, port: 8443 }
      mcp:
        provider: Keycloak
        resourceMetadata:
          resource: "https://mcp-gw.fzymgc.house/mcp/terraform-ro"
          bearerMethodsSupported: [header]
          scopesSupported: [offline_access]

Step 2: Validate

Run: kubectl kustomize argocd/app-configs/agentgateway | yamllint -c .yamllint.yaml - Expected: clean; both mcp-kubernetes.yaml and mcp-terraform.yaml now render (kustomization has both entries from Task 4).

Step 3: Commit

feat(agentgateway): routes /mcp/terraform-registry and /mcp/terraform-ro [hl-qffr]

Task 9: ArgoCD Application for the TF MCP app

Files:

Create: argocd/cluster-app/templates/terraform-mcp.yaml

Clones firewalla-mcp.yaml (keeps the ignoreDifferences ExternalSecret block, since this app has an ESO).

Step 1: Create the Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: terraform-mcp
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "6"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/fzymgc-house/selfhosted-cluster
    targetRevision: HEAD
    path: argocd/app-configs/terraform-mcp
  destination:
    server: https://kubernetes.default.svc
    namespace: terraform-mcp
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
  ignoreDifferences:
    - kind: ExternalSecret
      group: external-secrets.io
      jqPathExpressions:
        - .spec.data[].remoteRef.conversionStrategy
        - .spec.data[].remoteRef.decodingStrategy
        - .spec.data[].remoteRef.metadataPolicy

Step 2: Validate + Commit

Validate as Task 5 Step 2. Commit: feat(argocd): register terraform-mcp Application [hl-qffr]

Task 10: Velero exclusions, docs, and end-to-end verification

Files:

Modify: argocd/app-configs/velero/backup-schedule.yaml
Modify: docs/reference/services.md, docs/operations/mcp-gateway-clients.md, docs/getting-started/ai-tooling.md
Step 1: Exclude the two stateless namespaces from Velero

argocd/app-configs/velero/backup-schedule.yaml defines excludedNamespaces twice — once for the daily-backup Schedule and once for weekly-full-backup. Add kubernetes-mcp and terraform-mcp to both lists under an “MCP servers (stateless)” comment (per argocd/CLAUDE.md exclude-only strategy). Missing one leaves the namespaces backed up under that schedule.

Step 2: Document the endpoints

In docs/operations/mcp-gateway-clients.md add the three routes to the server table (kubernetes — OAuth mcp-kubernetes; terraform-registry/terraform-ro — OAuth mcp-public). In docs/getting-started/ai-tooling.md, add the claude mcp add --transport http lines. In docs/reference/services.md, note the two new namespaces.

Step 3: Commit

docs(mcp): document hosted kubernetes + terraform MCP endpoints; velero exclude [hl-qffr]

Step 4: Post-merge end-to-end verification (after ArgoCD syncs + keycloak/k3s applied)

# k8s route — read-only tools/list (OAuth mcp-kubernetes; use a Keycloak token)
curl -sS -X POST https://mcp-gw.fzymgc.house/mcp/kubernetes \
  -H "Authorization: Bearer <keycloak-jwt>" -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","method":"tools/list","id":1}' | jq '.result.tools[].name'

Expected: only read tools (pods_list, resources_get, events_list, …); NO pods_delete/resources_create_or_update.

# k8s passthrough backstop: a viewer-group token can list, an unauthorized verb 403s.
# Confirms the API server (not the gateway) is enforcing RBAC.

Expected: a resources_list on an allowed namespace succeeds; a denied namespace returns a 403 surfaced as a tool error.

# TF registry (no creds) + TF ro (token) tools/list
curl -sS -X POST https://mcp-gw.fzymgc.house/mcp/terraform-ro \
  -H "Authorization: Bearer <vk_claude_code-or-mcp-public-jwt>" -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","method":"tools/list","id":1}' | jq '.result.tools[].name'

Expected: list_runs, get_run_details, get_workspace_details, registry tools; NO create_run.

Step 5: Verify Secret-deny, audience acceptance, and TF token scope

Confirm a resources_get for a v1/Secret is denied by denied_resources, and that the k8s route returns tools (not 401 aud invalid) — proving mcp-kubernetes is accepted by both the gateway and k3s (gates any future Phase 2 write work). Also confirm the terraform-ro team token cannot read state: curl -sS https://app.terraform.io/api/v2/workspaces/<ws-id>/current-state-version -H "Authorization: Bearer <team-token>" returns 403/404 (state_versions=none), while .../runs returns run data — proving the read-scope decision holds in practice.

Follow-ups (not in this plan)

Phase 2 (k8s operational writes): flip to enabled_tools = [read… , pods_delete, pods_exec, resources_scale, pods_run], add an McpAuthorization CEL policy gating those four on jwt.groups ∈ {k8s-admins,k8s-developers}. Separate plan (ground the McpAuthorization CRD against the running gateway first).
Doc drift: correct the argocd/CLAUDE.md “Terraform GitOps automation runs through Temporal” line (real path is HCP auto-apply); refresh mcp-gateway-clients.md Authentik→Keycloak.

Plan: Central hosted Terraform + Kubernetes MCP servers (Phase 1)

Central Terraform + Kubernetes MCP Servers — Phase 1 Implementation Plan

File structure

Task 1: Keycloak mcp-kubernetes client

Task 2: Add mcp-kubernetes to k3s OIDC audiences

Task 3: Kubernetes MCP app-config (deployment)

Task 4: agentgateway route for the k8s MCP server

Task 5: ArgoCD Application for the k8s MCP app

Task 6: HCP Terraform read-only team + token → Vault

Task 7: Terraform MCP app-config (registry + ro deployments)

Task 8: agentgateway routes for the Terraform servers

Task 9: ArgoCD Application for the TF MCP app

Task 10: Velero exclusions, docs, and end-to-end verification

Follow-ups (not in this plan)

Task 1: Keycloak `mcp-kubernetes` client

Task 2: Add `mcp-kubernetes` to k3s OIDC audiences