agentgateway k8s-native (Gateway-API/CRD) Migration — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Replace the standalone agentgateway (local config.yaml) with the k8s-native Gateway-API/CRD control-plane model (agentgateway’s own controller + agentgateway.dev/v1alpha1 CRDs), at full feature parity, with per-server path-routed MCP and per-upstream auth choice.

Architecture: Install agentgateway’s self-contained controller (Helm) which registers GatewayClass agentgateway and XDS-drives data-plane pods from a Gateway + HTTPRoutes + AgentgatewayBackend/AgentgatewayPolicy CRDs. LLM proxy and a 6-server MCP gateway are expressed as backends/routes/policies; secrets via ESO from Vault. The standalone app is deleted at the end.

Tech Stack: Kubernetes Gateway API (gateway.networking.k8s.io/v1), agentgateway CRDs (agentgateway.dev/v1alpha1), Helm (OCI charts), ArgoCD, External Secrets Operator, cert-manager, Authentik (OIDC/JWT), Vault.

Spec: docs/engineering/specs/2026-06-06-agentgateway-k8s-native-design.md (decisions K1–K14). Design bead: hl-0sr.25.

Conventions for every task: validate manifests with kubectl --context fzymgc-house apply --dry-run=server -f <file> against the live CRDs before committing (spec risk mitigation); commit via jj (jj commit -m "..." on a feature bookmark; never push to main). All new manifests live under argocd/app-configs/.

File structure

Path	Responsibility
`argocd/app-configs/agentgateway-controller/`	Helm values for the controller + CRDs charts (kustomize/helm via ArgoCD)
`argocd/cluster-app/templates/agentgateway-controller.yaml`	ArgoCD Application for the controller
`argocd/app-configs/agentgateway/`	Replaced: Gateway, AgentgatewayParameters, Backends, HTTPRoutes, AgentgatewayPolicies, ESO, JWKS Service+ReferenceGrant, kustomization
`argocd/cluster-app/templates/agentgateway.yaml`	Existing ArgoCD Application (unchanged ref; new contents)

Note: the existing standalone manifests in argocd/app-configs/agentgateway/ (configmap.yaml, deployment.yaml, service.yaml, ingress.yaml, modern-auth.yaml, certificate.yaml, secrets.yaml, kustomization.yaml) are removed/replaced across Tasks 2–13. certificate.yaml, secrets.yaml, modern-auth.yaml, and the namespace are kept/adapted, not deleted.

Phase 1 — Control plane

Task 1: Install the agentgateway controller + CRDs

Files:

Create: argocd/app-configs/agentgateway-controller/kustomization.yaml
Create: argocd/app-configs/agentgateway-controller/namespace.yaml
Create: argocd/app-configs/agentgateway-controller/values-controller.yaml
Create: argocd/cluster-app/templates/agentgateway-controller.yaml
Modify: argocd/app-configs/velero/backup-schedule.yaml (exclude agentgateway-system)
Step 1: Confirm the published chart version + images before pinning.

Run:

helm show chart oci://ghcr.io/agentgateway/charts/agentgateway --version 0.0.2 2>&1 | head -20
helm show chart oci://ghcr.io/agentgateway/charts/agentgateway-crds --version 0.0.2 2>&1 | head -20

Expected: chart metadata for both. If 0.0.2 is not published, list tags via the GitHub release page and use the actual published version. Record the resolved appVersion (the data-plane/controller image tag) for Step 3.

Step 2: Create the controller namespace.

argocd/app-configs/agentgateway-controller/namespace.yaml:

---
apiVersion: v1
kind: Namespace
metadata:
  name: agentgateway-system
  labels:
    app.kubernetes.io/name: agentgateway-system

Step 3: Pin controller chart values (registry default cr.agentgateway.dev; pin image.tag to the resolved appVersion from Step 1 — prefer a digest once known).

argocd/app-configs/agentgateway-controller/values-controller.yaml:

image:
  registry: cr.agentgateway.dev
  tag: "<resolved-appVersion-from-step-1>"   # pin; replace with digest when known
  pullPolicy: IfNotPresent
controller:
  replicaCount: 1
  logLevel: info
  image:
    repository: controller
  xds:
    mode: tls
  service:
    enabled: true
    type: ClusterIP
proxy:
  image:
    repository: agentgateway
discoveryNamespaceSelectors: []   # watch all namespaces
inferenceExtension:
  enabled: false
monitoring:
  enabled: false

Step 4: Kustomization that pulls both Helm charts.

argocd/app-configs/agentgateway-controller/kustomization.yaml:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: agentgateway-system
resources:
  - namespace.yaml
helmCharts:
  - name: agentgateway-crds
    repo: oci://ghcr.io/agentgateway/charts
    version: "0.0.2"
    releaseName: agentgateway-crds
    namespace: agentgateway-system
  - name: agentgateway
    repo: oci://ghcr.io/agentgateway/charts
    version: "0.0.2"
    releaseName: agentgateway
    namespace: agentgateway-system
    valuesFile: values-controller.yaml

If ArgoCD’s kustomize helm support is not enabled cluster-wide, fall back to an ArgoCD Application with two sources (the two OCI charts) instead of helmCharts — see Step 6 alt.

Step 5: Velero-exclude the stateless controller namespace.

In argocd/app-configs/velero/backup-schedule.yaml, add under excludedNamespaces (Operators category):

    - agentgateway-system   # stateless agentgateway control plane (config is Git)

Step 6: ArgoCD Application for the controller.

argocd/cluster-app/templates/agentgateway-controller.yaml:

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: agentgateway-controller
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd.argoproj.io/sync-wave: "-1"   # CRDs + controller before the gateway app (wave 0)
spec:
  project: core-services   # match the sibling agentgateway app's project
  source:
    repoURL: https://github.com/fzymgc-house/selfhosted-cluster.git
    targetRevision: main
    path: argocd/app-configs/agentgateway-controller
  destination:
    server: https://kubernetes.default.svc
    namespace: agentgateway-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true   # large CRDs may exceed the client-side annotation limit

Alt (if helmCharts in kustomize is unavailable): replace source with two sources: entries (chart: agentgateway-crds and chart: agentgateway from repoURL: ghcr.io/agentgateway/charts, helm.valuesObject inline from Step 3).

Step 7: Commit.

jj commit -m "feat(agentgateway): install k8s-native controller + CRDs (hl-0sr.25)"

Step 8: After merge, verify the control plane.

kubectl --context fzymgc-house -n agentgateway-system get pods
kubectl --context fzymgc-house get crd | grep agentgateway.dev
kubectl --context fzymgc-house get gatewayclass agentgateway -o jsonpath='{.status.conditions[?(@.type=="Accepted")].status}{"\n"}'

Expected: controller pod Running; three CRDs (agentgatewaybackends, agentgatewaypolicies, agentgatewayparameters); GatewayClass agentgateway Accepted=True.

Phase 2 — Gateway foundation

Task 2: Gateway + AgentgatewayParameters

Files:

Create: argocd/app-configs/agentgateway/parameters.yaml
Create: argocd/app-configs/agentgateway/gateway.yaml
Modify: argocd/app-configs/agentgateway/certificate.yaml (keep; SANs already include all four hosts)
Step 1: AgentgatewayParameters (data-plane pod shape).

argocd/app-configs/agentgateway/parameters.yaml:

---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayParameters
metadata:
  name: agentgateway-params
  namespace: agentgateway
spec:
  logging:
    level: info
    format: json
  image:
    registry: cr.agentgateway.dev
    repository: agentgateway
    tag: "<resolved-appVersion>"   # match the controller's proxy image; pin to digest when known
    pullPolicy: IfNotPresent
  env:
    - name: TZ
      value: "America/Los_Angeles"
    - name: ADMIN_ADDR          # expose admin/UI listener (default is loopback-only) for Task 11
      value: "0.0.0.0:15000"
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: "1"
      memory: 512Mi

Step 2: Gateway with four hostnames (llm-gw, mcp-gw, agentgateway UI, litellm alias).

argocd/app-configs/agentgateway/gateway.yaml:

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: agentgateway
  namespace: agentgateway
  annotations:
    router-hosts.fzymgc.house/enabled: "true"
spec:
  gatewayClassName: agentgateway
  infrastructure:
    parametersRef:
      group: agentgateway.dev
      kind: AgentgatewayParameters
      name: agentgateway-params
  listeners:
    - name: llm-gw
      protocol: HTTPS
      port: 443
      hostname: "llm-gw.fzymgc.house"
      tls:
        mode: Terminate
        certificateRefs:
          - { group: "", kind: Secret, name: agentgateway-tls }
      allowedRoutes:
        namespaces: { from: Same }
    - name: litellm-alias
      protocol: HTTPS
      port: 443
      hostname: "litellm.fzymgc.house"   # transitional; removed at hl-0sr.21
      tls:
        mode: Terminate
        certificateRefs:
          - { group: "", kind: Secret, name: agentgateway-tls }
      allowedRoutes:
        namespaces: { from: Same }
    - name: mcp-gw
      protocol: HTTPS
      port: 443
      hostname: "mcp-gw.fzymgc.house"
      tls:
        mode: Terminate
        certificateRefs:
          - { group: "", kind: Secret, name: agentgateway-tls }
      allowedRoutes:
        namespaces: { from: Same }
    - name: ui
      protocol: HTTPS
      port: 443
      hostname: "agentgateway.fzymgc.house"
      tls:
        mode: Terminate
        certificateRefs:
          - { group: "", kind: Secret, name: agentgateway-tls }
      allowedRoutes:
        namespaces: { from: Same }

The agentgateway-tls cert (kept from the standalone, certificate.yaml) already has SANs llm-gw, mcp-gw, agentgateway, litellm.fzymgc.house.

Step 3: Dry-run validate.

kubectl --context fzymgc-house apply --dry-run=server -f argocd/app-configs/agentgateway/parameters.yaml -f argocd/app-configs/agentgateway/gateway.yaml

Expected: both ... (server dry run) accepted, no schema errors.

Step 4: Commit.

jj commit -m "feat(agentgateway): Gateway + AgentgatewayParameters (hl-0sr.25)"

Task 3: JWKS backend for Authentik (K14)

Files:

Create: argocd/app-configs/agentgateway/authentik-jwks.yaml

agentgateway’s XDS path cannot fetch a JWKS URL; the engram MCP policy’s jwks.remote.backendRef must point at a Service. Authentik’s server is authentik-server.authentik.svc; cross-namespace backendRef requires a ReferenceGrant in the authentik namespace.

Step 1: ReferenceGrant allowing the agentgateway namespace to reference the authentik-server Service.

argocd/app-configs/agentgateway/authentik-jwks.yaml:

---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: agentgateway-to-authentik-jwks
  namespace: authentik
spec:
  from:
    - group: agentgateway.dev
      kind: AgentgatewayPolicy
      namespace: agentgateway
  to:
    - group: ""
      kind: Service
      name: authentik-server

Step 2: Confirm the authentik-server Service name + port.

kubectl --context fzymgc-house -n authentik get svc authentik-server -o jsonpath='{.spec.ports[*].port}{"\n"}'

Expected: a port (commonly 80 or 9000). Record it for the engram policy (Task 8) jwks.remote.backendRef.port.

Step 3: Dry-run + commit.

kubectl --context fzymgc-house apply --dry-run=server -f argocd/app-configs/agentgateway/authentik-jwks.yaml
jj commit -m "feat(agentgateway): ReferenceGrant for Authentik JWKS backendRef (hl-0sr.25)"

Phase 3 — Secrets

Task 4: ESO — two secret shapes

Files:

Modify: argocd/app-configs/agentgateway/secrets.yaml (replace the standalone ESO)

Two distinct Secret shapes (spec K11): (a) the client VK accept-list for apiKeyAuthentication; (b) per-SaaS Authorization-key secrets for backend.auth; plus OpenRouter’s Authorization secret for the LLM backend.

Step 1: Replace secrets.yaml with the three ExternalSecrets.

argocd/app-configs/agentgateway/secrets.yaml:

---
# (a) Client virtual-key accept-list for apiKeyAuthentication.secretRef.
# Each data key is a client id; value is the per-key JSON {key, metadata}.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: agentgateway-vkeys
  namespace: agentgateway
spec:
  refreshInterval: 5m
  secretStoreRef: { name: vault, kind: ClusterSecretStore }
  target:
    name: agentgateway-vkeys
    creationPolicy: Owner
    template:
      type: Opaque
      data:
        claude-code: '{"key":"{{ .vk_claude_code }}","metadata":{"group":"human"}}'
        engram-embedder: '{"key":"{{ .vk_engram_embedder }}","metadata":{"group":"embedder"}}'
  data:
    - secretKey: vk_claude_code
      remoteRef: { key: fzymgc-house/cluster/agentgateway, property: vk_claude_code }
    - secretKey: vk_engram_embedder
      remoteRef: { key: fzymgc-house/cluster/agentgateway, property: vk_engram_embedder }
---
# (b) OpenRouter upstream auth (Authorization key) for the LLM backend.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: agentgateway-openrouter
  namespace: agentgateway
spec:
  refreshInterval: 5m
  secretStoreRef: { name: vault, kind: ClusterSecretStore }
  target:
    name: agentgateway-openrouter
    creationPolicy: Owner
    template:
      type: Opaque
      data:
        Authorization: "Bearer {{ .openrouter_api_key }}"
  data:
    - secretKey: openrouter_api_key
      remoteRef: { key: fzymgc-house/cluster/agentgateway, property: openrouter_api_key }
---
# (c) Per-SaaS MCP upstream auth (Authorization key each) for backend.auth.secretRef.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: agentgateway-mcp-upstreams
  namespace: agentgateway
spec:
  refreshInterval: 5m
  secretStoreRef: { name: vault, kind: ClusterSecretStore }
  target:
    name: agentgateway-mcp-exa
    creationPolicy: Owner
    template:
      type: Opaque
      data:
        Authorization: "Bearer {{ .exa_api_key }}"
  data:
    - secretKey: exa_api_key
      remoteRef: { key: fzymgc-house/cluster/agentgateway, property: exa_api_key }

Repeat the (c) pattern as separate ExternalSecrets producing Secrets agentgateway-mcp-firecrawl, agentgateway-mcp-context7, agentgateway-mcp-fal from firecrawl_api_key / context7_api_key / fal_api_key. (One ExternalSecret per target Secret keeps each backend’s auth isolated.)

Step 2: Dry-run + commit.

kubectl --context fzymgc-house apply --dry-run=server -f argocd/app-configs/agentgateway/secrets.yaml
jj commit -m "feat(agentgateway): ESO vkeys + openrouter + per-SaaS upstream secrets (hl-0sr.25)"

Phase 4 — LLM

Task 5: LLM backends (openrouter + ollama)

Files:

Create: argocd/app-configs/agentgateway/llm-backends.yaml
Step 1: openrouter + ollama AgentgatewayBackends with inline policies.ai (aliases, ZDR overrides, embeddings routes) + auth.

argocd/app-configs/agentgateway/llm-backends.yaml:

---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: openrouter
  namespace: agentgateway
spec:
  ai:
    provider:
      openai:
        model: openai/gpt-5.5        # default; aliases below remap per-request
      host: openrouter.ai
      port: 443
      pathPrefix: /api/v1            # OpenRouter's OpenAI-compatible base
  policies:
    auth:
      secretRef:
        name: agentgateway-openrouter   # reads "Authorization" key
    tls: {}                             # initiate TLS to openrouter.ai:443
    ai:
      modelAliases:
        or-claude-opus-4-7: anthropic/claude-opus-4.7
        or-claude-sonnet-4-6: anthropic/claude-sonnet-4.6
        or-claude-haiku-4-5: anthropic/claude-haiku-4.5
        or-gpt-5-5-pro: openai/gpt-5.5-pro
        or-gpt-5-5: openai/gpt-5.5
        or-gpt-5-4-mini: openai/gpt-5.4-mini
        or-gpt-5-3-codex: openai/gpt-5.3-codex
        or-gemini-2-5-pro: google/gemini-2.5-pro
        or-gemini-2-5-flash: google/gemini-2.5-flash
        or-deepseek-v4-pro: deepseek/deepseek-v4-pro
        or-deepseek-v4-flash: deepseek/deepseek-v4-flash
        or-deepseek-v4-flash-zdr: deepseek/deepseek-v4-flash
        or-glm-5-1: z-ai/glm-5.1
        or-glm-4-6: z-ai/glm-4.6
        or-kimi-k2-6: moonshotai/kimi-k2-6
        or-kimi-k2-thinking: moonshotai/kimi-k2-thinking
        or-minimax-m2-7: minimax/minimax-m2-7
        or-llama-4-maverick: meta-llama/llama-4-maverick
        or-qwen3-max: qwen/qwen3-max
        or-qwen3-coder-plus: qwen/qwen3-coder-plus
---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: ollama
  namespace: agentgateway
spec:
  ai:
    provider:
      openai:
        model: bge-m3
      host: 192.168.218.96
      port: 11434
      pathPrefix: /v1                  # Ollama OpenAI-compatible endpoints
  policies:
    ai:
      modelAliases:
        ollama/bge-m3: bge-m3          # embeddings; NO ollama/* chat wildcard (spec K5)

or-deepseek-v4-flash-zdr is a distinct alias to the same upstream model; the ZDR override (Task 6) is applied on its dedicated route, not here, so the non-zdr alias stays clean.

Step 2: Dry-run validate (catches alias/field errors against the live CRD).

kubectl --context fzymgc-house apply --dry-run=server -f argocd/app-configs/agentgateway/llm-backends.yaml

Expected: both backends accepted. If pathPrefix is rejected for openai provider, switch to path: /api/v1/chat/completions (mutually exclusive with pathPrefix) and add a second backend for embeddings — re-validate.

Step 3: Commit.

jj commit -m "feat(agentgateway): LLM backends openrouter + ollama with aliases (hl-0sr.25)"

Task 6: LLM routes + auth + transforms

Files:

Create: argocd/app-configs/agentgateway/llm-routes.yaml
Create: argocd/app-configs/agentgateway/llm-policies.yaml
Step 1: HTTPRoutes for chat/embeddings/models on llm-gw (+ litellm alias).

argocd/app-configs/agentgateway/llm-routes.yaml:

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm-chat
  namespace: agentgateway
spec:
  parentRefs:
    - { group: gateway.networking.k8s.io, kind: Gateway, name: agentgateway, sectionName: llm-gw }
    - { group: gateway.networking.k8s.io, kind: Gateway, name: agentgateway, sectionName: litellm-alias }
  hostnames: ["llm-gw.fzymgc.house", "litellm.fzymgc.house"]
  rules:
    - matches:
        - { path: { type: PathPrefix, value: /v1/chat/completions } }
        - { path: { type: PathPrefix, value: /v1/models } }
      backendRefs:
        - { group: agentgateway.dev, kind: AgentgatewayBackend, name: openrouter }
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm-embeddings
  namespace: agentgateway
spec:
  parentRefs:
    - { group: gateway.networking.k8s.io, kind: Gateway, name: agentgateway, sectionName: llm-gw }
    - { group: gateway.networking.k8s.io, kind: Gateway, name: agentgateway, sectionName: litellm-alias }
  hostnames: ["llm-gw.fzymgc.house", "litellm.fzymgc.house"]
  rules:
    - matches:
        - { path: { type: PathPrefix, value: /v1/embeddings } }
      backendRefs:
        - { group: agentgateway.dev, kind: AgentgatewayBackend, name: ollama }

Chat + models route to openrouter; embeddings route to ollama. (OpenRouter chat models are the alias set; the only Ollama path used is embeddings.)

Step 2: Policies — apiKey VKs (both routes), ZDR override (chat route), drop_params (embeddings route).

argocd/app-configs/agentgateway/llm-policies.yaml:

---
# Client virtual-key gate on both LLM routes.
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: llm-apikey
  namespace: agentgateway
spec:
  targetRefs:
    - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: llm-chat }
    - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: llm-embeddings }
  traffic:
    apiKeyAuthentication:
      mode: Strict
      secretRef:
        name: agentgateway-vkeys
---
# ZDR: inject provider.zdr=true for the dedicated zdr alias. Applied to the chat
# route's backend; gated to the zdr model via the alias being a separate request model.
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: llm-zdr
  namespace: agentgateway
spec:
  targetRefs:
    - { group: agentgateway.dev, kind: AgentgatewayBackend, name: openrouter }
  backend:
    ai:
      overrides:
        - field: provider
          value:
            zdr: true
            require_parameters: true
---
# drop_params: remove encoding_format from embeddings requests (Ollama rejects base64).
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: llm-drop-encoding-format
  namespace: agentgateway
spec:
  targetRefs:
    - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: llm-embeddings }
  traffic:
    transformation:
      request:
        body: 'json(request.body).filter(k, k != "encoding_format")'

VALIDATION GATE (spec risk): the backend.ai.overrides ZDR semantics apply to the whole openrouter backend, which would force ZDR on all OpenRouter models — wrong. During Step 3, verify whether overrides can be scoped per model/route (e.g. a dedicated backend for or-deepseek-v4-flash-zdr, or a route-scoped policy). If not scopable, split a dedicated openrouter-zdr AgentgatewayBackend (same provider, overrides only there) and route only or-deepseek-v4-flash-zdr to it. Likewise confirm the filter CEL map-drop idiom is supported (general-purpose grounding flagged it as unverified); if not, find the supported drop function or use .merge() to set encoding_format to a value Ollama accepts.

Step 3: Dry-run validate + resolve the two gates above.

kubectl --context fzymgc-house apply --dry-run=server -f argocd/app-configs/agentgateway/llm-routes.yaml -f argocd/app-configs/agentgateway/llm-policies.yaml

Expected: routes + policies accepted. Resolve the ZDR-scoping + CEL-drop questions before proceeding; adjust manifests and re-validate.

Step 4: Commit.

jj commit -m "feat(agentgateway): LLM routes + apiKey/ZDR/drop_params policies (hl-0sr.25)"

Task 7: Verify the LLM gateway live

Files: operational (no repo files).

Step 1: After the app syncs (Task 12), confirm rollout + run the suite (mirrors the standalone hl-0sr.10 checks):

VK=$(vault kv get -field=vk_claude_code secret/fzymgc-house/cluster/agentgateway)
B=https://llm-gw.fzymgc.house
# no-auth -> 401
curl -sS -o /dev/null -w '%{http_code}\n' -H 'Content-Type: application/json' \
  -d '{"model":"or-claude-haiku-4-5","messages":[{"role":"user","content":"hi"}],"max_tokens":3}' "$B/v1/chat/completions"
# authed chat -> pong
curl -sS -H "Authorization: Bearer $VK" -H 'Content-Type: application/json' \
  -d '{"model":"or-claude-haiku-4-5","messages":[{"role":"user","content":"say pong"}],"max_tokens":8}' "$B/v1/chat/completions"
# embeddings (base64 dropped) -> 1024-dim vector
curl -sS -H "Authorization: Bearer $VK" -H 'Content-Type: application/json' \
  -d '{"model":"ollama/bge-m3","input":"hello","encoding_format":"base64"}' "$B/v1/embeddings"

Expected: 401; pong; data[0].embedding length 1024. ZDR: send or-deepseek-v4-flash-zdr and confirm provider.zdr=true reaches OpenRouter (data-plane pod logs).

Phase 5 — MCP (per server)

Task 8: engram MCP (OAuth)

Files:

Create: argocd/app-configs/agentgateway/mcp-engram.yaml
Step 1: engram Backend + HTTPRoute + jwtAuthentication.mcp policy.

argocd/app-configs/agentgateway/mcp-engram.yaml:

---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: mcp-engram
  namespace: agentgateway
spec:
  mcp:
    targets:
      - name: engram
        static:
          host: memory-mcp.agent-memory.svc.cluster.local
          port: 8080
          path: /mcp
          protocol: StreamableHTTP
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: mcp-engram
  namespace: agentgateway
spec:
  parentRefs:
    - { group: gateway.networking.k8s.io, kind: Gateway, name: agentgateway, sectionName: mcp-gw }
  hostnames: ["mcp-gw.fzymgc.house"]
  rules:
    - matches:
        - { path: { type: PathPrefix, value: /mcp/engram } }
      backendRefs:
        - { group: agentgateway.dev, kind: AgentgatewayBackend, name: mcp-engram }
---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: mcp-engram-oauth
  namespace: agentgateway
spec:
  targetRefs:
    - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: mcp-engram }
  traffic:
    jwtAuthentication:
      mode: Strict                 # required with mcp{}
      providers:                   # EXACTLY ONE provider (required with mcp{})
        - issuer: "https://auth.fzymgc.house/application/o/mcp-public/"
          audiences: ["mcp-public"]
          jwks:
            remote:
              jwksPath: /application/o/mcp-public/jwks/
              backendRef:
                group: ""
                kind: Service
                name: authentik-server
                namespace: authentik          # cross-ns; allowed by Task 3 ReferenceGrant
                port: <authentik-server-port>  # from Task 3 Step 2
      mcp:
        clientId: "mcp-public"     # mock-DCR short-circuit (Authentik has no RFC 7591 DCR)
        # provider: omitted — Authentik is not Auth0/Keycloak/Okta; generic issuer/jwks used
        resourceMetadata:
          resource: "https://mcp-gw.fzymgc.house/mcp/engram"
          bearerMethodsSupported: '["header"]'

Step 2: Dry-run validate.

kubectl --context fzymgc-house apply --dry-run=server -f argocd/app-configs/agentgateway/mcp-engram.yaml

Expected: accepted. If jwks.remote.backendRef.namespace is unsupported, the ReferenceGrant + same-field is the documented cross-ns path; if the field truly lacks namespace, create a same-namespace ExternalName/Service mirror for authentik-server and reference that.

Step 3: Commit.

jj commit -m "feat(agentgateway): engram MCP backend/route/OAuth policy (hl-0sr.25)"

Task 9: SaaS MCP servers (VK client + per-target upstream auth)

Files:

Create: argocd/app-configs/agentgateway/mcp-saas.yaml
Step 1: Per-server Backend + HTTPRoute + Policy for exa/firecrawl/context7/fal (deepwiki = no upstream auth). Pattern shown for exa; replicate for the others.

argocd/app-configs/agentgateway/mcp-saas.yaml (exa shown; repeat blocks for firecrawl/context7/fal/deepwiki):

---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: mcp-exa
  namespace: agentgateway
spec:
  mcp:
    targets:
      - name: exa
        static:
          host: mcp.exa.ai
          port: 443
          path: /mcp?tools=web_search_exa,web_fetch_exa,web_search_advanced_exa
          protocol: StreamableHTTP
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: mcp-exa
  namespace: agentgateway
spec:
  parentRefs:
    - { group: gateway.networking.k8s.io, kind: Gateway, name: agentgateway, sectionName: mcp-gw }
  hostnames: ["mcp-gw.fzymgc.house"]
  rules:
    - matches:
        - { path: { type: PathPrefix, value: /mcp/exa } }
      backendRefs:
        - { group: agentgateway.dev, kind: AgentgatewayBackend, name: mcp-exa }
---
# Client auth (VK) on the route.
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: mcp-exa-apikey
  namespace: agentgateway
spec:
  targetRefs:
    - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: mcp-exa }
  traffic:
    apiKeyAuthentication:
      mode: Strict
      secretRef:
        name: agentgateway-vkeys
---
# Upstream auth (the SaaS key) injected to mcp.exa.ai.
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: mcp-exa-upstream
  namespace: agentgateway
spec:
  targetRefs:
    - { group: agentgateway.dev, kind: AgentgatewayBackend, name: mcp-exa }
  backend:
    auth:
      secretRef:
        name: agentgateway-mcp-exa   # reads "Authorization" key
    tls: {}                          # HTTPS to mcp.exa.ai:443

Replicate for: mcp-firecrawl (host mcp.firecrawl.dev, path /v2/mcp, secret agentgateway-mcp-firecrawl), mcp-context7 (host mcp.context7.com, path /mcp, secret agentgateway-mcp-context7), mcp-fal (host mcp.fal.ai, path /mcp, secret agentgateway-mcp-fal). For mcp-deepwiki (host mcp.deepwiki.com, path /mcp): include the Backend + HTTPRoute + apiKeyAuthentication (VK) + backend.tls but NO backend.auth (deepwiki needs no upstream key).

Step 2: Dry-run validate the full file.

kubectl --context fzymgc-house apply --dry-run=server -f argocd/app-configs/agentgateway/mcp-saas.yaml

Expected: all backends/routes/policies accepted.

Step 3: Commit.

jj commit -m "feat(agentgateway): SaaS MCP servers (exa/firecrawl/context7/fal/deepwiki) (hl-0sr.25)"

Task 10: Verify MCP live (per server)

Files: operational.

Step 1: After sync, per-server reachability + auth.

VK=$(vault kv get -field=vk_claude_code secret/fzymgc-house/cluster/agentgateway)
B=https://mcp-gw.fzymgc.house
# engram: no-auth -> 401 (OAuth-gated)
curl -sS -o /dev/null -w 'engram %{http_code}\n' "$B/mcp/engram"
# SaaS: with VK -> 200/initialize (per server)
for s in exa firecrawl context7 fal deepwiki; do
  printf '%s ' "$s"; curl -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $VK" "$B/mcp/$s"
done

Expected: engram 401 unauthenticated (then full OAuth round-trip via claude-code in Task 14); each SaaS returns a non-404 with the VK. Confirm engram’s end-to-end OAuth (DCR short-circuit → Authentik → JWT) by connecting claude-code.

Phase 6 — UI, cutover, decommission

Task 11: Admin UI (Authentik-gated)

Files:

Modify: argocd/app-configs/agentgateway/modern-auth.yaml (keep the corrected forward-auth middleware)
Create: argocd/app-configs/agentgateway/ui-ingress.yaml

The data-plane UI (admin port 15000) is exposed via a Traefik IngressRoute (reusing the working modern-auth Authentik forward-auth from hl-0sr; the Gateway listeners handle LLM/MCP, the UI keeps the proven Traefik+Authentik path).

Step 1: Confirm the data-plane Service name the controller provisions for the Gateway (commonly <gateway-name> in the gateway namespace):

kubectl --context fzymgc-house -n agentgateway get svc -l gateway.networking.k8s.io/gateway-name=agentgateway

Record the Service name + whether port 15000 is exposed (may require AgentgatewayParameters.spec.service overlay to add the admin port). If not exposed, add the port via the parameters service overlay (Task 2) and re-sync.

Step 2: IngressRoute for the UI host → data-plane admin port, gated by modern-auth.

argocd/app-configs/agentgateway/ui-ingress.yaml:

---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: agentgateway-ui
  namespace: agentgateway
  annotations:
    router-hosts.fzymgc.house/enabled: "true"
spec:
  entryPoints: [websecure]
  routes:
    - match: Host(`agentgateway.fzymgc.house`)
      kind: Rule
      middlewares:
        - name: modern-auth
      services:
        - name: <data-plane-service-from-step-1>
          port: 15000
  tls:
    secretName: agentgateway-tls

Step 3: Dry-run + commit.

kubectl --context fzymgc-house apply --dry-run=server -f argocd/app-configs/agentgateway/ui-ingress.yaml
jj commit -m "feat(agentgateway): UI IngressRoute behind Authentik forward-auth (hl-0sr.25)"

Task 12: Wire the new app + cut over hosts

Files:

Modify: argocd/app-configs/agentgateway/kustomization.yaml (list only the new CRD-based resources)
Modify: argocd/cluster-app/templates/agentgateway.yaml (ensure ServerSideApply=true; sync-wave 0)
Step 1: Rewrite the kustomization to the new resources (drop the standalone files):

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: agentgateway
resources:
  - namespace.yaml          # keep
  - certificate.yaml        # keep (agentgateway-tls, 4 SANs)
  - secrets.yaml            # ESO (Task 4)
  - parameters.yaml         # Task 2
  - gateway.yaml            # Task 2
  - authentik-jwks.yaml     # Task 3
  - llm-backends.yaml       # Task 5
  - llm-routes.yaml         # Task 6
  - llm-policies.yaml       # Task 6
  - mcp-engram.yaml         # Task 8
  - mcp-saas.yaml           # Task 9
  - modern-auth.yaml        # keep (Task 11)
  - ui-ingress.yaml         # Task 11

The standalone configmap.yaml, deployment.yaml, service.yaml, ingress.yaml are removed from the kustomization here (files deleted in Task 13).

Step 2: Ensure the agentgateway ArgoCD Application uses ServerSideApply=true (CRs + large CRDs), depends on the controller app (sync-wave 0 ≥ controller’s -1).
Step 3: Commit, open PR, merge; watch ArgoCD sync + Gateway status.

kubectl --context fzymgc-house -n agentgateway get gateway agentgateway -o jsonpath='{.status.conditions[?(@.type=="Programmed")].status}{"\n"}'
kubectl --context fzymgc-house -n agentgateway get pods

Expected: Gateway Programmed=True; a data-plane pod Running. Then run Task 7 + Task 10 verifications.

Task 13: Decommission the standalone agentgateway

Files:

Delete: argocd/app-configs/agentgateway/configmap.yaml
Delete: argocd/app-configs/agentgateway/deployment.yaml
Delete: argocd/app-configs/agentgateway/service.yaml
Delete: argocd/app-configs/agentgateway/ingress.yaml
Step 1: Remove the four standalone files (already dropped from the kustomization in Task 12). Confirm none of the new resources reference them.

rg -n 'configmap.yaml|deployment.yaml|ingress.yaml|service.yaml' argocd/app-configs/agentgateway/kustomization.yaml || echo "clean"

Expected: clean (the new kustomization lists only the new resources + kept files).

Step 2: Delete + commit.

rm argocd/app-configs/agentgateway/{configmap.yaml,deployment.yaml,service.yaml,ingress.yaml}
jj commit -m "chore(agentgateway): remove standalone manifests, superseded by CRD model (hl-0sr.25)"

Step 3: After merge, confirm ArgoCD pruned the standalone Deployment/ConfigMap/Service/IngressRoute (prune=true) and only the data-plane (controller-provisioned) pod remains:

kubectl --context fzymgc-house -n agentgateway get deploy,cm,svc,ingressroute

Expected: the old agentgateway Deployment + agentgateway-config ConfigMap + old IngressRoutes are gone; the new data-plane Deployment (controller-managed) + agentgateway-ui IngressRoute remain.

Task 14: Repoint clients + final verification + close-out

Files:

Modify: docs/reference/services.md (record the new MCP per-server URLs + the CRD model)
Step 1: Document the claude-code MCP config (per-server paths):

"engram":    { "type": "http", "url": "https://mcp-gw.fzymgc.house/mcp/engram" },          // OAuth
"exa":       { "type": "http", "url": "https://mcp-gw.fzymgc.house/mcp/exa",
               "headers": { "Authorization": "Bearer <vk_claude_code>" } },
"firecrawl-mcp": { "type": "http", "url": "https://mcp-gw.fzymgc.house/mcp/firecrawl",
               "headers": { "Authorization": "Bearer <vk_claude_code>" } },
"context7":  { "type": "http", "url": "https://mcp-gw.fzymgc.house/mcp/context7",
               "headers": { "Authorization": "Bearer <vk_claude_code>" } },
"fal-ai":    { "type": "http", "url": "https://mcp-gw.fzymgc.house/mcp/fal",
               "headers": { "Authorization": "Bearer <vk_claude_code>" } },
"deepwiki":  { "type": "http", "url": "https://mcp-gw.fzymgc.house/mcp/deepwiki",
               "headers": { "Authorization": "Bearer <vk_claude_code>" } }

LLM client (claude-code/codex): base URL https://llm-gw.fzymgc.house, key vk_claude_code.

Step 2: Full end-to-end — LLM (chat/embeddings/ZDR) per Task 7; each MCP server (engram OAuth round-trip; SaaS tool calls) per Task 10; UI loads behind Authentik.
Step 3: Update docs/reference/services.md with the new endpoints + the CRD deployment model; commit.
Step 4: Close-out — confirm hl-jd4 superseded (ADR addendum via capture-adrs), the standalone is gone, and LiteLLM decommission (hl-0sr.19–.21) can now drop the litellm.fzymgc.house listener alias (Task 2) when it runs.