Skip to content

Tandoor Recipes Decommission Implementation Plan

Tandoor Recipes Decommission Implementation Plan

Section titled “Tandoor Recipes Decommission Implementation Plan”

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Fully retire Tandoor Recipes (recipes.fzymgc.house) — remove its ArgoCD app, CNPG database, Authentik OIDC client, Vault paths/policies, agentgateway virtual key, uptime-kuma monitor, and docs — with the recipe data hard-deleted (no export).

Architecture: Two-PR drop-then-remove. PR1 empties the Tandoor app’s kustomization so its ArgoCD Application (prune: true) drains the workload / PVC / namespace, flips the CNPG Database + managed role to ensure: absent so the operator drops them, and removes the agentgateway virtual key. After PR1 is verified drained, PR2 removes the now-idle Application template, the CNPG manifests, all Terraform, and the docs. A manual Phase 3 scrubs operator-seeded Vault paths. The Tandoor Application has no resources-finalizer, so a single all-in-one PR would orphan the workloads — the two-PR split is a correctness requirement, not a preference.

Tech Stack: ArgoCD (app-of-apps + Kustomize), CloudNativePG, HashiCorp Vault + External Secrets, Terraform (HCP Terraform agent workspaces), Authentik, uptime-kuma.

Spec: docs/engineering/specs/2026-07-03-tandoor-decommission-design.md · Design bead: hl-nknn

Model intent (for plan-to-beads): default model:sonnet (infra changes with cluster-state verification nuance). The pure file-deletion / mechanical-edit tasks (Task 4, Task 5, Task 6) are model:haiku-suitable; the pre-flight, gate, and Phase-3 scrub tasks (Task 1, Task 3, Task 7) stay model:sonnet.


  • Repo VCS is colocated jj. Use jj for all VCS ops (jj describe, jj bookmark set, jj git push); never mutating git. Create commits via the commit-commands:commit skill. This work happens in the existing workspace .worktrees/hl-nknn-tandoor-decommission.
  • GitOps only. MUST NOT kubectl apply/delete directly — ArgoCD reconciles on merge. MUST NOT trigger TFC runs manually — HCP Terraform applies each workspace on merge to main.
  • Two PRs, strict order. PR2 MUST NOT merge until PR1’s drain is verified (Task 3). PR1 = branch feat/tandoor-decommission-drop; PR2 = branch feat/tandoor-decommission-remove.
  • Verification uses read-only tools: the Kubernetes MCP (live object state), the ClickStack MCP (logs), and read-only vault/terraform MCP calls. No mutating cluster access.

PR1 — modified (drop):

  • argocd/app-configs/tandoor/kustomization.yaml — emptied to resources: [] (renders zero manifests → Application prunes all children)
  • argocd/app-configs/tandoor/{namespace,middleware,secrets,deployment,service,certificate,ingress}.yaml — deleted (7 files)
  • argocd/app-configs/cnpg/db-tandoor.yamlensure: presentensure: absent
  • argocd/app-configs/cnpg/postgres-cluster.yaml — the tandoor managed-role block ensure: presentensure: absent
  • argocd/app-configs/agentgateway/secrets.yaml — remove the tandoor accept-list entry + the vk_tandoor remoteRef

PR2 — deleted / modified (remove):

  • argocd/cluster-app/templates/tandoor.yaml — deleted (the Application)
  • argocd/app-configs/tandoor/ — deleted (remaining kustomization.yaml)
  • argocd/app-configs/cnpg/{db-tandoor,users-tandoor}.yaml — deleted; dropped from cnpg/kustomization.yaml; tandoor managed-role block removed from postgres-cluster.yaml
  • tf/authentik/tandoor.tf — deleted (OAuth2 provider + application + random_password + vault_kv_secret_v2 cluster/tandoor)
  • tf/vault/policy-tandoor.tf, tf/vault/k8s-tandoor.tf — deleted
  • tf/vault/policy-terraform-workspaces.tf — remove the two cluster/tandoor read grants
  • tf/vault/policy-keycloak.tf — tidy the stale minus tandoor comment
  • tf/uptime-kuma/main.tf — remove uptimekuma_monitor_http.recipes + its stale comment
  • docs/reference/services.md, docs/reference/secrets.md, docs/operations/outbound-email.md, docs/operations/pushover.md — scrub Tandoor (these are the only four live docs that reference it; ~14 dated specs/plans/ADRs also mention it and are left as historical record)
  • docs/adr/hl-3328-retire-tandoor-rather-than-migrate-keycloak.md — the retirement ADR (written by the capture-adrs auto-fire step)

Phase 3 — manual (no repo change): Vault path deletes; PV / DB / role confirmation.


Task 1: Phase 0 — Pre-flight verification (gate; no commit)

Section titled “Task 1: Phase 0 — Pre-flight verification (gate; no commit)”

Hard delete was chosen — there is no export step. These read-only checks confirm the teardown is safe to start and capture the drop targets.

Files: none (verification only).

  • Step 1: Confirm the Longhorn storageClass reclaims the PV on prune

The PVC tandoor-data uses storageClassName: longhorn. Confirm that class reclaims (deletes) the PV when the PVC is pruned, so recipe media is actually freed.

Run (Kubernetes MCP resources_get, or read-only): kubectl get storageclass longhorn -o jsonpath='{.reclaimPolicy}' Expected: Delete. If it prints Retain, note it — Phase 3 (Task 7) must delete the released PV manually.

  • Step 2: Record the baseline drop targets

Run: kubectl get ns tandoor → expect Active; kubectl -n tandoor get deploy,pvc,ingressroute,certificate,externalsecret → record the objects that PR1 will prune; and via the CNPG pod psql -c '\l tandoor' / \du tandoor → confirm the tandoor DB + role currently exist. Expected: the namespace and objects exist; DB + role exist. This is the “before” snapshot for the Task 3 gate.

  • Step 3: Confirm no live use worth pausing for

Query recent access to recipes.fzymgc.house (ClickStack MCP over otel_logs/traefik access logs, last 7d). Hard delete is already the owner’s decision; this is a courtesy check, not a gate. Expected: whatever it returns is acceptable — proceed.

  • Step 4: Record findings on the bead

Run: bd note hl-nknn "Phase 0 pre-flight: longhorn reclaimPolicy=<value>; baseline objects recorded; proceeding with hard delete."


Task 2: PR1 — GitOps drop (drain workload + drop DB + remove vkey)

Section titled “Task 2: PR1 — GitOps drop (drain workload + drop DB + remove vkey)”

One PR that empties the Tandoor app source (ArgoCD prunes its resources), drops the CNPG DB + role, and removes the agentgateway virtual key.

Files:

  • Modify: argocd/app-configs/tandoor/kustomization.yaml
  • Delete: argocd/app-configs/tandoor/{namespace,middleware,secrets,deployment,service,certificate,ingress}.yaml
  • Modify: argocd/app-configs/cnpg/db-tandoor.yaml
  • Modify: argocd/app-configs/cnpg/postgres-cluster.yaml:88-95
  • Modify: argocd/app-configs/agentgateway/secrets.yaml
  • Step 1: Empty the Tandoor kustomization

Replace the entire contents of argocd/app-configs/tandoor/kustomization.yaml with:

---
# Tandoor decommission (hl-nknn), PR1 of 2: emptied so the tandoor ArgoCD
# Application (prune: true, selfHeal: true) prunes every previously-managed
# resource (namespace, Deployment, PVC, Service, IngressRoute, Certificate,
# ExternalSecrets). PR2 deletes this directory and the Application template.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: []
  • Step 2: Delete the seven child manifests

Run:

Terminal window
rm argocd/app-configs/tandoor/namespace.yaml \
argocd/app-configs/tandoor/middleware.yaml \
argocd/app-configs/tandoor/secrets.yaml \
argocd/app-configs/tandoor/deployment.yaml \
argocd/app-configs/tandoor/service.yaml \
argocd/app-configs/tandoor/certificate.yaml \
argocd/app-configs/tandoor/ingress.yaml
  • Step 3: Verify the app now renders empty

Run: kubectl kustomize argocd/app-configs/tandoor Expected: no output (empty render, exit 0). This is what makes ArgoCD prune all tracked resources.

  • Step 4: Flip the CNPG Database to ensure: absent

In argocd/app-configs/cnpg/db-tandoor.yaml, change the ensure value:

---
apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata:
name: tandoor
namespace: postgres
spec:
cluster:
name: main
ensure: absent
name: tandoor
owner: tandoor
  • Step 5: Flip the CNPG managed role to ensure: absent

In argocd/app-configs/cnpg/postgres-cluster.yaml, the tandoor managed-role block (lines 88-95) — change only its ensure:

- name: tandoor
ensure: absent
passwordSecret:
name: main-tandoor-credentials
superuser: false
login: true
connectionLimit: -1
inherit: true

Note (expected, harmless): the Database and role reconcilers run independently with no ordering guarantee. DROP ROLE tandoor fails with 2BP01 dependent_objects_still_exist until DROP DATABASE tandoor completes, and DROP DATABASE fails while the (now-pruning) Tandoor pods still hold connections. CNPG surfaces a transient cannotReconcile on the Cluster and retries until both drops succeed — do not treat it as a stuck teardown.

  • Step 6: Remove the agentgateway virtual key

In argocd/app-configs/agentgateway/secrets.yaml, delete the Tandoor accept-list entry (the comment + key, currently lines 35-37):

# tandoor AI recipe import/parsing via the LLM gateway, repointed off
# the decommissioned LiteLLM (hl-0sr.28). Endpoint/model: services.md.
tandoor: '{"key":"{{ .vk_tandoor }}","metadata":{"group":"service"}}'

and delete the matching data: remoteRef (currently lines 73-76):

- secretKey: vk_tandoor
remoteRef:
key: fzymgc-house/cluster/agentgateway
property: vk_tandoor

Also drop the word tandoor from the passthrough-isolation comment list (currently line 84: mealie, tandoor, octopus,mealie, octopus,) so no stale reference remains.

  • Step 7: Validate the two changed apps still build

Run:

Terminal window
kubectl kustomize argocd/app-configs/cnpg >/dev/null && echo cnpg-ok
kubectl kustomize argocd/app-configs/agentgateway >/dev/null && echo agentgateway-ok
yamllint argocd/app-configs/cnpg/db-tandoor.yaml argocd/app-configs/cnpg/postgres-cluster.yaml argocd/app-configs/agentgateway/secrets.yaml

Expected: cnpg-ok, agentgateway-ok, and yamllint clean (no errors).

  • Step 8: Confirm no dangling references remain in the changed apps

Run: rg -n -i 'tandoor' argocd/app-configs/agentgateway/ argocd/app-configs/cnpg/db-tandoor.yaml Expected: only db-tandoor.yaml (which PR2 deletes) and postgres-cluster.yaml’s ensure: absent block match; zero matches in agentgateway/.

  • Step 9: Commit (PR1)

Create branch/bookmark feat/tandoor-decommission-drop at @ and commit via the commit-commands:commit skill with message:

feat(tandoor): PR1 drop — drain workload, drop CNPG db+role, remove vkey [hl-nknn]
  • Step 10: Push and open PR1

Run: jj bookmark set feat/tandoor-decommission-drop -r @ && jj git push -b feat/tandoor-decommission-drop, then gh pr create --head feat/tandoor-decommission-drop --title "feat(tandoor): decommission PR1 — drop [hl-nknn]" --body "<summary + link to spec; note PR2 follows after drain verified>". Expected: PR created. Merge it (owner approval), then proceed to Task 3.


Task 3: PR1 verification gate (after merge + reconcile)

Section titled “Task 3: PR1 verification gate (after merge + reconcile)”

Confirm the drain fully completed before PR2. ArgoCD sync + CNPG drop may take a few reconcile cycles.

Files: none (verification only).

  • Step 1: Namespace + workloads gone

Run: kubectl get ns tandoor Expected: NotFound (may pass through Terminating first).

  • Step 2: PVC and PV reclaimed

Run: kubectl get pvc -A | rg tandoor and kubectl get pv | rg tandoor Expected: no matches. (If a Released PV lingers and Step-1-of-Task-1 showed Retain, record it for Task 7.)

  • Step 3: CNPG database + role dropped

Run (via the CNPG primary pod): psql -tAc "select 1 from pg_database where datname='tandoor'" → expect empty; psql -tAc "select 1 from pg_roles where rolname='tandoor'" → expect empty. Also confirm the Cluster is back to healthy (no lingering cannotReconcile): kubectl -n postgres get cluster main -o jsonpath='{.status.conditions}'. Expected: no tandoor DB or role; Cluster healthy.

Stuck-vs-transient bound: once Step 1 confirms the tandoor pods are gone, the drop resolves within ~2-3 reconcile cycles (a few minutes). If a cannotReconcile on the tandoor role persists > ~15 min with pods already gone, it is genuinely stuck — check for lingering sessions: psql -tAc "select count(*) from pg_stat_activity where datname='tandoor'" (expect 0); if non-zero, a client is still connected. As a last resort an operator may psql -c "DROP DATABASE tandoor WITH (FORCE)" then DROP ROLE tandoor.

  • Step 4: Virtual key gone from the live Secret

Run: kubectl -n agentgateway get secret agentgateway-vkeys -o jsonpath='{.data.tandoor}' Expected: empty (key absent).

  • Step 5: Host no longer routes

Run: curl -sS -o /dev/null -w '%{http_code}' https://recipes.fzymgc.house || echo unreachable Expected: DNS failure / unreachable / non-200 (the router-hosts entry auto-deregistered with the IngressRoute).

  • Step 6: Record the gate on the bead

Run: bd note hl-nknn "PR1 drain VERIFIED: ns/PVC/PV gone, CNPG db+role dropped, vkey absent, recipes.fzymgc.house dead. Clear to land PR2."


Task 4: PR2 (GitOps) — remove the Application template + CNPG manifests

Section titled “Task 4: PR2 (GitOps) — remove the Application template + CNPG manifests”

Now safe — the children are already pruned, so deleting the Application orphans nothing.

Files:

  • Delete: argocd/cluster-app/templates/tandoor.yaml
  • Delete: argocd/app-configs/tandoor/kustomization.yaml (then the empty dir)
  • Delete: argocd/app-configs/cnpg/db-tandoor.yaml, argocd/app-configs/cnpg/users-tandoor.yaml
  • Modify: argocd/app-configs/cnpg/kustomization.yaml
  • Modify: argocd/app-configs/cnpg/postgres-cluster.yaml (remove the role block)
  • Step 1: Delete the Application template and the app-config dir

Run: rm argocd/cluster-app/templates/tandoor.yaml && rm -rf argocd/app-configs/tandoor

  • Step 2: Delete the CNPG manifests

Run: rm argocd/app-configs/cnpg/db-tandoor.yaml argocd/app-configs/cnpg/users-tandoor.yaml

  • Step 3: Drop them from the CNPG kustomization

In argocd/app-configs/cnpg/kustomization.yaml, remove these two lines:

- db-tandoor.yaml
- users-tandoor.yaml
  • Step 4: Remove the tandoor managed-role block

In argocd/app-configs/cnpg/postgres-cluster.yaml, delete the entire tandoor role block (the ensure: absent version from Task 2):

- name: tandoor
ensure: absent
passwordSecret:
name: main-tandoor-credentials
superuser: false
login: true
connectionLimit: -1
inherit: true
  • Step 5: Validate the CNPG app + confirm no ArgoCD refs remain

Run:

Terminal window
kubectl kustomize argocd/app-configs/cnpg >/dev/null && echo cnpg-ok
rg -n -i 'tandoor' argocd/

Expected: cnpg-ok; rg returns nothing under argocd/.

  • Step 6: Commit

Commit via commit-commands:commit on branch feat/tandoor-decommission-remove (create it at @):

feat(tandoor): PR2 remove — delete ArgoCD app + CNPG manifests [hl-nknn]

Task 5: PR2 (Terraform) — remove Authentik, Vault, and uptime-kuma resources

Section titled “Task 5: PR2 (Terraform) — remove Authentik, Vault, and uptime-kuma resources”

Same branch/PR as Task 4. Each tf/ workspace applies on merge.

Files:

  • Delete: tf/authentik/tandoor.tf, tf/vault/policy-tandoor.tf, tf/vault/k8s-tandoor.tf
  • Modify: tf/vault/policy-terraform-workspaces.tf, tf/vault/policy-keycloak.tf, tf/uptime-kuma/main.tf
  • Step 1: Delete the three whole-file resources

Run: rm tf/authentik/tandoor.tf tf/vault/policy-tandoor.tf tf/vault/k8s-tandoor.tf

Deleting tf/authentik/tandoor.tf makes main-cluster-authentik destroy the OAuth2 provider, the application, the random_password, and the vault_kv_secret_v2 at secret/fzymgc-house/cluster/tandoor (desired — full retirement, no Keycloak co-writer).

  • Step 2: Remove the two Vault read grants for cluster/tandoor

In tf/vault/policy-terraform-workspaces.tf, delete the data grant (lines 125-127):

path "secret/data/fzymgc-house/cluster/tandoor" {
capabilities = ["create", "read", "update", "delete"]
}

and the metadata grant (lines 164-166):

path "secret/metadata/fzymgc-house/cluster/tandoor" {
capabilities = ["create", "read", "update", "delete", "list"]
}
  • Step 3: Tidy the stale keycloak-policy comment

In tf/vault/policy-keycloak.tf:24, drop the now-moot clause. Change:

# (policy-terraform-workspaces.tf:118-190), minus tandoor (retired, not migrated).

to:

# (policy-terraform-workspaces.tf:118-190).
  • Step 4: Remove the uptime-kuma monitor for recipes.fzymgc.house

In tf/uptime-kuma/main.tf, delete the recipes monitor and its misleading comment (lines 100-114):

# Name intentionally preserves the dashboard spelling ("Recipies") so the
# import adopts the existing monitor without a rename; the service is Mealie.
resource "uptimekuma_monitor_http" "recipes" {
name = "Recipies"
url = "https://recipes.fzymgc.house"
interval = 60
timeout = 30
retry_interval = 60
resend_interval = 0
max_retries = 0
active = true
accepted_status_codes = ["200-299"]
ignore_tls = false
}

Safe to delete: this monitor targets Tandoor’s dying host. Mealie is monitored separately by uptimekuma_monitor_http.mealie (tf/uptime-kuma/monitors-services.tf:73, http://mealie.mealie.svc.cluster.local:9000/api/app/about) — the “the service is Mealie” comment here is stale.

  • Step 5: Format and validate each touched Terraform module

Run:

Terminal window
terraform -chdir=tf/authentik fmt -check && terraform -chdir=tf/authentik validate
terraform -chdir=tf/vault fmt -check && terraform -chdir=tf/vault validate
terraform -chdir=tf/uptime-kuma fmt -check && terraform -chdir=tf/uptime-kuma validate

Expected: fmt clean, Success! The configuration is valid. for each. (If a module needs init, run terraform -chdir=<dir> init -backend=false first for local validate.)

  • Step 6: Confirm no Terraform references remain

Run: rg -n -i 'tandoor' tf/ Expected: nothing.

  • Step 7: Commit

Commit via commit-commands:commit (same PR2 branch):

feat(tandoor): PR2 remove — destroy Authentik client, Vault policy/role/grants, uptime-kuma monitor [hl-nknn]

Task 6: PR2 (Docs) — scrub every Tandoor reference

Section titled “Task 6: PR2 (Docs) — scrub every Tandoor reference”

Same PR2 branch. Removes all live doc references so the post-scrub rg sweep is clean.

Files:

  • Modify: docs/reference/services.md, docs/reference/secrets.md, docs/operations/outbound-email.md, docs/operations/pushover.md

  • Include: docs/adr/hl-3328-retire-tandoor-rather-than-migrate-keycloak.md (written by the capture-adrs auto-fire step; commit it here)

  • Step 1: services.md — remove all four loci

    • Delete the Quick-Reference index row (~L17): | [Tandoor](#tandoor) | recipes.fzymgc.house | tandoor | Application |
    • Delete the entire ### Tandoor detail section (~L162-187), including the “AI Provider” vk_tandoor provisioning steps.
    • Delete the CNPG “Databases” table row (~L569): | tandoor | tandoor | Tandoor Recipes |
    • In the OpenRouter-passthrough paragraph (~L282), drop tandoor from the example service-key list (karakeep, mealie, tandoor, octopus, …)(karakeep, mealie, octopus, …).
  • Step 2: secrets.md — remove both loci

    • Delete the cluster/tandoor row (~L47): | secret/fzymgc-house/cluster/tandoor | Tandoor app config | ... |
    • In the cluster/smtp row (~L45), drop Tandoor from the consumers note: the remaining consumers (Authentik, Tandoor) are decommissioningthe remaining consumer (Authentik) is decommissioning.
  • Step 3: outbound-email.md — update both loci

    • The cluster/smtp retirement gate (~L144): the decommissioning apps — Authentik, Tandoor — are gonethe decommissioning app — Authentik — is gone.
    • The Django SSL-migration row (~L92): | Django (Tandoor et al.) | ... | — since Tandoor is the named example and it’s gone, replace Tandoor et al. with a non-Tandoor descriptor, e.g. | Django apps | EMAIL_USE_TLSEMAIL_USE_SSL (”1”, not ”true”), port 465 |.
  • Step 4: pushover.md — drop Tandoor from the routing example

In docs/operations/pushover.md:23, the alert-routing example row names Tandoor. Change:

| `cluster-apps` | apps | tandoor, karakeep, user-facing apps (unused) |

to:

| `cluster-apps` | apps | karakeep, user-facing apps (unused) |
  • Step 5: Add the retirement ADR

The capture-adrs auto-fire step (run by the writing-plans skill before beads materialization) writes docs/adr/hl-3328-retire-tandoor-rather-than-migrate-keycloak.md documenting the retire-Tandoor decision. git add/commit that file as part of this task so it lands with the docs scrub.

  • Step 6: Lint the docs

Run: rumdl check docs/reference/services.md docs/reference/secrets.md docs/operations/outbound-email.md docs/operations/pushover.md docs/adr/hl-3328-retire-tandoor-rather-than-migrate-keycloak.md Expected: no errors.

  • Step 7: Confirm only dated historical records remain

Run: rg -l -i 'tandoor' docs/reference/ docs/operations/ docs/architecture/ docs/getting-started/ Expected: no matches — every live reference/ops doc is clean (services.md, secrets.md, outbound-email.md, pushover.md all scrubbed). A broader rg -l -i 'tandoor' docs/ will still list ~14 dated historical records under docs/engineering/specs/, docs/engineering/plans/, and docs/adr/ (plus this spec/plan and tools/docs-audit/groom-manifest.md) — those are intentionally preserved as historical record, not scrubbed.

  • Step 8: Commit, push, open PR2

Commit via commit-commands:commit:

docs(tandoor): PR2 remove — scrub services/secrets/outbound-email + retirement ADR [hl-nknn]

Then jj bookmark set feat/tandoor-decommission-remove -r @ && jj git push -b feat/tandoor-decommission-remove and gh pr create --head feat/tandoor-decommission-remove --title "feat(tandoor): decommission PR2 — remove [hl-nknn]" --body "<summary; note this MUST land after PR1 drain verified (Task 3)>". Expected: PR2 created. Merge after PR1 is verified drained.


Task 7: PR2 verification + Phase 3 manual scrub (after merge)

Section titled “Task 7: PR2 verification + Phase 3 manual scrub (after merge)”

Confirm the TFC applies and finish the operator-only cleanup.

Files: none (verification + manual Vault ops).

  • Step 1: Confirm the three TFC applies succeeded

Via the Terraform MCP (read-only get_run_details), confirm the merge-triggered runs for main-cluster-authentik, main-cluster-vault, and main-cluster-uptime-kuma applied cleanly. The authentik run destroys the OAuth2 provider + application + cluster/tandoor secret; the vault run destroys the policy, k8s role, and the two grants; the uptime-kuma run destroys the recipes monitor. Expected: three successful applies; a subsequent plan on each is clean (no drift).

  • Step 2: Delete the operator-seeded Postgres-user Vault path

secret/fzymgc-house/cluster/postgres/users/main-tandoor is operator-seeded — no Terraform or script writes it (verified by grep), so it is not destroyed by any apply and must be removed by hand.

Run: vault kv metadata delete secret/fzymgc-house/cluster/postgres/users/main-tandoor Expected: Success! Data deleted. Confirm: vault kv get secret/fzymgc-house/cluster/postgres/users/main-tandoorNo value found.

  • Step 3: Confirm the cluster/tandoor secret is gone

Run: vault kv get secret/fzymgc-house/cluster/tandoor Expected: No value found (destroyed by the authentik apply in Step 1). If it still exists, the authentik destroy didn’t run — investigate before proceeding.

  • Step 4: (Optional, low-priority) prune the dead vk_tandoor field

vk_tandoor is one field on the shared multi-field secret secret/fzymgc-house/cluster/agentgateway; nothing references it after PR1, so it’s inert dead data. Removing a single field requires a careful read-modify-write vault kv put of the whole secret — only do this if a clean audit is wanted, and re-put every other field unchanged. Otherwise leave it.

  • Step 5: Final cluster + PV confirmation

Re-run the Task 3 checks (ns tandoor NotFound; no tandoor PVC/PV; no tandoor DB/role). If Task 1 Step 1 reported Retain and a Released PV lingers, delete it now: kubectl delete pv <name>. Expected: nothing Tandoor-related remains in the cluster.

  • Step 6: Close the bead

Run: bd note hl-nknn "Phase 3 scrub complete: TFC applies confirmed, cluster/tandoor + postgres/users/main-tandoor Vault paths gone, PV/DB/role confirmed absent." then bd close hl-nknn --reason="Tandoor fully decommissioned (PR1 drop + PR2 remove merged; Vault/PV scrub done)."


  • Spec coverage: every §4 component-changes row maps to a task (app manifests → T2/T4; CNPG → T2/T4; agentgateway vkey → T2; Authentik+Vault+uptime-kuma → T5; docs → T6; Vault scrub + PV/DB confirmation → T7). The §5 two-PR sequencing and the §6 “PR1 is the point of no return” interlock are enforced by the Task 3 gate between PRs.
  • Data hard-delete: no export/backup step exists by design (D2); the CNPG ensure: absent drop (T2) + PVC prune are the destructive actions, verified in T3.
  • The one former open item is resolved in-plan: the main-tandoor Vault path is operator-seeded (grounded), handled manually in T7 Step 2.