Tandoor Recipes Decommission Design

Design bead: hl-nknn · Date: 2026-07-03 · Author: Sean Brandt (+ Claude)

Predecessors (reuse their drop-then-remove idiom, scaled down): Grafana-Labs sunset (epic hl-nlxl, DONE) and Prometheus decommission (epic hl-qjh3). Those needed multi-wave parity playbooks because downstream systems consumed their output (metrics, alerting). Tandoor has zero downstream consumers — nothing in the cluster reads its recipes — so this is a much smaller teardown: no parity waves, just pre-flight → drop → remove → scrub.

Decisions resolved (owner, 2026-07-03): (1) reuse hl-nknn as the design bead; (2) hard delete the recipe data — no export, no archive, no Mealie migration; (3) two-PR drop-then-remove teardown. A single all-in-one PR is intentionally not used — the Application lacks resources-finalizer, so removing its template in one shot would orphan the running workloads (see §5).

1. Context

Tandoor Recipes (recipes.fzymgc.house, image vabene1111/recipes:2.6.9) was slated for decommission on 2026-06-29 during Keycloak Phase-2 scoping (epic hl-5g05). It is therefore excluded from the Authentik→Keycloak OIDC migration — no keycloak_openid_client was ever built for it; it keeps authenticating against Authentik until retirement (parallel-run, harmless). The cluster already runs Mealie, a second recipe manager, which is the natural successor.

The decision was undocumented until now (hl-nknn tracks it); this spec plus its ADR close that gap.

1.1 Footprint (grounded 2026-07-03)

Tandoor spans four layers — a plain file-delete would orphan running workloads and leak Vault paths:

• GitOps (ArgoCD) — argocd/app-configs/tandoor/ (namespace, Deployment + PVC tandoor-data 10Gi Longhorn mounted at /opt/recipes/mediafiles, Service, Traefik IngressRoute, Middleware, cert-manager Certificate tandoor-tls, two ExternalSecrets); the Application template argocd/cluster-app/templates/tandoor.yaml; an accept-list entry + vk_tandoor virtual key in argocd/app-configs/agentgateway/secrets.yaml.
• CNPG Postgres — Database tandoor (db-tandoor.yaml), managed role tandoor (users-tandoor.yaml + a managedRoles block in postgres-cluster.yaml) on cluster main.
• Terraform — tf/authentik/tandoor.tf (OAuth2 provider + application + random_password + vault_kv_secret_v2 at secret/fzymgc-house/cluster/tandoor); tf/vault/policy-tandoor.tf; tf/vault/k8s-tandoor.tf; Vault read-grants in tf/vault/policy-terraform-workspaces.tf (lines ~125, ~164); a stale comment in tf/vault/policy-keycloak.tf:24; the uptimekuma_monitor_http.recipes monitor in tf/uptime-kuma/main.tf.
• Docs — docs/reference/services.md (three loci: the Quick-Reference index row, the ### Tandoor detail section incl. AI-provider provisioning, and the CNPG “Databases” table row, plus a tandoor mention in the OpenRouter-passthrough service-key example), docs/reference/secrets.md (cluster/tandoor row + the cluster/smtp consumers note), docs/operations/outbound-email.md (the cluster/smtp retirement gate that names Tandoor + the “Django (Tandoor et al.)” SSL-migration row), and docs/operations/pushover.md (the alert-routing example row). These four are the only live docs referencing Tandoor; ~14 dated specs/plans/ADRs also mention it and are left as historical record.

1.2 DNS is annotation-driven (no DNS resource to remove)

recipes.fzymgc.house / recipes.k8s.fzymgc.house are registered by the router-hosts controller watching the IngressRoute annotation router-hosts.fzymgc.house/enabled: "true" — not a Cloudflare/Terraform DNS record (grep of tf/cloudflare and tf/router-hosts finds nothing). When the IngressRoute is pruned the host auto-deregisters; no explicit DNS teardown step is needed.

2. Decisions

#	Decision	Choice	Status
D1	Migrate or retire?	Retire — Mealie is the successor; do not build a Keycloak client.	Settled (hl-5g05, 2026-06-29)
D2	Recipe-data disposition	Hard delete — no export, no archive, no Mealie migration. Drop the CNPG database + role (D4) and delete the PVC/PV; recipe data is discarded. Accept irreversibility.	Settled (owner, 2026-07-03)
D3	Teardown shape	Two-PR drop-then-remove (PR1 prunes workloads + drops DB; PR2 removes the Application template + all Terraform + docs). Bracketed by a manual runbook.	Settled (owner, 2026-07-03)
D4	CNPG data drop mechanism	Flip ensure: absent on the Database + managed role in PR1 so the operator issues DROP DATABASE/DROP ROLE (GitOps-auditable), verify, then remove the manifests in PR2. Default databaseReclaimPolicy: retain would otherwise leave the data behind.	Settled (grounded)
D5	Vault cluster/tandoor secret	Destroy (full retirement) by deleting tf/authentik/tandoor.tf. No dueling-writer risk — unlike the Phase-2 migration cases, no Keycloak state co-writes this path.	Settled (grounded)

3. Target end-state

Removed: the tandoor namespace and every workload/PVC/Service/IngressRoute/Certificate/Secret in it; the tandoor ArgoCD Application; the tandoor accept-list entry + vk_tandoor key from agentgateway; the CNPG tandoor database + role; the Authentik OAuth2 provider + application; the secret/fzymgc-house/cluster/tandoor Vault path; the Vault policy tandoor, the k8s auth role tandoor, and the TF-workspace read-grants to cluster/tandoor; the recipes uptime-kuma monitor; the Tandoor docs entries. recipes.fzymgc.house no longer resolves/routes.

Retained (deliberately): Mealie (unaffected); the shared cluster/smtp path (other consumers remain). No recipe data is preserved — any rolling Velero/CNPG backups that still contain Tandoor state age out on their normal retention (no deliberate pin, no export, no expedited purge required).

4. Component changes

Component	Change	File(s)	PR
Tandoor app manifests	Delete all child manifests (app self-prunes)	argocd/app-configs/tandoor/*	PR1
agentgateway vkey	Remove tandoor accept-list entry + vk_tandoor ExternalSecret data key	argocd/app-configs/agentgateway/secrets.yaml	PR1
CNPG database	Set ensure: absent (operator drops DB)	argocd/app-configs/cnpg/db-tandoor.yaml	PR1
CNPG role	Set managed role tandoor ensure: absent (operator drops role)	argocd/app-configs/cnpg/postgres-cluster.yaml	PR1
Tandoor Application	Delete the Application template (children already gone)	argocd/cluster-app/templates/tandoor.yaml	PR2
CNPG manifests	Delete db-tandoor.yaml + users-tandoor.yaml; drop both from kustomization; remove the tandoor managedRoles block	argocd/app-configs/cnpg/{db-tandoor,users-tandoor}.yaml, kustomization.yaml, postgres-cluster.yaml	PR2
Authentik + Vault secret	Delete file → destroys OAuth2 provider + application + random_password + vault_kv_secret_v2 cluster/tandoor	tf/authentik/tandoor.tf	PR2 (main-cluster-authentik)
Vault policy	Delete vault_policy.tandoor	tf/vault/policy-tandoor.tf	PR2 (main-cluster-vault)
Vault k8s auth role	Delete vault_kubernetes_auth_backend_role.tandoor	tf/vault/k8s-tandoor.tf	PR2
Vault TF-workspace grants	Remove the cluster/tandoor data + metadata read grants	tf/vault/policy-terraform-workspaces.tf (~125, ~164)	PR2
Vault keycloak-policy comment	Tidy the now-moot “minus tandoor” comment	tf/vault/policy-keycloak.tf:24	PR2 (optional)
uptime-kuma monitor	Remove uptimekuma_monitor_http.recipes	tf/uptime-kuma/main.tf	PR2
Docs — services	Remove all three Tandoor loci: the Quick-Reference index row (~L17), the ### Tandoor detail section incl. AI-provider steps (~L162–187), and the CNPG “Databases” table row (~L569); drop tandoor from the OpenRouter-passthrough service-key example list (~L282)	docs/reference/services.md	PR2
Docs — secrets	Remove the cluster/tandoor row (~L47); drop Tandoor from the cluster/smtp consumers note (~L45, leaving Authentik)	docs/reference/secrets.md	PR2
Docs — outbound-email	Drop Tandoor from the cluster/smtp retirement gate (~L144, leaving Authentik as the remaining decommissioning consumer); update the “Django (Tandoor et al.)” SSL-migration row (~L92) now that Tandoor is gone	docs/operations/outbound-email.md	PR2
Docs — pushover	Drop tandoor from the cluster-apps alert-routing example row (~L23)	docs/operations/pushover.md	PR2
ADR	New record documenting the retire-Tandoor decision	docs/adr/hl-nknn-*.md + bd create -t decision	PR2 (capture-adrs)

5. Sequencing

The Application tandoor.yaml has no resources-finalizer (many sibling apps do). In ArgoCD’s app-of-apps, deleting a child Application template only cascade-deletes its workloads when that finalizer is live on the object. Without it, removing the template orphans the Deployment / PVC / namespace. There is therefore no correctness-safe single PR that removes everything at once — the floor is a two-step drop-then-remove (the same idiom Grafana/Prometheus used). Tandoor’s own Application has prune: true, selfHeal: true, so emptying its source is what does the cleanup.

Phase 0 — Pre-flight (no data preservation)

Hard delete was chosen — there is no export/archive step and no recovery interlock. The only pre-flight checks are: confirm the app can be destroyed now (no in-flight use), and confirm the Longhorn storageClass backing tandoor-data uses reclaimPolicy: Delete so pruning the PVC actually frees the PV. Recipe data is discarded, not migrated.

PR1 — GitOps drop (ArgoCD prunes workloads; CNPG drops data)

• Delete every manifest under argocd/app-configs/tandoor/ → the tandoor Application prunes the namespace, Deployment, PVC tandoor-data (Longhorn PV reclaimed per storageClass — verify Delete), Service, IngressRoute (DNS auto-deregisters), Certificate, and both ExternalSecrets (deletionPolicy: Delete removes the target Secrets).
• Set ensure: absent on db-tandoor.yaml and on the tandoor managedRoles entry in postgres-cluster.yaml → CNPG issues DROP DATABASE / DROP ROLE. The Database and role reconcilers run independently with no ordering guarantee, so if the role drop is attempted before the DB drop completes, Postgres returns 2BP01 dependent_objects_still_exist and CNPG surfaces a transient cannotReconcile on the role — expected and harmless; it self-heals once the DB drop lands. Don’t treat that status as a stuck teardown.
• Remove the tandoor accept-list entry + vk_tandoor data key from agentgateway secrets.yaml.
• Verify: ns/tandoor gone; no tandoor-data PVC/PV; CNPG \l/\du show no tandoor db/role; agentgateway accept-list no longer lists tandoor; recipes.fzymgc.house no longer routes.

PR2 — Remove the management surface (GitOps + Terraform)

• Delete the Application template argocd/cluster-app/templates/tandoor.yaml (safe — children already pruned in PR1, so nothing to orphan).
• Delete the CNPG db-tandoor.yaml / users-tandoor.yaml, remove them from cnpg/kustomization.yaml, and remove the tandoor managedRoles block.
• Terraform (each applies on merge in its own TFC workspace):
  • main-cluster-authentik — delete tf/authentik/tandoor.tf → destroys the OAuth2 provider, application, random_password, and the cluster/tandoor Vault secret.
  • main-cluster-vault — delete policy-tandoor.tf, k8s-tandoor.tf, and the cluster/tandoor read-grants in policy-terraform-workspaces.tf; tidy the policy-keycloak.tf:24 comment.
  • uptime-kuma workspace — remove uptimekuma_monitor_http.recipes.
• Docs: scrub services.md + secrets.md; capture the ADR (capture-adrs).
• Verify: the three TFC applies succeed (authentik destroy shows the provider/app/secret destroyed; vault destroy shows policy/role/grants gone); terraform plan clean afterwards.

Phase 3 — Manual scrub (post-merge)

• Vault: vault kv metadata delete secret/fzymgc-house/cluster/postgres/users/main-tandoor (Postgres user creds path — verify its seeding source first; it is read by the ExternalSecret but not managed by the deleted Terraform, so confirm nothing else provisions it); vault kv patch secret/fzymgc-house/cluster/agentgateway to drop the residual vk_tandoor field.
• Confirm the Longhorn PV backing tandoor-data was actually deleted (reclaim policy).
• Confirm the CNPG tandoor database + role are gone (belt-and-suspenders after D4).
• No pinned backups exist (hard delete); any rolling Velero/CNPG backups still holding Tandoor state age out on normal retention. Optionally expire them explicitly for an immediate scrub.

6. Error handling / rollback

• PR1 / PR2 are git-revertable, but the data loss is intentional and permanent. Once PR1’s CNPG ensure: absent drops the DB (D4) and Longhorn reclaims the PV, reverting the manifests restores empty shells, not data — and by the hard-delete choice there is deliberately no export or pinned backup to restore from. Treat PR1 as the point of no return; the per-phase verification is the sole safety net.
• The vault_kv_secret_v2 cluster/tandoor destroy is irreversible (a re-apply would mint a fresh Django SECRET_KEY); immaterial once the app is gone.
• Two-PR ordering matters: PR2 must not merge until PR1’s prune is confirmed complete, or the Application template delete could still see live children and orphan them.

7. Testing / verification

• After PR1: kubectl get ns tandoor → NotFound; kubectl get pvc,pv -A | rg tandoor → empty; CNPG psql -c '\l' / \du → no tandoor; agentgateway ExternalSecret no longer contains a tandoor key; curl -sS https://recipes.fzymgc.house fails to resolve/route.
• After PR2: TFC run logs for the three workspaces show the expected destroys and a subsequently clean plan; rg -l -i tandoor tf/ argocd/ docs/reference/ docs/operations/ returns nothing (no live config or reference/ops docs remain — services.md, secrets.md, outbound-email.md, pushover.md all clean). A broader rg -i tandoor docs/ still lists ~14 dated historical specs/plans/ADRs (+ this spec/plan and the tools/docs-audit groom manifest), intentionally preserved.

8. Out of scope / accepted deltas

• Mealie migration — explicitly not done (D2 = hard delete); recipe data is discarded. Mealie does have a native Tandoor importer if a future manual re-entry is ever wanted, but this work produces no export.
• Any Keycloak client for Tandoor — never built by design (excluded from hl-5g05 Phase 2).
• Historical references in prior specs/plans (2026-06-29-keycloak-phase2-*, docs-audit groom manifest) — left as-is; they are dated records, not live config.

9. Grounding trace

All load-bearing facts are recorded as bd note entries on hl-nknn (footprint probe; DNS annotation model; CNPG databaseReclaimPolicy: retain default + ensure: absent drop semantics, per deepwiki cloudnative-pg/cloudnative-pg; Mealie native Tandoor importer, per deepwiki mealie-recipes/mealie; the missing-finalizer teardown constraint).

10. Decisions resolved

All three gating decisions were resolved by the owner on 2026-07-03:

1. Design bead — reuse hl-nknn.
2. Recipe data (D2) — hard delete (no export, archive, or Mealie migration).
3. Teardown shape (D3) — two-PR drop-then-remove.

The one remaining open item is a plan-time verify, not a decision: confirm what provisions secret/fzymgc-house/cluster/postgres/users/main-tandoor (read by an ExternalSecret but not managed by the deleted Terraform) before the Phase 3 Vault scrub.