Vault tf-runner unit decommission — 2026-09-07

Captured against context fzymgc-house with kubectl, and against https://vault.fzymgc.house with the Vault CLI authenticated as an operator OIDC identity (vault login -method=oidc, policies admin and default). All Vault reads in this record go through an explicit jq key projection; the reason is stated in the section below.

Why this file exists: the Flux era left a complete dead tf-runner unit in the cluster — a Secret holding a Vault token, the ServiceAccount that token belonged to, and a ClusterRoleBinding that binds that ServiceAccount to a ClusterRole which does not exist. Decommissioning it means touching a live credential, and the only defensible order is to neutralise the credential first and delete its visible trace second. That ordering, and the proof that each step actually took, is what this file records. It is a dated record of work performed on a specific day, not standing operational prose, so it lives in evidence/ rather than in docs/operations/vault.md. The one durable procedure this work produced — how to revoke a Vault token you do not hold — was written into docs/operations/vault.md instead.

Where the planning artifacts cited below live. Several passages cite a phase planning artifact by bare filename. Every one of them lives in .planning/workstreams/cluster-health/phases/04-vestigial-and-dead-references/. That directory is a planning record kept in the repository and is deliberately not published to the documentation site, so a reader on the site should expect to open those artifacts from a checkout rather than follow a link from here.

What this file records, and what it must never record

A Vault accessor is a non-credential handle. vault token lookup -accessor and vault token revoke -accessor both operate on it, and it cannot be used to authenticate. It is therefore the safe handle to write down. The token value is not, and never appears here in any form or prefix.

Recorded here	Never recorded here
the accessor, when one is obtainable	the token value, or any prefix of it
display_name, policies, entity_id, path, creation_time, expire_time, orphan, renewable	the base64 data.VAULT_TOKEN blob
the pre-flight verdict — live, already-dead, or lookup-broken — verbatim	a kubectl get secret vault-token -o yaml dump
the post-revoke 403, verbatim, with the control lookup that must succeed beside it	any realistic-looking example token
the Secret’s creationTimestamp, sole manager, and the kubectl delete confirmations

One control deserves naming explicitly, because removing it would look like tidying. vault token lookup -format=json includes .data.id, which is the token value; the default table output does not. Every JSON lookup below is piped through an explicit jq projection selecting only non-credential keys. That projection is a control, not decoration — dropping it is equivalent to printing the credential.

The token value was read exactly once, from the Secret into a shell variable, passed straight to a single vault token lookup, and unset in the same shell. It was never echoed, logged, or re-read.

The dead unit

Three objects, decommissioned as one group. All three were created in June 2025, all three are solely field-owned by kustomize-controller — the Flux controller that no longer runs in this cluster — and none of the three is declared anywhere in ansible/, tf/ or argocd/, nor tracked by ArgoCD.

Object	Created	Sole manager	Sole-Flux fields	ArgoCD tracking	Consumers
Secret/vault/vault-token	2025-06-08T19:16:53Z	kustomize-controller	data.VAULT_TOKEN, type	none	0 pods, Deployments, StatefulSets, Jobs, CronJobs
ServiceAccount/vault/tf-runner	2025-06-08T15:23:45Z	kustomize-controller	labels only	none	0 pods
ClusterRoleBinding/vault-tf-runner	2025-06-08T15:32:06Z	kustomize-controller	roleRef, subjects	none	binds ServiceAccount vault/tf-runner to ClusterRole/tf-runner-role

ClusterRole/tf-runner-role does not exist in the cluster. The binding therefore grants nothing today, which is exactly why it is deleted rather than repaired: a dangling binding to a missing ClusterRole is a privilege that re-arms silently the day anyone creates a ClusterRole by that name. Creating tf-runner-role to make the binding valid would arm something nothing needs.

The one vault-token string that does appear in the repository is an unrelated vault-token-creator-binding ClusterRoleBinding at argocd/app-configs/vault/rbac.yaml:23, whose subject is ServiceAccount vault/vault — a different identity, not in scope. A repository-wide search for tf-runner and vault-token across ansible/, tf/ and argocd/ returns only that binding, the unrelated cert-manager-vault-token-creator RBAC, and Ansible references to the vault CLI’s local ~/.vault-token file. None of the three objects above is declared in git.

Pre-flight, verbatim

Every negative reading below is paired with a control, because an empty result from a broken query reads identically to a genuine zero.

=== pre-flight: 2026-09-07T22:52:09Z ===

--- (0) Secret metadata, projected — never `-o yaml`, never `.data` values ---
$ kubectl --context fzymgc-house -n vault get secret vault-token --show-managed-fields -o json \
    | jq '{kind, name, namespace, creationTimestamp, type, keys: (.data|keys), labels,
           annotations, managers, managedFieldsDetail}'
{
  "kind": "Secret",
  "name": "vault-token",
  "namespace": "vault",
  "creationTimestamp": "2025-06-08T19:16:53Z",
  "type": "Opaque",
  "keys": [ "VAULT_TOKEN" ],
  "labels": {
    "kustomize.toolkit.fluxcd.io/name": "apps",
    "kustomize.toolkit.fluxcd.io/namespace": "flux-system"
  },
  "annotations": {},
  "managers": [ "kustomize-controller" ],
  "managedFieldsDetail": [
    { "manager": "kustomize-controller", "operation": "Apply",
      "time": "2025-08-07T12:58:22Z", "fields": [ "f:data", "f:metadata", "f:type" ] }
  ]
}

--- (1-5) workloads referencing Secret vault-token, cluster-wide ---
(1-5) pods referencing Secret vault-token: 0
(1-5) deployments referencing Secret vault-token: 0
(1-5) statefulsets referencing Secret vault-token: 0
(1-5) jobs referencing Secret vault-token: 0
(1-5) cronjobs referencing Secret vault-token: 0

--- (1-5 CONTROL) the same traversal, widened, must find secret references ---
pods matching a broader secret-name probe: 43

--- (6) pods using ServiceAccount tf-runner (any namespace) ---
0

--- (6 CONTROL) pods using ServiceAccount vault (same query shape) ---
3

--- (7) ClusterRole/tf-runner-role ---
NOT FOUND (expected)

--- (7 CONTROL) clusterrole lookup works ---
clusterrole.rbac.authorization.k8s.io/cluster-admin

--- (8) ServiceAccount vault/tf-runner, projected ---
{ "name": "tf-runner", "ns": "vault", "created": "2025-06-08T15:23:45Z",
  "labels": { "kustomize.toolkit.fluxcd.io/name": "apps",
              "kustomize.toolkit.fluxcd.io/namespace": "flux-system" },
  "annotations": {}, "secrets": [], "managers": [ "kustomize-controller" ] }

--- (9) ClusterRoleBinding vault-tf-runner, projected ---
{ "name": "vault-tf-runner", "created": "2025-06-08T15:32:06Z",
  "labels": { "kustomize.toolkit.fluxcd.io/name": "apps",
              "kustomize.toolkit.fluxcd.io/namespace": "flux-system" },
  "annotations": {},
  "roleRef": { "apiGroup": "rbac.authorization.k8s.io", "kind": "ClusterRole",
               "name": "tf-runner-role" },
  "subjects": [ { "kind": "ServiceAccount", "name": "tf-runner", "namespace": "vault" } ],
  "managers": [ "kustomize-controller" ] }

--- (10) the one read of the credential — read once, projected, unset ---
$ TOKEN=$(kubectl --context fzymgc-house -n vault get secret vault-token \
            -o jsonpath='{.data.VAULT_TOKEN}' | base64 -d)
$ vault token lookup -format=json "$TOKEN" | jq '{accessor, display_name, policies, orphan,
    renewable, creation_time, expire_time, ttl, num_uses, entity_id, path}'
$ unset TOKEN

lookup exit code: 2
Error looking up token: Error making API request.

URL: POST https://vault.fzymgc.house/v1/auth/token/lookup
Code: 403. Errors:

* bad token

--- (10 CONTROL A) same endpoint, same operator policy, a KNOWN-VALID token ---
$ vault token lookup -format=json "$(cat ~/.vault-token)" | jq '.data | {accessor, display_name,
    policies, entity_id, path}'
control lookup-by-value exit code: 0
{
  "accessor": "<operator-accessor-redacted>",
  "display_name": "oidc-ab096608-3991-4465-9bb8-d1fe640650ac",
  "policies": [ "admin", "default" ],
  "entity_id": "43f3e334-28f3-20a4-b699-41dd1ffc8770",
  "path": "auth/oidc/oidc/callback"
}

--- (10 CONTROL B) the lookup-accessor path works, on the operator's own accessor ---
$ vault token lookup -accessor -format=json <operator-accessor-redacted> | jq '.data | {accessor,
    display_name, policies, entity_id, path, expire_time}'
control lookup-by-accessor exit code: 0
{
  "accessor": "<operator-accessor-redacted>",
  "display_name": "oidc-ab096608-3991-4465-9bb8-d1fe640650ac",
  "policies": [ "admin", "default" ],
  "entity_id": "43f3e334-28f3-20a4-b699-41dd1ffc8770",
  "path": "auth/oidc/oidc/callback",
  "expire_time": "2026-09-08T09:07:09.58662818Z"
}

[VERIFIED: vault + kubectl, 2026-09-07T22:52:09Z] — the pre-flight outcome is the second of the three branches: Code: 403 … bad token. The subject token is already expired or revoked. There is nothing to revoke.

That verdict is load-bearing, so it is read two-sided. A 403 on its own is compatible with three different worlds: the token is dead, the lookup endpoint is broken, or the operator’s own policy forbids third-party lookup. Control A rules out the second and third together: the identical command, against the identical endpoint (POST /v1/auth/token/lookup), with the identical operator policy, returns 200 for a token known to be valid. Control B additionally proves the lookup -accessor path — the one a revoke would have been proved against — is working. So the 403 is a statement about the subject token and nothing else.

No accessor is recorded for the subject token. An accessor is only obtainable from a successful lookup, and the subject’s lookup failed. This is the expected shape of the already-dead branch, and it is why the revoke step below has no accessor argument: there is no live token behind it. The one accessor these transcripts would otherwise carry is the operator’s own, and it is redacted to <operator-accessor-redacted> everywhere it appeared. An accessor is not a credential, but that one is a live handle to an operator session which outlives this capture — it expires 2026-09-08T09:07:09Z — and a handle to a live session has no business sitting in a committed file. Nothing evidentiary is lost: the commands it was passed to are recorded verbatim with their exit codes and their projected output, which is what makes them controls.

Nothing was revoked and nothing was deleted in the pre-flight. All three objects were still present at the end of it, and the token value was read exactly once and unset.

Revoke and deletion, verbatim

The operator selected revoke-and-delete-all-three at the one-way-door gate on 2026-09-07, with the explicit instruction that the revoke outcome be recorded as “already dead — nothing to revoke” rather than as a revocation that happened. What follows is that decision carried out.

The banner timestamps below are UTC and read 2026-09-08 because the work crossed midnight UTC; the date in this file’s name is the local date of the capture, which is the same session as the pre-flight above.

The revoke that was not needed

The pre-flight already settled the credential question: Code: 403 … bad token, read two-sided against both controls on the same endpoint under the same operator identity. The subject token was already expired or revoked before this work began. There was nothing to revoke, and no accessor exists to revoke it by — an accessor is obtainable only from a successful lookup, and the subject’s lookup failed.

That is deliberately not dressed up as a completed revocation. vault token revoke -accessor was never run, because there was no accessor to pass it and no live token behind it. The plan’s ordering constraint — revoke strictly before deleting — is therefore satisfied vacuously rather than by an act. The property it exists to guarantee still holds: at no point in this work did a live credential outlive its Kubernetes trace.

The subject’s 403 is not re-run here, and that is a choice worth naming. Re-proving it would have required reading the token value out of the Secret a second time, and every read of that value is a disclosure surface — the one this record’s controls are mostly built against. The pre-flight read it exactly once. That 403, with Control A and Control B beside it in the same transcript at the same instant, is the two-sided proof; re-running the negative would have bought a second copy of a settled reading at the price of a second exposure.

What is re-run below is the positive control, which needs no credential. It establishes that the token-lookup endpoint and the operator identity were still working at the moment the deletions ran — so the “already dead” verdict is not being carried forward from a session that had since gone stale.

=== revoke: 2026-09-08T00:22:11Z ===

--- (R1) revoke: NOT RUN — the subject token is already dead, no accessor exists ---

--- (R2 CONTROL) the operator identity and the token-lookup endpoint still work, now ---
$ vault token lookup -format=json | jq '.data | {display_name, policies, entity_id, path,
    expire_time}'
{
  "display_name": "oidc-ab096608-3991-4465-9bb8-d1fe640650ac",
  "policies": [ "admin", "default" ],
  "entity_id": "43f3e334-28f3-20a4-b699-41dd1ffc8770",
  "path": "auth/oidc/oidc/callback",
  "expire_time": "2026-09-08T09:07:09.58662818Z"
}
control self-lookup exit code: 0

Note the form of that control: bare vault token lookup, with no argument and no flag naming self. Self is the default. There is no -self flag on this subcommand and passing one is rejected outright — see the correction recorded at the foot of this file.

The three deletions

Deleted binding-first, so that no window exists in which the ClusterRoleBinding outlives the ServiceAccount it names. ClusterRole/tf-runner-role was not created; the dangling binding was removed rather than repaired, which is a strict reduction in privilege.

=== deletions: 2026-09-08T00:22:19Z ===

--- (D0) all three present immediately before deletion ---
$ kubectl --context fzymgc-house get clusterrolebinding vault-tf-runner -o name
clusterrolebinding.rbac.authorization.k8s.io/vault-tf-runner
$ kubectl --context fzymgc-house -n vault get serviceaccount tf-runner -o name
serviceaccount/tf-runner
$ kubectl --context fzymgc-house -n vault get secret vault-token -o name
secret/vault-token

--- (D1) ClusterRoleBinding/vault-tf-runner — the dangling privilege, deleted first ---
$ kubectl --context fzymgc-house delete clusterrolebinding vault-tf-runner
clusterrolebinding.rbac.authorization.k8s.io "vault-tf-runner" deleted

--- (D2) ServiceAccount/vault/tf-runner ---
$ kubectl --context fzymgc-house -n vault delete serviceaccount tf-runner
serviceaccount "tf-runner" deleted from vault namespace

--- (D3) Secret/vault/vault-token ---
$ kubectl --context fzymgc-house -n vault delete secret vault-token
secret "vault-token" deleted from vault namespace

Proof the deletions took, paired

Each NOT FOUND is paired with a control of the same kind that must still resolve, because a NotFound from a broken query reads identically to a genuine absence.

=== post-deletion proof: 2026-09-08T00:22:27Z ===

--- (P1-P3) each of the three, re-queried; the API error is recorded verbatim ---
Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "vault-tf-runner" not found
Error from server (NotFound): serviceaccounts "tf-runner" not found
Error from server (NotFound): secrets "vault-token" not found

--- (P1-P3 CONTROL) objects of the SAME three kinds that must still be found ---
clusterrolebinding: clusterrolebinding.rbac.authorization.k8s.io/cluster-admin
serviceaccount:     serviceaccount/default
secret:             secret/authentik-tf-secrets

--- (P4) ClusterRole/tf-runner-role was NOT created ---
still absent (expected)

--- (P4 CONTROL) clusterrole lookup works ---
clusterrole.rbac.authorization.k8s.io/cluster-admin

[VERIFIED: vault + kubectl, 2026-09-08T00:22:27Z] — the dead tf-runner unit is gone. All three objects return NotFound while a live object of each kind still resolves; ClusterRole/tf-runner-role remains absent; and the Vault token behind the deleted Secret was already dead before the work started, so nothing was revoked and nothing needed to be.

Why this was a direct kubectl action

CLAUDE.md and argocd/CLAUDE.md say ArgoCD owns deployments and that changes land through GitOps, not through kubectl. This work deleted three live objects with kubectl anyway, and that is the argued exemption rather than a violation of the constraint.

None of the three objects is declared anywhere in ansible/, tf/ or argocd/, and none carries an ArgoCD tracking annotation — all three were solely field-owned by kustomize-controller, the Flux controller that no longer runs in this cluster. So there was no manifest to change and no sync that could have effected the deletion: GitOps had no lever on these objects at all. The action converges the cluster toward a git that declares none of them, which is the same argument argocd/app-configs/velero/backup-schedule.yaml lines 74-90 already makes in tree for its own direct patch. Because nothing declares these objects, nothing will recreate them.

Corrections to the planning record

Correction: D-06 scoped the work to one object; the dead unit is three

04-CONTEXT.md D-06 names only Secret/vault/vault-token. Research measured two more objects in the same dead unit — ServiceAccount/vault/tf-runner and ClusterRoleBinding/vault-tf-runner — and the operator ruled all three in on 2026-09-07. Deleting the Secret alone would have removed the visible trace while leaving a binding that grants nothing today and everything tf-runner-role would grant the day anyone creates a ClusterRole by that name. D-06 as written is superseded by that three-object ruling; the scope executed here is the ruling, not the original entry.

Correction: vault token lookup -self is not a valid flag

04-RESEARCH.md §RQ-6.2 and this plan’s Task 1 both prescribed vault token lookup -self for the operator’s own control lookup. Vault v2.0.4 rejects it outright:

flag provided but not defined: -self

Self is the default for vault token lookup; the correct invocation is the bare command with no argument. Every control lookup in this file uses that working form, and the procedure written into docs/operations/vault.md carries the working form with the flag’s absence called out, so the broken form is not propagated into the runbook.

Correction: the operator’s own accessor is redacted from this record

The pre-flight transcript as first committed carried the operator’s own accessor verbatim, as the handle its two controls were run against. An accessor is not a credential and cannot authenticate, but that particular one is a live handle to an operator session that outlives this capture, and a handle to a live session does not belong in a committed file. It is replaced by <operator-accessor-redacted> throughout. The commands, their exit codes and their projected output are unchanged, so the controls remain readable as controls. The superseded revision remains in git history at commit 6146175ac; the session it points at expires 2026-09-08T09:07:09Z.