Confine the hl-assets R2 credential by resource selector, dedicated Vault path and Velero exclusion — measured against the shared backup bucket, with a coarse egress policy accepted because toFQDNs does not work on this cluster

Date: 2026-08-01 Status: Accepted Deciders: Sean Brandt

Context

Milestone v1.5’s uploader pod needs an R2 credential that can write. The split read/write ADR fixed the architecture and accepted the R2 list-permission residual in principle; the edge policy ADR fixed what the read path does. This record fixes the credential: where it lives, what it can reach, what happens when it rotates, and what this project measured rather than assumed about all three. It supersedes no part of either sibling — it discharges an obligation the first one left open.

One fact forces the work. R2 offers no write-only permission. Every write-capable permission group also grants read and list, so the only least-privilege lever R2 exposes is the token’s resources selector. The design’s security claim is therefore not “the token cannot list” — it can — but “the token cannot see anything outside its own two buckets.” That is a claim about a boundary, and a boundary claim that is not measured is a belief.

This estate shares one bucket, fzymgc-cluster-storage, across Longhorn backups, Velero backups, CNPG barman archives and agent-memory snapshots. It is the bucket that would hurt.

The discipline this record applies

Phase 2’s ADR narrowed a claim after measurement contradicted it, and named the practice. This record continues it, and adds one rule of its own because a security auditor reads this document: every protection below is labelled either ESTABLISHED — with the committed evidence file that established it — or OWED, phrased as a constraint on a future phase. An untested protection written into an ADR is worse than no protection, because someone will rely on it. Where a question was deliberately not answered, it appears under Open assumptions with the reason, not as a silence.

Decision

One workload token, confined by its resources selector to the two hl-assets buckets; one dedicated Vault path; ESO authenticating as itself; the namespace excluded from both Velero schedules; and a coarse L4 egress policy, accepted because the host-scoped one is not available on this cluster.

Control	Shape
Credential	cloudflare_account_token, minted by terraform apply in Phase 1, two entries in its resources map — one per hl-assets bucket
S3 key pair	Derived, not separately issued: access_key_id = <token>.id, secret_access_key = sha256(<token>.value)
Vault path	secret/fzymgc-house/cluster/hl-assets/r2 on the KV-v2 mount secret. Nothing else writes there
Delivery	ExternalSecret hl-assets/hl-assets-r2, refreshInterval: 15m, six discrete keys, ClusterSecretStore vault
Backup posture	hl-assets in excludedNamespaces on both cluster-wide Velero schedules
Egress	CiliumNetworkPolicy hl-assets/hl-assets-egress — TCP/443 to toEntities: [world], plus the L4 DNS allow

Rationale

The isolation boundary — measured, one key pair, three buckets, one run

ESTABLISHED. Evidence: iso-matrix.observed.txt, captured 2026-08-01 against live Cloudflare R2 by the verifier’s isolation phase (since removed with its tests).

Arm	Bucket	Observed
iso-allow-public	hl-assets-public	ListObjectsV2 allowed
iso-allow-private	hl-assets-private	ListObjectsV2 allowed
iso-deny-backup	fzymgc-cluster-storage	refused, S3 error code AccessDenied, exit 254

All three arms ran in ONE process, with ONE key pair, read from Vault at that one path. That is not a convenience of implementation, it is the whole evidentiary content of the test, and this phase measured why: with a key pair that could sign for neither hl-assets bucket, iso-deny-backup still reported AccessDenied and still passed. R2 refuses an out-of-scope bucket before it validates the signature. So the denial arm alone is satisfied by a wholly broken credential, and a 403-shaped observation taken in isolation is indistinguishable from one. The positive controls are what make the denial mean anything, and they must run first, in the same process, on the same key.

The criterion’s wording was wrong and has been amended rather than quietly satisfied. ROADMAP success criterion 3 said the key pair “returns 403”. The aws CLI renders an S3 error code, never an HTTP status, so the instrument this criterion is verified with could never have produced a 403 to compare against — the literal number was neither confirmed nor contradicted, it was unmeasurable by construction. SC#3 is restated on AccessDenied, with the original wording preserved inline and marked superseded. The measurement the literal 403 would still need is named there and was not taken: a wire-level capture through a client that surfaces the HTTP status.

Least privilege here comes from the resource selector, not from a narrower permission group — R2 does not offer one. The workload token holds the account-level Workers R2 Storage Write / Read groups, restricted through resources. This corrects an earlier premise in the roadmap that had the uploader holding a bucket-level Bucket Item Write; it does not, and the difference matters below.

The committed evidence carries no free-text field. IsolationObservation has exactly four fields — role, returncode, code, outcome — outcome is a closed set of five module-defined tokens selected by control flow rather than parsed out of text, and code is bounded by charset, a forty-character cap and an all-hex rejection. Raw aws stderr does not reach disk in any form, sanitised or otherwise. The evidence boundary therefore does not depend on the completeness of a redaction function — which was the right call, since a gap in that function was found within the hour of writing it (Authorization:\s*\S+ redacted the word Bearer and left the token standing).

The accepted residual — stated, not implied

REQUIREMENTS.md requires this be stated here rather than left implicit. It is.

R2 offers no write-only permission. Every write-capable group also grants read and list. A compromised uploader pod can therefore enumerate every object key in both hl-assets buckets — and since the key is the capability, that leaks every link ever issued. DeleteObject is a free operation, so mass deletion produces no billing signal.

Bounded, precisely. The pod holds the workload token, confined by its resources selector to hl-assets alone. It never holds the bootstrap token, which is the control plane credential and never enters the cluster. So the exposure is every object in those two buckets and nothing outside them — which is exactly what the measurement above demonstrates rather than asserts.

Precondition: compromise of an in-cluster pod, at which point the attacker holds cluster access and there are larger problems. The assets are copies of originals held elsewhere. Accepted deliberately.

ESTABLISHED mitigations

Each cites the evidence that established it.

1. A dedicated bucket pair, provably unable to reach the shared backup bucket. iso-matrix.observed.txt, 2026-08-01, with both positive controls green in the same run.
2. A dedicated Vault path, with no per-app policy and no per-app auth role. secret/fzymgc-house/cluster/hl-assets/r2; ESO authenticates as itself. Asserted by the iso-no-vault-change check rather than assumed — see The Vault-policy attribution below.
3. The credential never enters the shared backup bucket. hl-assets is excluded from both Velero schedules, in the same pull request that creates the namespace, so the window in which the Secret could be captured never opens. argocd/app-configs/velero/backup-schedule.yaml.
4. An egress policy bounding the pod to TCP/443 and stopping in-cluster lateral movement. argocd/app-configs/hl-assets/networkpolicy.yaml. At that strength and no more — it restricts the port, not the host. See The egress decision below.
5. No List* call anywhere in the uploader’s application code, so the capability the credential unavoidably carries is demonstrably unused. DISCHARGED 2026-08-02 by Phase 4 — this was the single OWED constraint this record carried. See The discharged bound below for the enforcing artifact, the scoping property that makes it auditable, and the run in which the gate was observed going red.

OWED constraints

None remain. This section carried exactly one entry and it has been discharged.

AMENDED 2026-08-02, when Phase 4 shipped the uploader. Original wording, preserved verbatim and superseded:

1. Phase 4 must prove that no List* call appears anywhere in the uploader’s application code, so that the capability the credential unavoidably carries is demonstrably unused. This is not a fact about this repository today. No uploader code exists; every UP-* requirement is unchecked and assigned to Phase 4. The gate that discharges it is Phase 4 success criterion 3 in ROADMAP.md, which additionally fixes the mechanism: the tombstone/dedup check is a single HeadObject, never a list. This ADR is amended to record the verified fact when Phase 4 ships. Until then the residual is unmitigated on this head, and REQUIREMENTS.md’s Bounded by: sentence carries the same qualification so the source and this document agree by construction rather than by coincidence.

Why: the amendment this record promised itself has come due. Phase 4 shipped the uploader, and the bound is now a fact about this repository rather than a constraint on a future phase. It moves to ESTABLISHED mitigation 5, and the same sentence in REQUIREMENTS.md §Accepted residual risk is amended in the same pass — Phase 3 recorded that the identical qualification was carried in three places and swept unevenly, and sweeping both remaining copies together is the correction.

What did NOT change: the residual itself. The credential still holds the list capability; R2 offers no write-only permission group and none of that is affected by what the application chooses to call. What is now established is that the capability is unused by first-party code, enforced by a blocking gate rather than by convention.

The discharged bound — the control, its scope, and the run it was seen failing in

ESTABLISHED 2026-08-02. The write path’s own decisions are recorded in the uploader-service ADR; this section records only what discharges the bound above.

Control	Where it lives
The blocking gate	the source-gates job in .github/workflows/build-hl-assets-uploader.yaml, first of its three checks
The scoping property that makes it auditable	internal/store is the only package that imports the S3 client, so the audit is one package rather than a whole repository
A structural absence, not merely an undocumented one	the Store interface declares no list method and no delete method — the mistake is un-writable rather than discouraged
A second enforcement path	TestNoObjectListingCallAnywhereInThePackage runs inside the Dockerfile’s test stage, so a violation fails the image build, not only a CI job
Publishing depends on it	build-amd64 and build-arm64 both declare source-gates as a prerequisite, so a push whose gates went red cannot publish

The run in which the gate was observed going RED. A real, compiling enumeration call — not a text-only mutation — was appended to internal/store/r2.go and the gate matched it at r2.go:355, with go build ./internal/store confirming the mutant compiled. After git checkout of that one file the gate returned no match. Recorded in 04-05-SUMMARY.md’s mutation ledger, row 3.

One precondition is load-bearing, and without it this control was vacuous. rg is not preinstalled on the runner profile these jobs use. if rg …; then with rg absent is “command not found” — a non-zero exit, so the condition is false and the gate reports clean on any input, permanently and silently, looking exactly like a passing gate. That was discovered only because an availability assertion had been placed in front of the searches; the job now installs ripgrep and asserts its presence before any gate runs. Recorded here because the fix is one line in a workflow and its removal would be invisible in review. Measured during Phase 4, plan 01; the job’s first real run failed at that assertion.

Two narrower caveats, recorded rather than glossed. The gate’s exclusion for vendored source is anchored to the job’s working directory rather than to its search-path argument, so it excludes a vendor/ directory at the repository root rather than one inside the module — harmless today because no such directory exists, and silently wrong the day someone vendors. And the scoping property in row two is checked by hand rather than by a CI gate; a fourth source gate asserting that exactly one file imports the S3 client is a recommended follow-up that has not landed.

The egress decision, and the consequence of the branch that shipped

Decision: the fallback shape. toEntities: [world] on TCP/443 plus the L4 DNS allow.

Why, measured rather than inferred. Host-scoped egress requires Cilium toFQDNs, and Cilium populates the FQDN-to-IP cache exclusively from DNS-proxy observation — so a toFQDNs rule must be paired with an L7 rules.dns block or it selects an empty IP set and denies rather than allows. The two were therefore bound as one variable: testing them apart would have measured a construct nobody would ship.

Evidence: 03-D42-PROBE.md, 2026-08-01, in a throwaway namespace destroyed afterwards.

Reading	Unpolicied control	With toFQDNs + L7 rules.dns bound
DNS resolution	resolves	connection timed out; no servers could be reached
curl to the R2 endpoint	HTTP 400 from R2 in 0.285 s	HTTP 000, failing at resolution after 12.6 s
cilium-dbg fqdn cache list	—	header row, zero rows
cilium-dns-egress redirects	—	three, installed and active (endpoint 2302, proxy port 39329)

The last row is what makes the negative interpretable. The redirect was installed, so this is not a selector mistake and not a proxy that never engaged — queries went into the eBPF→TPROXY→proxy hop and nothing came back. Without that reading the result could not be told apart from a denied connection. Root cause: this cluster runs dnsproxy-enable-transparent-mode: false, the hl-bdr workaround, and that workaround has not healed. The probe also resolved a two-record disagreement in the repository: argocd/app-configs/dolt/network-policies.yaml:99-104 is correct, and ansible/roles/cilium/defaults/main.yml:82-98’s implication — that disabling transparent mode makes L7 DNS rules usable — is wrong.

The consequence, stated plainly. The shipped policy restricts the port and stops in-cluster lateral movement. It does not restrict which host is reached. A compromised pod can open TCP/443 to anywhere on the internet. That is strictly weaker than the roadmap originally specified, it is written into the manifest itself rather than implied by its absence, and ROADMAP SC#4 was amended under D-46 with its original wording preserved — because a criterion left standing that a measurement invalidated reads as a criterion the phase failed, and this one was consciously revised.

The decision is committed on the object, as the annotation hl-assets.fzymgc.house/d42-egress-shape: fallback, so a gate can read the branch from somewhere other than the manifest it is checking, and so both branches of every gate stay executable rather than one becoming unreachable text.

Revisit in Phase 6, against the named upstream condition: Cilium 1.20 GA and Standalone DNS Proxy graduation, which ansible/roles/cilium/defaults/main.yml:91 already anticipates. The tighter policy is not available today, and claiming it would be claiming a control that does not exist.

Data placement — where the credential is allowed to come to rest

Decision: exclude. hl-assets is added to excludedNamespaces on both the daily-backup and weekly-full-backup schedules. It was not accept and not label-exclude-requested, so no accept residual is owed here — the credential never enters fzymgc-cluster-storage at all.

Sean’s reasoning, verbatim, 2026-08-01:

Matches argocd/CLAUDE.md’s MUST for stateless namespaces. PORT-01 forbids the uploader persistent state, so the Secret is reconstructible from Vault by ESO within one refreshInterval — nothing in this namespace is unrecoverable. One line per schedule, shipped in the same PR as the namespace, so the window in which the Secret could be captured never opens.

The exclusion carries that reason in the file that implements it, in the file’s own # Category - reason idiom, and says explicitly that it is a decision rather than a habit — modelled on the worked negative annotation the same file carries around keycloak, which is deliberately not excluded. Without both halves, a later reader adding a namespace has one example and no rule.

A correction to the record, because the plan’s own reasoning was half wrong. Plan 03-03 withdrew a narrower resource-policy arm partly on the grounds that Velero’s includeExcludePolicy key (added in v1.17) “may or may not exist here”, the server version having gone unmeasured. It was then measured, live: Velero server is v1.18.1 (chart velero-12.0.3), on both the deployment and the node-agent. So the mechanism is available on this cluster and that half of the rationale is falsified. The withdrawal still stands on the half that was never about versions — the arm had no implementable file scope. Recorded here because a later phase reasoning from the plan text alone would conclude the narrow instrument is speculative, and it is not.

Rotation, and the step it does not reach

Rotation of this credential is a terraform apply. No human copies a secret, ever. The token is re-minted, vault_kv_secret_v2.hl_assets_r2 writes the new value, and ESO propagates it to the Kubernetes Secret within one refreshInterval — 15 minutes, per argocd/app-configs/hl-assets/external-secret.yaml.

That is not the end of the operation, and the gap is silent. Environment variables sourced from a Secret are materialised at container start and are not hot-reloaded. A running pod therefore keeps the old credential indefinitely after the Secret beneath it has changed. Nothing errors; the pod simply carries a value that no longer matches Vault, and an operator who watched the terraform apply succeed and the Secret update has every reason to believe rotation completed.

Operational step, explicit: after rotating the credential, restart the uploader Deployment. Rotation is not complete until the pods have restarted.

Phase 4 handoff, named here so it is not lost and not adopted out of scope. This estate already runs Stakater Reloader cluster-wide (argocd/cluster-app/templates/reloader.yaml), and several workloads already carry reloader.stakater.com/auto: "true". A single annotation on Phase 4’s Deployment would close this gap and make the manual restart unnecessary. That annotation belongs to the phase that owns the Deployment, which does not exist yet — it is recorded as a handoff, not adopted here, and it is not counted among the mitigations above.

The Vault-policy attribution — corrected, and asserted rather than assumed

No per-app Vault policy and no per-app Kubernetes auth role was added for hl-assets. ESO authenticates as itself, and the grant chain is:

ClusterSecretStore vault → auth.kubernetes.role = external-secrets (tf/cluster-bootstrap/external-secrets.tf:71) → vault_kubernetes_auth_backend_role.external-secrets with token_policies = ["default", "external-secrets-operator"] (tf/vault/k8s-external-secrets.tf:6-13) → vault_policy.external_secrets_operator’s path "secret/data/*" { capabilities = ["read","list"] } (tf/vault/policy-external-secrets-operator.tf:17-20).

Two Terraform comments previously attributed that read to fzymgc-cluster-secret-reader, a policy that carries an identically-shaped secret/data/* stanza and is bound to nothing — verified by a repo-wide search returning only its own declaration and the two comments now corrected. The failure mode this closes is specific: an agent traces the grant through the old name, finds it attached to no role, concludes ESO has no grant, and “fixes” it by adding a per-app Vault read policy — which fails the criterion by appearing to satisfy it. Both comments now say so.

The absence is asserted by a gate, over resource IDENTITY rather than block bodies, and the scoping is load-bearing in a way that will look wrong to a later reader. tf/vault/policy-terraform-workspaces.tf legitimately grants secret/data/fzymgc-house/cluster/hl-assets/* to the Terraform workspace: that grant predates this milestone, and the cross-workspace apply fails without it. A whole-block search would go red on it and report the security claim as failed. What the criterion actually forbids is a policy or role that identifies this workload — so the check reads declaration labels and identity attributes, comment-stripped in both clauses, with counted floors (55 declarations, 88 identity attributes) so that a pattern which silently stopped matching cannot report the same clean green as one that searched everything. Do not “tighten” this back to block bodies.

Open assumptions — recorded, not resolved

These are questions this phase deliberately did not answer. Each carries the reason it was left open. None of them is a protection, and none may be relied on as one.

1. Whether the token’s resources selector also gates R2 bucket-lock mutation is UNTESTED — deliberately. An earlier note held that an uploader token “structurally cannot remove a lock”, because lock editing needs the account-level Workers R2 Storage Write while the uploader holds only a bucket-level Bucket Item Write. That premise does not match the token this estate mints. The workload token uses the account-level groups, scoped through resources — so the uploader holds the very group that edits locks, merely restricted to its own buckets. Whether resources also constrains lock mutation was not measured. It was left open because bucket locks are not adopted in v1.5, so nothing depends on the answer. No lock-based protection is claimed anywhere in this record or in any runbook, and none may be added without testing first.

2. The live post-merge delivery verification has NOT been captured. Three of this phase’s seven isolation checks read the cluster (iso-eso-ready, iso-secret-keys, iso-cnp-applied) and are red by construction until ArgoCD has synced the namespace — and ArgoCD syncs from the default branch, so they cannot run before the implementation pull request merges. At the time of writing, that capture is outstanding, tracked as an open entry in .planning/WINDOWS.md. Nothing in this document should be read as reporting a live green for Secret delivery or for the applied CiliumNetworkPolicy. What is established is structural: iso-no-vault-change is cluster-independent and passes today, and both manifests validate against the live cilium.io/v2 and external-secrets.io/v1 CRD schemas.

3. Pre-merge manifest validation runs at client, not server, and that was measured rather than presupposed. A server-side dry run is rejected by the NamespaceLifecycle admission plugin because the target namespace does not exist yet — a rejection about the namespace, not a schema verdict. Retargeted at an existing namespace, the identical bytes validate against the live CRDs and a deliberately invalid field is rejected with a strict decoding error, so live-CRD validation is established for the manifests; but the check that runs in the ordinary pre-merge position is local-schema only. Recorded so nobody re-derives the weaker conclusion from the exit status alone.

4. The literal HTTP status behind the R2 refusal was not measured. See the boundary section above: the harness asserts on the S3 error code, which is what its instrument can actually observe.

Alternatives Considered

• A narrower R2 permission group instead of the resources selector (chosen alternative does not exist). R2 offers no write-only group. There is nothing narrower to ask for.
• toFQDNs egress confined to *.r2.cloudflarestorage.com — measured unavailable on this cluster, not rejected on preference. See the egress section.
• Accepting Velero capture of the credential Secret — rejected. The namespace holds no unrecoverable state, so exclusion costs one line per schedule and removes the exposure outright rather than bounding it.
• A Velero resource-policy / includeExcludePolicy scoped to the Secret — withdrawn. The mechanism does exist here (server v1.18.1), but the arm had no implementable file scope in the plan that would have shipped it, and implementing it would have meant editing an undeclared file.
• A velero.io/exclude-from-backup label on the ESO-generated Secret — not taken. For an ESO-generated Secret the label must live in ExternalSecret.spec.target.template.metadata.labels, which makes the exclusion depend on a template rendering correctly; namespace exclusion does not.
• A per-app Vault read policy and a per-app Kubernetes auth role for hl-assets — rejected, and actively guarded against. ESO already authenticates as itself with a secret/data/* read grant; adding one would create a second credential path to maintain and would fail the criterion it appears to serve.
• Adopting R2 bucket locks as a deletion backstop — deferred, and explicitly not written down as a protection, because whether the token’s selector gates lock mutation is untested.

Consequences

Positive:

• The central security claim of this phase is a measurement, taken with both positive controls in the same process on the same key pair, rather than an argument from configuration.
• The credential exists only in Vault and in a Secret ESO rebuilds from it; rotation is a terraform apply and no human ever copies it.
• The credential is provably absent from the shared backup bucket, by a decision that shipped in the same pull request as the namespace.
• The Vault grant chain now traces through the policy that is actually bound, so the specific wrong “fix” this design is most exposed to has been closed off in the two places an agent would look.

Negative:

• The egress policy restricts the port, not the host. A compromised pod can reach any internet host on TCP/443. Accepted; revisited in Phase 6.
• The list capability is real, and no configuration removes it — R2 offers no write-only permission group. Amended 2026-08-02: the no-List* rule that makes it demonstrably unused by first-party code is now established rather than owed, enforced by a blocking CI gate observed going red against a compiling enumeration call. The capability itself is unchanged; what is bounded is what the application does with it.
• Rotation is incomplete without a pod restart, and the incompleteness is silent.
• Whether the resource selector gates bucket-lock mutation is unknown, so locks cannot be reasoned about at all until someone measures it.

Neutral:

• The verifier’s isolation phase required a Vault token, and a missing credential was a FAIL rather than a skip: without the key pair all three arms measure nothing, and a skip folded into a green run is how a gate reports on a property it never evaluated. That phase has since been removed; the fail-closed posture remains the verifier’s contract.
• The resources-scoped account-level grant means the uploader holds groups that look broader than the design implies. The confinement is real but lives in the selector, so reading the permission groups alone overstates what the token can reach.

Sign-off: decision, date (2026-08-01) and deciders (Sean Brandt) are recorded above. Human approval of the pull request that lands this ADR is the sign-off for the accepted residual and for the two amended success criteria.