Split the hl-assets read path from the write path across two R2 buckets, accepting the R2 list-permission residual

Date: 2026-07-30 Status: Accepted Deciders: Sean Brandt

Context

Milestone v1.5 publishes a personal file-sharing surface at i.hl-assets.dev, backed by Cloudflare R2. Its core value is that a shared link keeps working while the cluster is down, which means no tier below the Cloudflare edge may participate in a public GET. hl-assets.dev was registered through Cloudflare Registrar on 2026-07-29 specifically so that the public surface carries no name derived from the homelab estate.

Two facts about R2 shape everything below.

First, an R2 custom domain is read-only. It serves GET and HEAD on a known key and nothing else: there is no listing surface, no directory index, no index.html resolution, and presigned URLs do not work against it. PutBucketWebsite, GetBucketPolicy and PutPublicAccessBlock are unimplemented, so there is no public-access setting to disable — the hardening is by construction rather than by configuration. Writes therefore have to go somewhere else entirely: the signed S3 endpoint at <account>.r2.cloudflarestorage.com.

Second, R2 offers no write-only permission. Every write-capable permission group also grants read and list. There is no narrower group to ask for; the only least-privilege lever R2 exposes is the token’s resources selector, which confines a token to named buckets.

The uploader that Phase 4 builds runs as an in-cluster pod and must be able to write. That forces a decision about what a compromise of that pod costs, and about where retained original files and their metadata live. This record fixes both, and fixes the surrounding key-format and salt decisions that later phases inherit, before any of it is built.

Decision

Split the surface into a public read path that terminates at the Cloudflare edge and a write path that reaches R2 only through the signed S3 endpoint, backed by two separate buckets — hl-assets-public behind the custom domain and hl-assets-private with no custom domain at all — because an R2 custom domain is structurally incapable of accepting a write, and because anything sitting in a bucket that a custom domain fronts is world-readable by URL, so a “private prefix” inside the public bucket would be private to nobody who matters.

Accepted alongside it, explicitly rather than by omission: the workload token holds R2 read and write over both buckets, and that residual is accepted at its full radius as set out below.

Rationale

The read path cannot become a write path. This is the property that makes the split more than an intention. A custom domain is read-only in the API, and presigned URLs — the usual way an object store accepts a browser upload — do not function on one. The published hostname is not a hardened write endpoint; it is not a write endpoint at all.

Two buckets, not one bucket with a private prefix. The recipient of a shared link already knows the key. A sidecar at i.hl-assets.dev/meta/<hash>.json is therefore public to exactly the people it is meant to hide from. A second, differently-salted key would work in principle, but then the public bucket holds private data protected only by key entropy — and the uploader’s credential can list it anyway. With a second bucket, “the public bucket contains nothing private” holds by construction.

What each store holds. The public bucket holds exactly three namespaces and nothing else: f/ for shared assets, _health/ for the read-path canary, and robots.txt at the root. That is a stated invariant, and it is auditable — listing the bucket either shows those three or it does not. The private bucket holds the whole original file unmodified at orig/<hash>, a small JSON record carrying the original filename, upload timestamp, uploader identity, content-type and size, and the tombstone/ namespace. Tombstones live on the private side because a publicly fetchable tombstone would confirm “this was taken down” to anyone still holding a dead link — which is precisely the fact worth protecting.

Retaining the whole original is deliberate. It is lossless, it needs no EXIF parse-and-reassemble logic, and it cannot lose information. It roughly doubles storage, which at R2’s per-GB-month pricing and family volume is pennies. Storage is the cheap resource here; parsing logic is the expensive one.

Consequences of the published key format. Two properties are fixed here because they are properties of the URL rather than of the uploader, and every later phase inherits them. The key carries the extension only and never the original filename, so nothing derived from user content travels through chat logs and Referer headers; the extension is retained because cache handling and client content-type sniffing both key off it. And because keys are content-addressed, two uploads of identical bytes collapse to one key: the first write’s record is authoritative, later identical uploads do not overwrite it, and the collision is surfaced to the caller rather than swallowed. That gives the upload endpoint two distinct success shapes — fresh upload and dedup hit — which every ingest surface must render explicitly. The failure mode this guards against is three of four surfaces silently reporting plain success.

The x-amz-meta-* question, stated carefully. An earlier draft of this design assumed that storing stripped EXIF in R2 object custom metadata would serve that EXIF back to every viewer, because S3-compatible custom metadata is returned on GET. Cloudflare’s own presigned-URL documentation says the opposite: public buckets do not render custom metadata into response headers. This record therefore does not assert either behaviour as established fact. The argument that survives regardless is stronger: relying on undocumented header suppression as a security boundary would be wrong architecture whatever the current behaviour is, because it is inconsistent with the S3 endpoint’s treatment of the same object and because Cloudflare could change it without notice. The private bucket removes the dependency by construction. Note also that R2’s object custom-metadata ceiling is 8,192 bytes, not the 2 KB this design’s earlier notes assumed — so the size argument is weaker than it looked, and is not what the decision rests on.

Probe M — observed 2026-07-31. A throwaway object was written to the public bucket carrying one custom metadata key with a sentinel value, then read three ways. Over the custom domain — plain HEAD, and again with an Origin header so the browser-shaped case was covered — the response carried no x-amz-meta-* header and no trace of the sentinel. Over the signed S3 endpoint, head-object on the same object returned Metadata: {"probe": "<sentinel>"}. So a public custom-domain GET does not serve object custom metadata, while the S3 view of the identical object does — which is what Cloudflare’s presigned-URL documentation says, and the opposite of what an earlier draft of this design assumed.

The S3 read is a deliberate positive control, not decoration: without it, “no x-amz-meta-* over the custom domain” would be indistinguishable from a put-object whose --metadata never took, and the probe would be reporting a null result as a measurement. The throwaway object was removed by a cleanup trap registered on the line after it was created; the public bucket ends the phase holding only _health/canary.

This changes nothing, deliberately. The two-bucket split is not relied on being confirmed here, and header suppression is not relied upon as a security boundary regardless of this result — it is undocumented for this path, inconsistent with the S3 endpoint’s view of the same bytes, and Cloudflare could change it without notice. The private bucket removes the dependency by construction, which is the only form of the argument worth having. Had the probe come back the other way, this record would read the same. Evidence: .planning/milestones/ws-image-host-r2-2026-09-19/milestones/v1.5-phases/01-read-path-domain/probe-m.observed.txt.

The CAS salt is not rotatable — with an exit condition. Object ids are salted content-addressable. Rotating the salt does not error; it silently stops deduplication, because the same file now hashes to a second key and lands twice. The salt exists only because there is no database: no database means the hash must be the URL, which means the hash must be salted, which means the salt is permanent. That chain is the whole justification, and it has an exit condition: if a metadata lookup table ever lands, storage key and public id decouple, public ids can become opaque, and the salt can be retired. This is a decision with an exit condition, not a permanent law, and it is recorded that way so that a future reader does not treat it as one.

The one-way door. cloudflare_r2_custom_domain supports no terraform import on provider 5.22.0 (upstream PR #7267 pending). One dashboard connect is permanent, unfixable drift. The mitigations are therefore preventive only: the resource is created by Terraform on the first attempt, and two MUST-NOT-Import rows in tf/cloudflare/CLAUDE.md — one for the resource, one for the proxied CNAME the R2 API creates — exist before the resource does.

R2 bucket locks are not claimed as a structural guarantee. The workload token holds the account-level R2 write group scoped through resources. Whether that scope also gates lock mutation is untested. Any future sentence about bucket locks protecting these objects must carry that caveat until someone measures it.

A corollary the phase gets for free — and the one claim in it that measurement corrected. Because a miss on the custom domain returns a Cloudflare-branded HTML 404 rather than the S3 XML error, the <Error><Code>NoSuchKey</Code> body never reaches the public. That body carries no bucket name either, but it does mark the surface as S3.

An earlier draft added “The HTML page does not.” That is not what was observed. The verbatim body was captured on 2026-07-31 and committed as .planning/milestones/ws-image-host-r2-2026-09-19/milestones/v1.5-phases/01-read-path-domain/404-body.observed.html: 27 KB of Cloudflare-branded HTML titled Not Found, reading Error 404 / Object not found, asking “Is this your bucket?” and linking to developers.cloudflare.com/r2/data-access/public-buckets/. It marks the surface as R2 as plainly as the XML marks it as S3.

That does not weaken the design, because vendor attribution was never the property being protected — non-correlation with the estate was, and the captured body holds it: no bucket name, no shared hl-assets- stem, no R2 S3 account endpoint, and no listing or directory-index marker, verified across eight probe paths on both bodies and headers. Anyone can see the asset is served from R2; nobody can see whose R2 it is. The claim is narrowed to what was measured rather than left overstated.

How that is asserted matters as much as the result. The gate is written purely as an absence of bucket name, shared stem, S3 endpoint and listing markers — never as a match against any fixed 404 string. That page is Cloudflare’s, is not customisable below the Pro plan, and its text has already changed at least twice across the years it has been reported. A positive match would be a latent false failure that passes today and silently breaks later, and it would propagate into Phase 6’s monitoring, which inherits this gate. The body is committed verbatim so that a future vendor change arrives as a reviewable diff rather than as a surprise.

Accepted residual risk: the uploader credential reads and writes both buckets

This is the risk being accepted, stated at full radius. An accepted risk that understates its own blast radius has not actually been accepted.

The uploader pod holds one Terraform-minted cloudflare_account_token whose resources map carries both buckets — one token, two entries. R2 offers no write-only group, so that credential reads and writes everything in both buckets. Concretely, a compromised uploader pod can:

Enumerate every object key in the public bucket. The key is the capability, so this leaks every link ever issued.
Read the unmodified originals at orig/<hash> — including the EXIF and GPS data that was stripped from the published asset. The stripping protects the public reader; it does not protect against this credential.
Read the JSON records, which carry original filenames, upload timestamps and uploader identity.
Read the tombstone/ namespace, disclosing what was taken down — the exact fact moved to the private side to keep it from leaking.
Delete across both buckets. DeleteObject is a free operation, so mass deletion produces no billing signal, and destroying the retained originals also destroys what makes the reversible unshare verb reversible.

Bounds. The exposure stops at these two buckets. The token is a workload token, not the bootstrap token, and its resources selector confines it to the hl-assets pair — the Longhorn and Velero backup buckets are unreachable from it, which is what the isolation requirement demands and what a later phase’s 403 test proves. Least privilege here comes from the resource selector, because R2 offers no narrower permission group.

Why it is accepted. The precondition is compromise of an in-cluster pod, at which point the attacker holds cluster access and there are larger problems; and the assets are copies of originals held elsewhere. Compensations: a dedicated bucket pair rather than a shared one, a dedicated Vault path, the resources selector, an egress NetworkPolicy in Phase 3 restricting the pod to the R2 S3 endpoint, and a Phase 4 rule that no List* call appears anywhere in application code so the capability is demonstrably unused.

Alternatives Considered

Two buckets, public + private (chosen) — the public bucket carries a custom domain and holds only f/, _health/ and robots.txt; the private bucket carries no custom domain and is reachable only through the signed S3 API.
One bucket with a private prefix — rejected. Anything inside a bucket fronted by a custom domain is world-readable by URL, and the holder of a link already knows the key, so the “private” prefix is public to precisely the people it would hide from.
R2 object custom metadata (x-amz-meta-*) for the retained EXIF — rejected. Whether or not those headers are served on a public GET (see above), depending on undocumented header suppression as a security boundary is the wrong shape of argument, and it is inconsistent with the S3 endpoint’s view of the same object.
Presigned direct-to-R2 uploads instead of a mediating pod — rejected for this milestone. R2 has no write-only permission, so presigning would not have shrunk the pod’s credential at all; it would only have moved where the same credential is used. Presigned upload for large video remains backlogged as additive future work.
A Worker in the read path to shape responses or block listing — rejected. It would put compute back into a public GET, which is the one thing the read path is defined not to have; and there is no listing surface to block.

Consequences

Positive:

A shared link is served entirely by the Cloudflare edge. No tier below it participates in a public GET, so the link survives a full cluster and tunnel outage.
“The public bucket contains nothing private” is true by construction, not by key entropy.
The published read surface cannot accept a write, structurally.
The public bucket’s three-namespace invariant is auditable with one listing call.
The accepted residual is written down at full radius, so a later reader inherits the real bound rather than a narrower restatement of it.

Negative:

The uploader credential reads and writes both buckets, with the radius set out above. This is accepted, not eliminated.
Retaining whole originals roughly doubles storage.
The CAS salt is permanent operational state that must never be rotated, and nothing enforces that mechanically — rotation fails silently rather than loudly.
cloudflare_r2_custom_domain cannot be imported on provider 5.22.0, so the one-way door stays open and the guard is detection rather than prevention.

Neutral:

Bucket names never appear on the public read path, so repository naming convention does not conflict with the non-correlation goal.
Whether a public GET serves x-amz-meta-* was measured rather than assumed. Observed 2026-07-31: it does not — the custom domain returns no such header while the signed S3 endpoint returns it for the same object. As stated when the probe was specified, the outcome changed the record’s wording and not its decision; the private bucket would still be the right answer had it gone the other way.
The 404 body identifies the surface as Cloudflare R2. That is vendor attribution, not estate correlation, and only the latter was ever the property under protection.

Sign-off: decision, date (2026-07-30) and deciders (Sean Brandt) are recorded above. Human approval of the pull request that lands this ADR is the sign-off for the accepted residual.