The hl-assets write path: key the object by the bytes actually stored, with cross-architecture encoder determinism measured and the derivation frozen; the audit record written last as the commit marker; video and camera-RAW refused

Date: 2026-08-02 Status: Accepted Deciders: Sean Brandt

Context

Milestone v1.5 splits hl-assets.dev into a read path served entirely from Cloudflare’s edge and a write path that runs in the cluster. The split read/write ADR fixed the architecture; the edge policy ADR fixed what the read path does; the credential-isolation ADR fixed what the uploader’s R2 token can reach. This record fixes the write path itself — what the object key is derived from, what the service does to an uploaded file before storing it, what it refuses, and in what order it writes.

This is a third hl-assets ADR rather than an amendment to either sibling, for the reason Phase 2 gave for splitting its own record out: the documents have different lifetimes. The split read/write architecture is settled for the life of the milestone, and the edge rules are explicitly revisited in Phase 6 against measured usage. These are the write path’s own decisions, and one of them is a door that has now been closed.

Two facts force the shape of the work.

First, the object key is a function of the file’s contents, and that function is a one-way door. Keys are handed to people outside the household, in chat threads and text messages, with no channel to notify anyone. Any change to how a key is derived — the encoder, its quality setting, what the hash is taken over — re-keys everything: the old URLs keep working, deduplication silently stops, and the same photo begins landing twice. There is no error at any point in that sequence.

Second, nothing in this design may list a bucket. R2 offers no write-only permission, so the uploader’s token unavoidably can list; the credential-isolation ADR accepted that residual and made “no List* call anywhere in the codebase” an obligation on this phase. A design that cannot enumerate has to be correct by ordering rather than by sweeping up afterwards, and that constraint decides the write sequence below.

The discipline this record applies

Phase 1’s ADR asserted a property of the vendor 404 page that measurement later contradicted, and the claim had to be narrowed after publication. Phase 2’s ADR named the practice that came out of it and this one continues it: every claim below carries either the committed observation file that supports it, or an explicit argument by construction label. Where a measurement contradicted a prediction, the measurement is what is written down.

This record adds one more rule of its own, because much of what Phase 4 built has not yet run in production. The image is not published, nothing is deployed, and several assertions are written and reviewed but have never executed. Those are collected in What is not yet true and named individually. A reader who takes any claim outside that section as live-verified, or any claim inside it as pending paperwork, has been misled by this document rather than by the system.

AMENDED 2026-08-04. The paragraph immediately above is preserved verbatim and superseded: the image is published (0.1.2), the service is deployed and pinned to its OCI index digest, and the assertions it describes as never having executed have now executed and passed. The What is not yet true section is superseded in full and carries its own amendment header listing what replaced each row. The rule this paragraph introduces still stands — it is simply no longer load-bearing, because the set it governs is now empty. See W-12 in 04-VERIFICATION.md.

Decision

Key every public object by HMAC(salt, the bytes that are actually stored); transcode HEIC to WebP with a frozen encoder; strip metadata from everything else by container surgery that never decodes; refuse video and camera-RAW; bound memory rather than file size; and write the audit record last, conditionally, as the sole commit marker.

Area	Shape	Decisions
Service home	Go module at apps/hl-assets-uploader/, its own Dockerfile, its own workflow	D-47
Build	Native per-arch build on amd64 and arm64 runners, then docker manifest create	D-48 / PORT-03
Image	Private GHCR package, pinned by semver and digest	D-49, D-50
UI	Embed seam scaffolded, no UI shipped	D-51
Transcode	HEIC → WebP via libheif + libwebp through cgo, glibc-linked	D-52, D-53, D-54
Passthrough	JPEG / PNG / GIF / WebP metadata-stripped losslessly, never re-encoded	D-55
Key	HMAC(salt, stored bytes), base32, 16 characters	D-56
Cross-arch	Measured identical on amd64 and arm64; the derivation is frozen	D-66
Admission	50 MB byte cap, 50 Mpx pixel cap, disk-backed spill, in-process byte budget	D-57, D-58, D-59, D-60
Identity	Cloudflare Access headers read but not verified; unauthenticated accepted	D-61, D-62
Record	One immutable JSON object per upload, written last with If-None-Match: *	D-63
Telemetry	Every upload attempt emitted over OTLP, including dedup hits	D-64
Dedup signal	201 fresh / 200 dedup, plus a response header on both content types	D-65
Refusal	Video and camera-RAW rejected with a structured 415	D-67

Rationale

Where the service lives, and how it is built — D-47, D-48, D-49, D-50, D-51

D-47: a new top-level apps/hl-assets-uploader/. Argument by construction. images/ was rejected because all eight entries there are thin wrappers around upstream images — a Dockerfile and at most a shell script — so a Go module there redefines what the directory means. tools/ was rejected because it means developer and CI tooling, not a deployed runtime service. A separate repository was rejected because it splits one phase’s work across two repos and out of the planning system’s reach. Reversibility is costly: the module path appears in every import, so moving it later is a mechanical but repo-wide rename.

D-48: native per-arch build, no cross-compilation. Argument by construction, with a repo-wide observation. Six workflows in this estate already build this way — one job on the amd64 runner profile, one on arm64, then a create-manifest job. The cross-building directive appears in zero workflows here, not because multi-arch is absent but because the estate builds natively per architecture rather than under emulation. That choice pays for itself twice in this phase: it is what satisfies PORT-03, and it is what made D-66’s measurement nearly free, because both native runners were already in the pipeline.

D-49: the image is pinned by semver tag and digest, with Renovate opening the bump. Matches headroom-otel and firewalla-mcp. The short-SHA style used elsewhere was rejected because it carries no digest and needs a manual manifest edit per deploy. The digest half is not in force yet — see What is not yet true.

D-50: the GHCR package is private, with a ghcr-pull-secret ExternalSecret in the namespace, matching headroom-agents, firewalla-mcp and temporal-workers. It reuses the shared fzymgc-house/cluster/ghcr/pull-secret Vault path, so no new path and no new Vault grant. Argument by construction; the grant chain was checked rather than assumed.

D-51: the Svelte embed seam is scaffolded and ships no UI. The embed.FS mount point, the static-file handler and the Node build stage all exist and compile an empty bundle. Adding a Node stage in Phase 5 would mean re-testing the dual-architecture build, which is the expensive part; doing it now costs nothing. A hand-written stub page was rejected — that is exactly what the Svelte decision replaced.

One measured detail belongs here because it will otherwise be rediscovered: a bare //go:embed dist directive fails the build against a directory holding only a dot-file, because a bare directory pattern excludes every entry whose name starts with . or _. The all: prefix is what makes the empty seam compile. Observed both ways during 04-06.

The transcode, and what the key is taken over — D-52 … D-56

D-52: HEIC is re-encoded to WebP, for renderability alone. The requirement asks for a “web-renderable and edge-cacheable” format, but Phase 2 already delivered the cacheable half — the /f/* cache rule applies Cache Everything with a 30-day edge TTL and a .heic was measured going MISS → HIT. Evidence: .planning/milestones/ws-image-host-r2-2026-09-19/milestones/v1.5-phases/02-edge-policy/cache-heic.observed.txt. So the target format did not have to be chosen for cache behaviour; the surviving reason is that HEIC does not render on Android or in most desktop browsers.

D-53: decode and encode through cgo, in-process. Go has no standard-library WebP encoder and golang.org/x/image/webp is decode-only. Shelling out to a converter was rejected on three counts: a subprocess in the request path, a temp-file dance that brushes against PORT-01, and a whole image-processing suite in the runtime image.

D-54: plain cgo against gcc and glibc on a Debian trixie base, not a static musl link. A hermetic pinned C toolchain with static linking was considered and rejected as unnecessary risk: the native per-arch build means nothing is cross-compiled, so that toolchain’s headline feature is unused here, while statically linking libheif’s dependency tree against musl is a known-fiddly exercise nothing in this repo has ever done. Dynamic linking is also the licence-clean choice — libheif and libde265 are LGPL, and dynamic linking is the arrangement that keeps the obligation trivial rather than requiring relinkable object distribution.

One correction to the plan of record, found on the first build and worth stating because the two pins look independent and are not. The libheif Go binding is upstream libheif’s own, and its module version is the C library version. Binding v1.23.1 against runtime libheif1 = 1.19.8-1 fails with seventeen could not determine what C.… refers to errors, and the reverse fails too. Both are pinned to 1.19.8 and the Dockerfile records that they move together or not at all. Observed during 04-01.

D-55: non-HEIC images are metadata-stripped and passed through unchanged. Normalising everything to WebP was rejected — generation loss re-encoding an already-lossy JPEG, lossy output for a lossless PNG screenshot, CPU on every upload. A long-edge resize was rejected for a sharper reason: resizing changes the post-strip bytes and therefore changes the key, so the deduplication story would have to absorb it.

The strip is segment and chunk surgery over a default-deny keep-list, and nothing on any strip path decodes. For JPEG the byte range from the scan marker to the first end-of-image marker is copied verbatim; for PNG, the image and colour chunks; for WebP, the image and profile payloads. Two independent confirmations: the entropy-coded data is byte-identical across the strip, and ImageMagick’s compare -metric AE between a stripped JPEG and its source is 0 — 1,192 bytes of metadata removed, not one pixel touched. Observed during 04-02.

Taking the first end-of-image marker rather than the last is a decision, not an implementation detail: an Apple HDR JPEG appends a whole second image (the gain map) after the first marker, and so does every polyglot file with a payload glued to the tail. Taking the last would carry both into the stored object.

D-56: the key is HMAC(salt, the bytes that are actually stored publicly) — the WebP for a HEIC upload, the stripped JPEG for a JPEG upload. The key always describes the object it names, and deduplication means “this exact public object already exists”. Hashing the pre-transcode bytes would have been stable across encoder upgrades but breaks that property; hashing the raw uploaded bytes defeats the purpose, since the same photo exported twice with differing EXIF would no longer collapse to one key.

The risk this creates is stated, not mitigated away. An encoder version bump can change output bytes, so the same source file re-uploaded after an upgrade hashes differently and lands twice. The mitigation is that the encoder is pinned and the derivation is frozen — which converts a silent event into a detected one, and does not make it impossible. See the next section for what “detected” costs.

The one-way door: cross-architecture determinism, measured — D-66

The question. D-56 derives the key from libwebp-encoded bytes, and libwebp dispatches hand-written SSE kernels on amd64 and NEON kernels on arm64. Whether they are bit-identical for the same input is not documented upstream and was not resolvable from documentation. Nothing breaks today — every node in this cluster is arm64 — but PORT-03 ships an amd64 image specifically so the service can relocate to Cloudflare Containers, which runs x86. If the architectures diverge, that relocation silently lands every subsequent upload of already-stored content under a second key.

Three outcomes were pre-agreed before the measurement was taken, so that the gate presented a decision rather than reopening the question: (a) arches agree → accept and record; (b) arches diverge → hash the decoded pixel bytes instead, which reverses D-56; (c) arches diverge and (b) is unacceptable → build the encoder with SIMD disabled.

The measurement. The same golden HEIC was decoded and re-encoded inside the shipped image on a native amd64 runner and a native arm64 runner, in one CI run, and both lines were recorded verbatim before the gate was presented:

Job	Runner	SHA-256 of the stored WebP	Derived key	Bytes
test-amd64	namespace-profile-linux-amd64-2x4	f889e87f…3271bd21	ykba5txppspa3235	1308968
test-arm64	namespace-profile-linux-arm64-2x4	f889e87f…3271bd21	ykba5txppspa3235	1308968

Compared as a set: size 1. Compared as a set rather than pairwise on purpose — a pairwise assertion passes vacuously if one job never emits a line at all.

The outcome, operator-decided 2026-08-02: (a) arches-agree. Outcomes (b) and (c) were both conditioned on divergence and were therefore inadmissible on this measurement. D-56 stands unreversed. The operator confirmed both lines against the CI logs before deciding rather than taking the evidence file on trust. Evidence: .planning/milestones/ws-image-host-r2-2026-09-19/milestones/v1.5-phases/04-uploader-service/encoder-crossarch.observed.txt.

Then the baseline was re-frozen once, deliberately, and that must be on the record too.

The transcode originally rebuilt the WebP from decoded pixels alone, so nothing from the source container survived — including the 536-byte Display P3 ICC colour profile that every iPhone photo carries. The stored object held exactly one chunk. Every HEIC upload was therefore served untagged, browsers assumed sRGB, and Display P3 primaries rendered oversaturated. The plan that found this deliberately did not fix it, and instead encoded the broken behaviour as a tripwire, precisely because emitting a profile changes the stored bytes and therefore re-keys every HEIC asset. The tripwire worked; the operator was asked; the answer was to spend the window while the bucket was still empty.

	before	after
SHA-256	f889e87f…3271bd21	e2bbd0d480cd645229ca4683576df6663a2f9033e14765aed541bfde1d3e265e
key	ykba5txppspa3235	m7fazwalhih2wo5c
bytes	1308968	1309530 (+562)
chunks	[VP8 ]	[VP8X ICCP VP8 ]

The +562 accounts to the byte — 8 + 10 for the extended header, 8 + 536 for the profile chunk — which is the check that the container grew by the profile and by nothing else. The re-measurement was taken the same way as the original: both native runners, one CI run, compared as a set, size 1. The arches still agree, and D-66’s outcome is unchanged. Only the frozen value moved. The superseded measurement is retained in the evidence file rather than deleted, because it is the record of what the gate was originally answered from. Evidence: the same file.

This re-key was free only because the bucket was empty and nothing was deployed. That window is now closed. After the first production upload there is no cheap window left, and any future change to the stored bytes re-keys every asset already issued.

What is now unchangeable without re-keying every asset ever issued:

1. The encoder versions — chai2010/webp v1.4.0 with its vendored libwebp, and libheif 1.19.8 in both the Go binding and the apt pin. Their output bytes are the HMAC input.
2. keys.WebPQuality = 82. A different quality is a different WebP.
3. D-56 itself — the key is derived over the bytes actually stored, not over decoded pixels.

The consequence nobody should have to rediscover: a libwebp security update is also a deduplication-breaking event. Those two properties are unrelated in every other service in this estate and are the same event here. There is no version of this where a CVE fix in the encoder is a routine bump.

The single detection mechanism is TestEncoderOutputIsFrozen, comparing against testdata/golden.webp.sha256 and testdata/golden.key, running in both native build jobs. It is the only thing that will notice if any of the three above moves, and it must stay in CI on both runners forever rather than being trimmed later as redundant duplication — running it on one architecture tests the encoder, running it on two is what tests the relocation. A red golden-file test is a decision point, not a flake, and its own failure message says so in the place a future reader will actually meet it. Re-baselining it as a routine test fix is forbidden; the ICC amendment above is what a permitted re-baseline looks like — recorded, accepted, and naming exactly what changed.

Renovate would otherwise have taken that decision unattended, and this was checked rather than assumed. .github/renovate.json’s “all non-major dependencies” rule matches everything and carries automerge: true with platformAutomerge: true, so a minor bump of the WebP binding would have merged before 4am on a Monday and re-keyed the world. Renovate evaluates packageRules in order with later rules overriding earlier ones, so position is the whole mechanism: a key-affecting-encoders rule sits at index 19, the last entry, after the automerge group at index 1. It names github.com/chai2010/webp, github.com/strukturag/libheif, libheif1 and libheif-plugin-libde265, with automerge: false, a 30-day minimum release age and a dedup-affecting label. Verified by reading the parsed array’s indices rather than by inspection.

Two of those four names are machine-tracked; two are not, and the difference matters. The gomod half — github.com/chai2010/webp and github.com/strukturag/libheif — is real: those are exactly the dependencies Renovate could otherwise automerge unattended, and the rule stops it. The apt half is decorative. libheif1 and libheif-plugin-libde265 are pinned by ARG LIBHEIF_VERSION=1.19.8-1 interpolated into a RUN apt-get install in apps/hl-assets-uploader/Dockerfile; Renovate’s dockerfile manager reads FROM lines and nothing else, the ARG carries no # renovate: annotation, and no customManager matches it. Renovate will therefore never open a PR for those two, and the rule that would have protected them never fires.

The control is not defeated by this, but it is narrower than the rule’s own description implies. An apt bump can only reach the image through a human editing that ARG, and the golden-file test running on both native runners is what catches that. So the honest statement is: the automated half of the guard covers the automated half of the risk, and the manual half of the risk is covered by a test, not by a package-manager rule. Closing the gap means adding a customManager with a deb datasource — which must be observed opening a real PR before it is described as a guard, on pain of repeating exactly the mistake this paragraph is correcting.

If that rule is ever removed, the golden-file test is the only thing left. It would fail loudly on the resulting pull request, which is the intended backstop — but the backstop only works while the test runs on both architectures. Deleting the rule and trimming the test are individually survivable and jointly silent.

Admission control: bound the memory, not the file — D-57 … D-60

D-57: the app-enforced ceiling is 50 MB, below Cloudflare’s roughly 100 MB proxied-body cap, so a caller meets this service’s structured JSON refusal rather than Cloudflare’s HTML one. 25 MB was rejected as leaving no headroom for burst and Live Photo bundles; 90 MB was rejected because the margin to the HTML 413 would be thin and the buffer worst case hostile on arm64 nodes.

A byte cap on the request body does not bound memory, and that is the whole reason internal/admit exists. Peak resident memory is a function of pixel count, not file size: a 45-byte PNG can declare 20000×20000 and ask for 1.6 GB of RGBA. Measured — the test fixture is 45 bytes on disk declaring 400,000,000 pixels. So the service probes dimensions from the container header, before any decode, refuses above 50 Mpx and above WebP’s own 16383-pixel side limit, and then reserves max(fileSize, w × h × 4 × 2) against an in-flight byte budget. Each dimension is validated before the multiplication, because a wrapped product is a small positive number that grants rather than refuses. Observed: with the per-side check removed, a 65536×65536 image was granted a 1 KB reservation.

D-58: uploads spill to a disk-backed scratch volume above a threshold, so peak memory is independent of upload size. Disk-backed rather than memory-backed deliberately: a tmpfs volume is RAM and counts against the pod’s memory limit, so it would give the ergonomics of a file with none of the memory relief that is the entire point of spilling.

PORT-01 holds, and the reading is recorded here so a later auditor does not have to guess. PORT-01 forbids “a database, a PVC, or a filesystem write that must survive a restart”. The spill is request-scoped scratch that must survive nothing — the file is unlinked immediately after creation, so it has no directory entry for the rest of its life. internal/admit holds the only file-creating call site in the module, asserted by parsing the source and comparing call sites as a set in both directions. Observed red three ways: an extra call site in the same package, one in a different package, and an expected entry that no longer exists.

One claim in the plan of record was measured false and weakened rather than quietly satisfied: “the spill directory never holds an entry at any observation point” is not achievable. Creating and unlinking are two syscalls and the window between them is real — a poll during 32 concurrent writers found five entries. The assertion is now the strongest statement that is true: no spill file outlives its own creation. The residual is a microsecond-wide window in which a SIGKILL leaves at most one file, which the node reclaims with the pod.

D-59: concurrency is bounded by a byte budget on total in-flight bytes, not a request count. At a 50 MB ceiling, request count is a poor proxy for memory — ten 1 MB uploads and one 50 MB upload must not weigh the same. Over-budget requests receive a structured JSON 503 carrying Retry-After; never HTML, never a redirect, never a hang.

D-60: horizontal autoscaling is deferred, and was rejected for this purpose. An autoscaler reacts over tens of seconds while an upload burst is over in seconds, so it cannot deliver a zero-restart criterion — only in-process admission control can. No autoscaler object of any kind exists anywhere under argocd/ today, and this phase deliberately did not introduce the estate’s first one as a side effect.

Identity, the audit record, and the write ordering — D-61 … D-65, D-63 in detail

D-61: Cloudflare Access headers are read and not verified. The authenticated-user email, or the service-token client id for scripted clients, is recorded as-is with an explicit unauthenticated marker when absent. The reasoning is that the record shape is final in Phase 4 and only the trust changes in Phase 5 — adding JWT verification in front of the identity read re-keys nothing and migrates no record. Full validation now was rejected because it pulls Phase 5’s auth work into a phase whose criteria are verified by port-forwarded curl, before Access is configured at all.

D-62: uploads carrying no identity are accepted and recorded as unauthenticated, not refused. Phase 4’s own success criteria are verified without Access in front, so refusing would make them unverifiable. Phase 5’s Access layer is what makes an unauthenticated request unable to reach the pod at all.

D-63: the audit record is one immutable JSON object per upload at records/<hash>.json in the private bucket, written with a conditional put so it is created once and never overwritten. That is first-write-wins enforced by R2 rather than by application logic, and append-only holds by construction. Appending to a rolling log object was rejected outright: object storage has no append, so it means read-modify-write, which races under concurrency and makes the private bucket’s hottest object the one most likely to be corrupted.

The write ordering, and its residual. The service writes public object, then retained original, then record — the record last, and conditionally. That reads as the wrong order until you price the intuitive one.

Observed, against a mutant with the record written first and a failing public write injected:

CONFIRMED WRONG SUCCESS: the record survives a failed public PUT, so the retry reports a
DEDUP HIT (Exists=true, State="live") and the service returns a URL for an object that was
never stored. public bucket holds: []

public bucket holds: [] is the finding. The record says the asset exists; the bucket is empty; the client holds a URL that 404s permanently, because first-write-wins means the record is never revisited. Under the shipped ordering the same injected failure writes nothing at all, so a retry starts clean.

The residual the ordering leaves, stated rather than argued away. If the public write succeeds and the record write does not, an orphaned public object exists that nothing can find — a design that forbids listing has no sweep. That is exactly why D-64 emits an event for every attempt including the failure, and why store distinguishes an orphaned-public-object outcome from the other two rather than folding it into a generic error: the emitted event is the only reconciliation channel an orphan will ever have. Phase 6 owes a query against that stream. Until it exists, the ledger is being written and nobody is reading it.

That R2 honours the conditional write as create-once was measured against live R2, not assumed. A second conditional put to the same key returned PreconditionFailed, and — the half that is easy to skip and is the whole point — reading the body back confirmed the first write survived. A 412 that still let the second write land would satisfy “the second call failed” while breaking first-write-wins entirely. Evidence: .planning/milestones/ws-image-host-r2-2026-09-19/milestones/v1.5-phases/04-uploader-service/r2-conditional-put.observed.txt, sections (a1) / (a2) / (a3).

The D-63 carve-out, stated plainly because a naive reading forbids it. “Created once and never overwritten” means immutable with respect to uploads, which is precisely what the conditional put enforces. It does not mean the object is frozen against the operator. Phase 6’s unshare and purge are operator actions and they must be able to rewrite the record’s state metadata — that is where the tombstone lives, and it is what lets one HeadObject serve both the deduplication check and the tombstone check on the same round trip.

That mechanism was measured live rather than deferred to Phase 6 to discover: CopyObject with the metadata directive set to replace rewrote the object’s state from live to unshared, confirmed by reading the metadata back. Evidence: the same file, sections (b1) / (b2). Two details Phase 6 should carry forward rather than re-derive: the copy returned the same version id it started from, so on this bucket the rewrite is in place rather than a new version; and the metadata key is reported lower-cased with the vendor prefix stripped, matching what the SDK presents.

D-64: every upload attempt is emitted over OTLP to ClickStack, including dedup hits, which write no record. This captures attempts, not only outcomes. The endpoint arrives from an environment variable — an in-cluster DNS name hard-coded in application source would violate PORT-02 — and the emitter batches, never blocks the request path, and degrades to a no-op when the collector is unset, malformed or unreachable.

Getting that stream to actually arrive took two changes, not one, and the second was found rather than planned. The namespace’s network policy permitted only DNS and TCP/443 to the world, so egress to the in-cluster collector was denied. Amending the policy alone would have moved the failure from “dropped by policy” to “401 discarded by the collector” — because cs-otel-collector rejects unauthenticated OTLP, as six other senders in this estate already document. Both failures are silent from the sender’s side, and both leave the ledger empty. The namespace therefore also projects an ingest-token Secret into OTEL_EXPORTER_OTLP_HEADERS, read by the OpenTelemetry SDK directly from the process environment. The token is presented raw, with no Bearer scheme — the collector’s rejection message echoes the presented token into sender logs, so a scheme mismatch is a credential exposure rather than a configuration typo.

D-65: the deduplication signal is carried redundantly, by both a distinct status code (201 fresh, 200 dedup) and an explicit response header on both content types. This resolves a real conflict: under Accept: text/plain the body is the bare URL, so “your bytes already existed and your filename was not recorded” has nowhere to live in the body. The redundancy is deliberate — the worry is a surface silently reporting a dedup hit as plain success, and a client that ignores one signal still catches the other. A 200 on the dedup path does not contravene the phase’s success criterion, which itself names two success shapes.

The capability token that permits a later takedown is returned on a fresh upload only, in a header, with no re-issue path by design. Observed red: issued unconditionally, both content-type cases failed. Phase 5’s four ingest surfaces must capture and persist it — a client that drops it loses delete capability for that object permanently.

Refusal: video and camera-RAW — D-67

Video and camera-RAW are refused with a structured 415 whose body points at the backlogged presigned direct-to-R2 path. This narrows what the roadmap left open, which is why it was escalated rather than assumed.

The evidence is specific. An iPhone .mov stores the shooting location as an ISO-6709 string in a container atom, alongside a location-accuracy field and make/model/software fields identifying the exact device and OS build. Accepting it unmodified under the unknown-type-passthrough rule would ship exactly the data the metadata-stripping requirement exists to remove, on a public URL, from the same request handler that strips it from photos. Stripping it instead means walking an ISO-BMFF atom tree and rewriting the movie box — a demuxer’s worth of work that can invalidate chunk-offset tables. That is a project, not a Phase 4 task, and it was rejected on that basis rather than on principle.

The unknown-type-as-attachment rule is unchanged for everything else. Video and RAW are a named refusal tier above it, not a hole in it. The tier is pinned to exactly {video, RAW} by set equality in both directions, so it goes red when it widens and when it narrows — widening silently narrows the attachment promise, narrowing publishes location metadata this phase cannot strip. Observed red in both directions during 04-03.

The classification is total over the type enum, and the detected type decides everything while the claimed type decides nothing: a JPEG named .png and declared image/png is stored with a .jpg extension, with both types recorded separately in the audit record.

Two properties that hold by a different mechanism than the obvious one

Recorded so that nobody adds redundant code for either.

1. There is no extension-versus-content cross-check, and none is needed. A conventional uploader has to worry that a file stored under one extension and sniffed as another type will be served with the wrong Content-Type. That worry requires an origin that resolves keys, and this design has none: the read path is R2 behind Cloudflare with no origin below the edge. Both the stored extension and the stored content type derive from the detected kind alone, so a mismatched extension is simply a different R2 key, which 404s. The property holds by the absence of a component rather than by a check. Argument by construction. Adding the check later would be harmless and pointless; deleting the component that makes it unnecessary would not be.

2. The private bucket now holds a credential, and that belongs on the record. Each audit record carries the capability token minted for its upload. The private bucket therefore stops being purely descriptive metadata and starts holding secrets. Against the accepted residual this adds no meaningful blast radius — a compromised uploader pod already holds a token that reads and lists both buckets, so it can already read every record — but the sentence “the private bucket holds no credentials” is no longer true and should not be written down anywhere as though it were. Argument by construction.

Orientation and the colour profile are kept deliberately — do not “finish the job”

The strip is a default-deny keep-list, and two things survive it on purpose. Both look like oversights to a reader who has just understood the privacy argument, and deleting either is a plausible-looking cleanup.

Exif Orientation is re-emitted, as a synthetic 32-byte block carrying that tag and nothing else. Dropping it rotates every portrait photograph in every browser. The emitted block is asserted structurally rather than by size: the test re-parses it and fails on any tag other than orientation, so “small enough to be harmless” is not what is being checked.

The ICC colour profile is preserved on the strip paths, and — since the amendment above — is carried across the HEIC transcode too. An ICC profile is a colour transform and carries no personal data; the blocks beside it in the same container carry GPS. That is the whole distinction, and it is why the exception is enumerated rather than general: the two helpers that implement the carry copy the profile and nothing else, and there is deliberately no “copy metadata from the source” helper for a later change to reach for.

The privacy argument and the deduplication argument point the same way, which is the part worth carrying forward. Container surgery — splicing bytes, never decoding — is deterministic forever. A decode-and-re-encode is deterministic only for a fixed encoder version, so choosing it anywhere on the strip paths would extend D-66’s one-way door from one format to four. The strip therefore keeps the entropy-coded data byte-identical, and the profile crosses the transcode by a container splice rather than by an encoder option.

Two further guards exist because they are cheap and their absence is silent: the profile is bounded at 4 MiB and an oversized one refuses the upload rather than being dropped, so the function stays total — carried, or refused, with no third outcome where a photo is quietly stored untagged. And a source with no profile emits no profile chunk, rather than a synthesised sRGB one; synthesising would make the stored bytes depend on something other than the source, which is exactly the coupling D-56 keeps exact.

The no-List* bound, discharged

The credential-isolation ADR carried this as an OWED constraint: Phase 4 must prove that no bucket-enumeration call appears anywhere in the uploader’s application code, so that the capability the credential unavoidably carries is demonstrably unused. It is now ESTABLISHED. That amendment is made in that ADR and in REQUIREMENTS.md, with the original wording preserved in both. The controls are:

Control	Where
A blocking CI gate	source-gates job in .github/workflows/build-hl-assets-uploader.yaml, first step of three
Its scoping property	internal/store is the only package that imports the S3 client, so the audit is one package rather than a whole repo
A structural absence	the Store interface declares no list method and no delete method — the mistake is un-writable, not merely discouraged
A Go test in the image build	TestNoObjectListingCallAnywhereInThePackage, which runs in the Dockerfile’s test stage and therefore fails the image build, not only a CI job
Publishing depends on it	build-amd64 and build-arm64 both declare source-gates as a prerequisite, so a push whose gates went red cannot publish

The gate was observed going red against a real, compiling enumeration call appended to the store’s live client — not against a text-only mutation — and green again after the revert. Observed during 04-05.

One caveat that is load-bearing, because without it this control was vacuous. rg is not preinstalled on the runner profile these jobs use. if rg …; then with rg absent is “command not found” — a non-zero exit, so the condition is false and the gate reports clean on any input, permanently and silently. That was discovered only because an availability assertion had been placed in front of the gate; the job now installs ripgrep and asserts its presence before any search runs. A gate’s preconditions need the same treatment as its logic.

Two narrower caveats, recorded rather than glossed. The gate’s exclusion for vendored source is anchored to the job’s working directory rather than to the search path, so it excludes a vendor/ directory at the repository root rather than one inside the module — harmless today because no such directory exists, and silently wrong the day someone vendors. And the scoping property in row two is checked manually rather than by a CI gate; a fourth source gate asserting it is a recommended follow-up that has not landed.

What is not yet true

AMENDED 2026-08-04. THIS ENTIRE SECTION IS SUPERSEDED. Everything below it was measured true on 2026-08-02 and is measurably FALSE today. The original wording is preserved verbatim rather than deleted, because this record’s governing rule is that where a measurement contradicts a prediction, the measurement is what gets written down — and that rule applies just as much when the measurement is better than the prediction.

Read the rows below as a historical snapshot of the pre-release state, not as the current one. What changed: releases 0.1.1 (PR #1774) and 0.1.2 (PR #1782) published; the deployment pinned to 0.1.2’s OCI index digest sha256:100eafe4…add580 (PR #1783); ArgoCD Synced and Healthy with one Ready pod at restarts=0.

Row below	Superseded by
Nothing is deployed	One Ready pod, hl-assets-uploader-66cd657cff-5kh9q, image and resolved imageID both the pinned index digest
Version 0.1.0 is not on GHCR	0.1.2 published by run 30903285558, both platforms
D-49’s digest pin — not in force	In force. Pinned in one commit that appended the digest and deleted the unresolved annotation together; the biconditional test passes in its pinned arm
PORT-03’s docker manifest inspect — has never executed	Executed and passed: job create-manifest, step Emit dual-arch evidence (PORT-03), + observed=linux/amd64,linux/arm64. Re-measured independently at verification
The live upload gate — written, never run against a service	Run against the deployed service; an upload returned its public URL and the event was correlated in ClickHouse by key
The zero-restart load verdict (load-sc5) — unmeasured, reader unwired	Passed at concurrency 32, restartCount unchanged at 0; proved non-vacuous by correctly failing at concurrency 2
The mid-transcode scratch inspection (load-spill) — an explicit abstention	Discharged. Measured in both directions (04-UAT test 8): client abort and pod delete, /spill empty and reservation 0 in each
Any upload event reaching ClickStack — never observed	Observed. ScopeVersion 0.1.2, ResourceAttributes {'service.name':'hl-assets-uploader','service.version':'0.1.2'}, correlated by hl_assets.key. Evidence: otlp-resource-version.observed.txt
The whole-phase gate transcript — not written	The UAT round ran; see 04-UAT.md and 04-VERIFICATION.md round 2 (19/19 must-haves, 15/15 requirement IDs)

Why this amendment exists at all. The gap it closes is the inverse of the one this phase spent seventeen plans avoiding — not a claim outrunning its evidence, but evidence outrunning its claims. It costs the same, because a downstream reader cannot tell the difference and this document is what the security auditor reads. Recorded as W-12 in 04-VERIFICATION.md.

Everything above describes decisions taken and code written. This section is what has not happened. Each item is named individually rather than summarised, because a reader looking for one of them will not find it in a summary.

Nothing is deployed. Measured read-only on 2026-08-02: kubectl -n hl-assets get deploy hl-assets-uploader returned NotFound, and kubectl -n hl-assets get all reported “No resources found”. The cluster is reachable and the namespace exists; there is simply nothing in it.

Version 0.1.0 is not on GHCR. The three publishing jobs are guarded on the event not being a pull request, so the merge to the default branch is what publishes. ArgoCD syncs from the default branch and has synced nothing. This is the landing sequence working as designed, not a defect — but every downstream item below follows from it.

What	State	Why
D-49’s digest pin	not in force	The digest does not exist until the merge publishes. The Deployment carries the image by semver tag only, plus an annotation declaring the unresolved state, and a test asserts the biconditional — the annotation is present if and only if the image carries no digest. Appending the digest without deleting the annotation is red, and so is the reverse. No placeholder digest was fabricated; a fabricated one was applied as a mutation and observed red precisely so it cannot be done later by accident. A human closes this in one commit after the merge.
PORT-03’s docker manifest inspect assertion	has never executed	Written and reviewed, but create-manifest is guarded the same way. Recorded as status: unknown with human_judgment: true, not as a pass. Its evidence file does not exist.
The live upload gate (up-e2e, up-readpath, up-dedup, up-record, up-strip, up-plaintext, up-oversize, up-video415, up-cleanup)	written, never run against a service	There is no endpoint. The gate fails closed saying so rather than skipping — an unconfigured endpoint is a FAIL, because a skip folded into a green run reports on a property nothing evaluated. Every one of these ids was driven red by a permanent test rather than a one-off mutation.
The zero-restart load verdict (load-sc5)	unmeasured, and its reader is unwired	Unmet on two counts. There is no pod to load; and even with one, reading the container restart count needs the cluster CLI, whose name is forbidden outside the one module exempt for it, and that module was out of scope. load-sc5 therefore fails closed saying the counter was never read, which is correct and is not the same as a pass.
The mid-transcode scratch inspection (load-spill)	an explicit abstention	It needs an exec into a running pod, and that surface is forbidden in every module of the harness. Recorded on a note channel that cannot move the exit code: a FAIL would assert a defect nobody observed, a PASS would be a lie, and a SKIP would enter the counted stream as though the property had been evaluated.
Any upload event reaching ClickStack	never observed	The policy permits it and the token is projected, but acceptance needs a running pod and a live collector. Query the log store for hl_assets.upload after the first upload; if it is empty, check the collector for an authentication rejection before suspecting the emitter, which degrades silently by design.
The whole-phase gate transcript	not written	A whole-gate run today fails at preflight on absent credentials, so no assertion is evaluated. Committing that transcript as phase evidence would file a preflight failure where a reader expects a gate result.

Three evidence files are named in the planning record and were deliberately NOT written, for the reason above — a committed file recording a run that did not happen is the precise defect this phase spent nine plans avoiding:

File	Status
manifest-inspect.observed.json	NOT WRITTEN — create-manifest has never run
upload-e2e.observed.txt	NOT WRITTEN — no service to upload to
load-sc5.observed.txt	NOT WRITTEN — no pod to load
phase4-gate.observed.txt	NOT WRITTEN — preflight failed, so no assertion was evaluated

AMENDED 2026-08-04. The four rows above are preserved verbatim; their STATUS is unchanged but every REASON is superseded. Stated precisely, because the distinction is the whole point of this record: all four files are still absent from the repository — git ls-files matches none of them — but none of the four reasons still holds, and each measurement they were waiting on has since been taken and recorded elsewhere.

File	Reason superseded by
manifest-inspect.observed.json	create-manifest has run. The assertion’s output is in run 30903285558 and is retained as the CI run artifact manifest-inspect-observed — a build artifact, not a committed repo file
upload-e2e.observed.txt	There is a service to upload to. The upload was driven and its event recorded in otlp-resource-version.observed.txt; the gate results are in 04-UAT.md
load-sc5.observed.txt	There is a pod to load. Passed at concurrency 32 with restartCount unchanged at 0, and shown non-vacuous by failing at concurrency 2 — recorded in 04-UAT.md test 10
phase4-gate.observed.txt	Preflight no longer fails on absent credentials; the gate round ran. Results are in 04-UAT.md and adjudicated in 04-VERIFICATION.md round 2

Whether these four should now be written as committed files under their original names is a deliberate open question, not an oversight: the evidence exists, but it lives in 04-UAT.md, in the sibling *.observed.txt files, and in a CI artifact rather than under these paths. Writing a file today whose name promises a transcript that was captured elsewhere would be its own species of the defect this section exists to prevent.

Two behaviours differ from what the decisions imply, and both are open rather than settled:

• AVIF and APNG fall to the attachment tier. Read strictly, D-67 puts AVIF in the reject tier: it is ISO-BMFF, it can carry an Exif item, and this phase can neither strip it nor transcode it — the shipped image carries only the HEVC decoder plugin. Closing the gap needs a new constant in the type enum, which was under whole-set assertion by a plan running in parallel at the time. The gap is open and unclaimed. The residual is bounded: an AVIF is served as a download rather than rendered, which is the same treatment other unknown types already get by design.
• GET /api/v1/upload returns 404 with an Allow: POST header, not 405. The error serializer derives status from a code’s single mapping and there is no code meaning “wrong method”; the three ways to produce one were to add a code to a file outside the owning plan’s scope, to declare a code outside the enum the coverage test iterates (fragmenting the exact mechanism the structured-error guarantee rests on), or to override the status behind the serializer’s back and emit a body whose code disagrees with its status. The shipped response is JSON, one status, one code, and they agree. Adding a method_not_allowed code later is purely additive and belongs to a plan that owns the error enum.

One in-tree stub worth naming: attachment-tier uploads are materialised in memory to derive the key, because the key function takes a byte slice and has no streaming form. Adding one is a change to the frozen key package, which nothing here needs. It is bounded by the 50 MB cap and fully accounted by the reservation.

Evidence register

Every file below exists on disk and was read while writing this record.

File	What it establishes
.planning/milestones/ws-image-host-r2-2026-09-19/milestones/v1.5-phases/04-uploader-service/encoder-crossarch.observed.txt	D-66: both native-architecture measurements, before and after the ICC amendment, with the superseded one retained
.planning/milestones/ws-image-host-r2-2026-09-19/milestones/v1.5-phases/04-uploader-service/r2-conditional-put.observed.txt	The conditional put is create-once and the first body survives; metadata replace works, which is what Phase 6’s takedown needs
.planning/milestones/ws-image-host-r2-2026-09-19/milestones/v1.5-phases/04-uploader-service/COVERAGE.md	All sixteen object-store capabilities decided — three integrated, thirteen opted out with reasons
.planning/milestones/ws-image-host-r2-2026-09-19/milestones/v1.5-phases/02-edge-policy/cache-heic.observed.txt	The cacheable half of the transcode requirement was already delivered, so D-52 rests on renderability alone
.planning/milestones/ws-image-host-r2-2026-09-19/milestones/v1.5-phases/03-credentials-bucket-isolation/iso-matrix.observed.txt	The credential boundary this service’s residual is bounded by

Claims not backed by a file above carry an explicit argument by construction label, or cite a named test and the mutation that was observed driving it red.

Open assumptions — recorded, not resolved

None of these is a protection and none may be relied on as one.

1. Cross-architecture encoder agreement is an empirical result for these library versions, not an upstream guarantee. It was measured for one fixture at one quality with two pinned libraries. It is not a promise about any other input, and the golden-file test on both runners is the only thing that would notice it ceasing to hold.
2. The keep-lists cannot be proven complete. Default-deny means an unanticipated metadata container is dropped by construction, but no test can prove completeness against formats that do not exist yet. Asserted for the vendor blobs the fixtures actually carry, and no further.
3. Whether a HEIC carrying a gain map or an unusual item layout resolves the intended profile is delegated to libheif’s own resolution rather than re-implemented, on the grounds that a wrong profile is worse than no profile. Not independently verified against such a file.
4. The relocation checklist is a prediction. apps/hl-assets-uploader/PORTABILITY.md is checkable today against the manifests and the config loader, which is why it was written in this phase. Whether it is accurate as a prediction of a move nobody has performed is a manual verification and is not gated.

Alternatives Considered

• A module under images/ or tools/, or a separate repository — rejected; see D-47.
• Cross-building both architectures under emulation — rejected. The estate builds natively per architecture, and the native runners are what made D-66’s measurement nearly free.
• A short-SHA image tag instead of semver plus digest — rejected; carries no digest and needs a manual manifest edit per deploy.
• A hand-written stub upload page in Phase 4 — rejected; that is what the Svelte decision replaced.
• Shelling out to an image-conversion binary instead of linking the decoders — rejected: a subprocess in the request path, a temp-file dance against PORT-01, and a large install in the runtime image.
• A hermetic pinned C toolchain with a static musl link — rejected. Its headline feature is cross-compilation, which the native build does not use, and statically linking the decoder dependency tree is a known-fiddly exercise with no precedent in this repo. Dynamic linking is also the licence-clean arrangement for the LGPL decoders.
• Normalising every image to WebP — rejected: generation loss on already-lossy sources, lossy output for lossless screenshots, CPU on every upload.
• A long-edge resize on upload — rejected: it changes the stored bytes and therefore the key.
• Hashing the pre-transcode stripped bytes — rejected. Stable across encoder upgrades, but it breaks the key-describes-the-object property that deduplication is defined on.
• Hashing the raw uploaded bytes — rejected; the same photo exported twice with differing metadata would no longer collapse to one key.
• Hashing decoded pixels (D-66 outcome b) and building the encoder with SIMD disabled (outcome c) — both were pre-agreed and both were conditioned on the architectures diverging. The measured set has size 1, so neither was admissible.
• Emitting the colour profile later, after real uploads exist — foreclosed. The amendment above took the cheap window deliberately; taking it later would have re-keyed every stored HEIC.
• Synthesising an sRGB profile for sources that carry none — rejected; it would make the stored bytes depend on something other than the source.
• A 25 MB or 90 MB size ceiling — rejected; too little headroom, and too thin a margin to the vendor’s HTML refusal respectively.
• A memory-backed scratch volume — rejected; tmpfs is RAM and counts against the pod’s memory limit, so it buys file ergonomics with none of the memory relief.
• A request-count concurrency limit — rejected; at a 50 MB ceiling, request count is a poor proxy for memory.
• A horizontal autoscaler as the answer to burst memory — rejected for this purpose; it reacts over tens of seconds while the burst lasts seconds.
• Verifying the Access JWT in Phase 4 — deferred deliberately. The record shape is final now; only trust changes, so nothing re-keys and no record migrates.
• Refusing uploads that carry no identity — rejected for Phase 4; it would make this phase’s own criteria unverifiable, and Phase 5’s Access layer is what makes an unauthenticated request unable to reach the pod at all.
• Appending to a rolling audit-log object — rejected outright; object storage has no append, so it means read-modify-write, which races and makes the hottest object the most corruptible.
• Writing the audit record first — rejected, and the consequence was made observable rather than argued: a failed public write leaves a record that reports a dedup hit for an object that was never stored.
• Signalling deduplication by status code alone, or by header alone — rejected. The bare-URL response shape has nowhere to carry a body signal, and the failure this guards against is a client silently reading a dedup hit as plain success.
• Stripping location metadata from video instead of refusing it — rejected on scope, not on principle: it means rewriting an ISO-BMFF movie box and can invalidate chunk-offset tables.
• Accepting video under the unknown-type-passthrough rule — rejected; it would publish exactly the data the stripping requirement exists to remove, from the same handler that removes it from photos.

Consequences

Positive:

• The one-way door in this design was measured before it was walked through, on two native architectures in one CI run, with the evidence recorded before the decision was presented.
• The derivation is frozen and the freeze is enforced in two places — a golden-file test on both runners, and a package-manager rule that cannot automerge the two Go dependencies that would move it. The rule also names the two apt packages, but Renovate does not read the ARG-interpolated pin they live behind, so for those the golden-file test is the only detector. The places are independent; the coverage is not uniform.
• The strip is provably lossless: entropy-coded data byte-identical, and an independent tool reporting a zero pixel difference against the source.
• The metadata claim is asserted in both directions — what must be gone and what must remain — across a fixture matrix. That is not belt-and-braces: a do-nothing strip passes the losslessness assertions perfectly, and only the two-sided matrix catches it.
• First-write-wins is enforced by R2 rather than by application logic, so concurrent identical uploads resolve to one record with no check-then-write race.
• Memory is bounded by the quantity that actually causes the failure — decode working set — rather than by request size, which does not bound it at all.
• Video and camera-RAW are a named refusal with a structured body, pinned by set equality in both directions, rather than an unstated gap in a passthrough rule.

Negative:

• The encoder, the quality setting and the choice to key over stored bytes are now unchangeable without re-keying every asset ever issued. A libwebp security update is therefore also a deduplication-breaking event, and there is no version of this where that is routine.
• The frozen baseline was already re-cut once. That was free because the bucket was empty. The window is closed.
• An orphaned public object is undiscoverable, because nothing may list. The emitted event stream is the only ledger, and the query that reads it does not exist yet.
• The digest pin is not in force, so the pod would run today from a tag-only reference.
• Nothing has been verified live. The write path and the read path have still never been exercised as one system.
• AVIF is stored rather than refused, which a strict reading of D-67 does not permit.
• The private bucket now holds credentials, in the form of the per-upload capability token.
• Access headers are recorded without verification for the whole of Phase 4.

Neutral:

• The service is the first first-party application source in this repository outside tools/, and the first cgo build in the estate. Both are precedents later work will copy.
• The canonical test environment for this module is the container image, not a CI runner’s package set. The derived key depends on which decoder library read the file, so a baseline frozen from a runner’s own packages would have been measuring an environment that is not production, from the day it was armed.
• The module cannot be built or type-checked on a macOS host with current Homebrew headers. That is a host artefact of the pinned binding, not a defect, and every verification in this phase was run inside the container.
• /statusz exists as a third endpoint, deliberately not wired to a kubelet probe: it performs an object-store reachability check, and a probe that calls R2 turns an upstream blip into a restart loop.

Sign-off: decision, date (2026-08-02) and deciders (Sean Brandt) are recorded above. Human approval of the pull request that lands this ADR is the sign-off for the frozen derivation, for the one recorded re-baseline, and for the residuals named above.