hl-assets Uploader

Operational guide for the Access-protected upload endpoint at drop.fzymgc.net and the four ways in.

What this is

hl-assets-uploader accepts a file, stores it in the private R2 bucket, and returns a public URL on i.hl-assets.dev. It sits behind Cloudflare Access on drop.fzymgc.net, reached through the cloudflared-main tunnel; there is no public path to it that does not pass an Access policy. There are four ways to upload: the browser drop page (visit https://drop.fzymgc.net and sign in with SSO), an iOS Shortcut on the share sheet, scripts/share.fish from a terminal, and a curl one-liner.

The Access application, its two policies, the four service tokens and the expiry controls are documented once, in Cloudflare Operations. This page does not repeat them — it covers the client side.

Property	Value
Upload endpoint	https://drop.fzymgc.net/api/v1/upload
Public asset host	https://i.hl-assets.dev/f/<key>
Multipart field name	file (fixed server-side)
Credentials	secret/fzymgc-house/cluster/hl-assets/access-token (Vault), 1Password (iOS)
Terminal client	scripts/share.fish

Credentials

Three client classes hold three separate service-token pairs, so revoking one family device’s credential leaves the terminal clients working. All six fields live in one Vault secret at secret/fzymgc-house/cluster/hl-assets/access-token, written by the same Terraform apply that creates the tokens.

Client class	Fields	Where the authoritative copy lives
share.fish	cli_client_id, cli_client_secret	Vault, read at run time
curl one-liner	curl_client_id, curl_client_secret	Vault, read at run time
iOS Shortcut	ios_client_id, ios_client_secret	1Password, pasted into the Shortcut at setup

The iOS arrangement buys custody and rotation bookkeeping, not runtime secrecy. Shortcuts cannot read 1Password at run time, so the pair lives inside the Shortcut body and syncs through iCloud with the secret in it. That was accepted with eyes open (D-73): a Keychain lookup is the only shape where the Shortcut definition is safe to share, and it adds a prompt to a share-sheet flow that should be one tap. Treat the Shortcut body as sensitive — do not screenshot it, do not share it through the Shortcuts gallery.

Rotation is client_secret_version (increment it to mint a new secret) plus previous_client_secret_expires_at, which is the overlap window during which the old secret still works. Move that timestamp into the past to revoke a compromised credential immediately. The terminal clients pick up the new value on their next run at no cost; the iOS pair has to be re-pasted on every family device, which is why the overlap window exists.

The curl one-liner

# Read the curl client's pair from Vault. Nothing is written to disk and neither
# half is ever echoed.
creds=$(vault kv get -format=json secret/fzymgc-house/cluster/hl-assets/access-token)
cf_id=$(jq -r '.data.data.curl_client_id' <<<"$creds")
cf_secret=$(jq -r '.data.data.curl_client_secret' <<<"$creds")

# Upload. Both Access headers reach curl through a config on STDIN (-K -), never
# as -H arguments.
printf 'header = "CF-Access-Client-Id: %s"\nheader = "CF-Access-Client-Secret: %s"\n' \
    "$cf_id" "$cf_secret" \
  | curl -sS -f -K - \
      -H 'Accept: text/plain' \
      -F "file=@$1" \
      https://drop.fzymgc.net/api/v1/upload

Why the config-on-stdin form, when -H "CF-Access-Client-Secret: $cf_secret" is shorter. The short form works, and it leaves no shell-history trace when the secret is never typed on the command line — and it still publishes the secret in curl’s process argument list, where ps and any local process inspection can read it for the lifetime of the request. History hygiene does nothing about that. A one-liner in a runbook is copied verbatim into terminals for years, so documenting the -H form teaches every future reader to put the secret in argv. scripts/share.fish carries the same mechanism for the two Access headers.

printf is safe here and is not an exception to the rule: it is a shell builtin in both bash and fish (type -t printf reports builtin in each), so no process is forked and the expanded secret never becomes anyone’s argv. The bytes go to curl’s stdin and stop there. Do not “simplify” this by writing the config to a temp file — that only trades an argv exposure for an on-disk one.

Two things are fixed server-side and are not negotiable: the request must be multipart/form-data (so -F, not --data), and the part must be named file. Accept: text/plain returns the bare URL plus exactly one newline and nothing else, which is what makes … | pbcopy safe; omit it and you get the JSON envelope (url, key, dedup, message) instead.

share.fish

# One file: the URL is printed and copied to the clipboard.
./scripts/share.fish ~/Desktop/screenshot.png

# Several: one line of output per file, in order.
./scripts/share.fish *.heic

It reads the cli pair from Vault at run time (so vault login first), checks its dependencies before doing anything, and exits non-zero with the HTTP status and the response error code on any failure. It never prints either credential half and never runs curl in verbose mode, because verbose mode prints the request headers.

The state file, and why it exists. The management token for an uploaded object is returned in the X-Token response header on a fresh upload only. It is never reissued — not on a repeat upload of the same bytes, not by any support path — because anyone holding the bytes could otherwise mint a takedown capability for someone else’s object. share.fish therefore appends every token it sees to ~/.local/state/hl-assets/tokens.tsv (mode 600) before printing the URL:

2026-08-14T18:20:11Z  https://i.hl-assets.dev/f/abc123.heic  <management token>

Lose that line and the object can no longer be taken down. A deduplicated upload returns 200 rather than 201 and carries no X-Token, which is not an error — it means those exact bytes were already stored.

The iOS Shortcut

Build it once per device, from these steps. The pair to paste comes from 1Password, not from Vault.

1. Receive — Images and Files from the Share Sheet.
2. Get Contents of URL — https://drop.fzymgc.net/api/v1/upload
   • Method: POST
   • Headers: CF-Access-Client-Id and CF-Access-Client-Secret, values pasted from 1Password
   • Request Body: Form
   • One field, type File, named file, value Shortcut Input
3. Get Dictionary Value — key url.
4. Copy to Clipboard.

Shortcuts sends no Accept header, which the server maps to JSON deliberately, so Get Dictionary Value on url works with no header configuration. The response body is {"url", "key", "dedup", "message"}.

Known residual: the management token is lost on this surface. X-Token is a response header, and the Shortcuts Get Contents of URL action is not known to expose response headers. Objects uploaded from a phone therefore cannot be taken down with a token — a takedown goes through the operator instead. This is accepted rather than worked around: moving the token into the response body would change the shipped response contract and put a takedown capability into a body a family member may screenshot.

Taking a link down

There are three verbs, and choosing between them is the whole job. Read the table before you type anything: getting it wrong means either keeping bytes someone asked you to erase, or destroying an original because someone merely wanted a link to stop working.

Verb	Deletes	Keeps	Writes	Reversible by	Choose it when
unshare	the public object f/<hash><ext>	the retained original orig/<hash>, indefinitely (D-83)	state=unshared on the record; body untouched	republish	someone wants a link to stop working
purge	the public object and the retained original	nothing recoverable — only an audit tombstone	the record body replaced by a tombstone carrying state=purged	nothing. This cannot be undone	the bytes must actually go
republish	nothing	everything	the public object written back from the retained original, then state=live	unshare	undoing an unshare

unshare is the reversible verb precisely because the retained original survives it. A retained original never expires, so a republish works a year later. purge deletes that original, which is what makes it irreversible — and it is why purge carries a confirmation token that the other two do not.

An exec is not a kubectl apply, and this does not cross the repository’s no-direct-apply rule. That rule (CLAUDE.md — “MUST NOT apply kubectl changes directly (ArgoCD manages deployments)”) is about who writes Kubernetes objects. These commands create, patch and delete nothing in the cluster: they run a process inside a container that already exists, and the only things they change are objects in R2. ArgoCD remains the sole writer of every manifest. This is written out rather than implied because a later reader with the rule in mind will otherwise “correct” the runbook.

The three commands

Each runs on the already-deployed pod, which already holds the R2 credential — so the operator handles no secrets at any point.

# `<reason>` is the rest of the line and does not need quoting. `<actor>` and
# `<reason>` are mandatory: the command exits non-zero with a usage line if
# either is missing, before it constructs an R2 client or issues a single call.
kubectl exec -n hl-assets deploy/hl-assets-uploader -- \
  /usr/local/bin/uploader unshare <hash> <actor> <reason>

# `--confirm` is checked at that EXACT position, before the hash. It is not a
# flag that can be moved or spelled differently, and it is deliberately not
# accepted anywhere else in the line: the check runs in the pure argument parse,
# so an unconfirmed purge is refused before any credential is even resolved.
kubectl exec -n hl-assets deploy/hl-assets-uploader -- \
  /usr/local/bin/uploader purge --confirm <hash> <actor> <reason>

# `republish` takes EXACTLY two operands and no reason. A stray third token is
# rejected rather than ignored, so an operator who types a reason learns it was
# not recorded instead of believing it was.
kubectl exec -n hl-assets deploy/hl-assets-uploader -- \
  /usr/local/bin/uploader republish <hash> <actor>

All three are idempotent. A second run reports the state was already reached and issues no write — in particular a second purge does not overwrite the first tombstone, which is what protects the original takedown’s actor, reason and timestamp from being replaced by a retry’s.

republish does not merely flip the state back. It reads the retained original, re-runs the upload path’s own normalisation, re-derives the public key, and refuses unless that key equals the one the record stores. A restore that wrote to a different key would hand back a URL nobody has, and a state flip with no object would hand back a URL that 404s.

The purge sequence, in order

The steps are numbered because a sequence that is not handed to you whole is one that gets half-performed under pressure — and half-performing this one leaves the link working. The binary prints step 3 under the header STEP 3 OF 4, matching the numbering below. That agreement is load-bearing rather than tidy: a screen that numbers the same action differently from the runbook is what makes an operator stop early, which is the failure this whole numbered sequence exists to prevent.

1. Fetch the URL first and confirm 200 with cf-cache-status: HIT. This is not decorative. The Cloudflare purge API returns success for a well-formed request whose URL matches nothing cached, so a purge that silently did nothing is indistinguishable from one that worked unless you know the object was cached to begin with. A lone post-purge 404 cannot tell “the purge worked” from “it was never cached in the first place”.

2. Run the verb, then read back the record’s state to confirm the transition. The command reports <from> -> <to>; that line is the confirmation, not the absence of an error.

3. Run the cache-purge command the verb printed. It arrives with the URL already filled in. Until you run it, Cloudflare keeps serving the deleted object for up to 30 days — the edge TTL. The printed command looks like this:

# $CF_API_TOKEN is the Cloudflare bootstrap token, read from Vault at
# secret/fzymgc-house/infrastructure/cloudflare/bootstrap-token (field: token).
# $CF_ZONE_ID is the hl-assets.dev zone id; read it from Terraform rather than
# pasting it, so there is one copy: tf/cloudflare/r2.tf, local
# hl_assets_dev_zone_id.
# The token reaches curl on STDIN (-K -), never as a -H argument, for the same
# reason the upload one-liner above does it that way.
printf 'header = "Authorization: Bearer %s"\n' "$CF_API_TOKEN" \
  | curl -sS -K - -X POST \
      "https://api.cloudflare.com/client/v4/zones/$CF_ZONE_ID/purge_cache" \
      -H "Content-Type: application/json" \
      --data '{"files":["https://i.hl-assets.dev/f/<hash><ext>"]}'

4. Re-fetch and confirm 404. Confirm the transport succeeded before believing the absence: a curl that cannot resolve the host returns 000, which satisfies any absence-shaped check just as well as a real 404 does.

Record the URL from the first purge run’s output. A second purge cannot re-emit step 3, and that is by design rather than a gap. The public URL ends in the stored object’s extension, the extension survives only in the record’s key field, and the tombstone drops that field rather than keeping a description of content someone asked to have erased. The binary will not guess an extension — a guessed URL purges some other object, or nothing.

Why the purge token is not on the pod, and will not be. Putting a Cloudflare Cache-Purge credential in the uploader’s environment would give a zone credential to the one workload that by design feeds attacker-controlled bytes into C decoders (libheif, libde265, libwebp), and it would re-open the environment set-equality gate (PORT-02) in both directions for a value only an operator ever uses. The Cache Purge grant already exists on the bootstrap token; it was added in Phase 2 explicitly for this (see Cloudflare Operations).

republish prints no step 3, deliberately. A cache purge removes a cached copy; a restore needs none. What it does say is the thing an operator would otherwise get wrong: the edge may still be serving a short-lived 404 for that URL, so a link that is not instantly warm is not a failed restore.

An exec is not a kubectl apply, and this does not cross the repository’s no-direct-apply rule. That rule (CLAUDE.md — “MUST NOT apply kubectl changes directly (ArgoCD manages deployments)”) is about who writes Kubernetes objects. This command creates, patches and deletes nothing in the cluster: it runs a process inside a container that already exists, and the only things it changes are objects in R2. ArgoCD remains the sole writer of every manifest. This is written out rather than implied because a later reader with the rule in mind will otherwise “correct” the runbook.

The call order, and why it is what it is

Each verb deletes or writes the object before it writes the record’s state. If a command is interrupted between the two, unshare and purge leave a record still saying live while the object is gone — the link 404s, which is visible, and republish repairs it. The reverse order would leave a record saying unshared while R2 and the edge still serve the bytes, which nothing in this system detects, because nothing lists the bucket. republish inverts it for the same reason: it writes the public object first, then sets the state back to live.

Who is allowed to do this

Anyone who can kubectl exec into the hl-assets namespace can run all three verbs. The control is Kubernetes RBAC on pods/exec, and nothing else. The binary performs no authentication of its own: it does not check the actor argument against anything, and <actor> is a label you type, not an identity it verifies. Do not read the argument as an authentication step, and do not add one to the binary — an in-binary check is theatre against someone who already holds exec.

What is durably recorded, and what is not

<actor> and <reason> are mandatory arguments — the command exits non-zero with a usage line if either is missing, before it constructs an R2 client or issues a single call — and they are printed to stdout. They are expected to be captured in the change record from there.

They are not written into records/<hash>.json. That object is a compatibility surface from the moment the first record lands and it has no migration path, so a field added today would exist on takedowns from today onward and on nothing earlier. purge is different: it replaces the record body with a tombstone, and that tombstone does carry purged_by, purge_reason and purged_at (D-82).

So the audit is asymmetric — unshare’s and republish’s “who and why” live in the change record, purge’s lives in the object. That is an accepted residual, stated here rather than left to be discovered.

What a purge keeps, and what it drops. The record survives at its own key carrying state=purged; only its contents are erased. Deleting the key instead would make the upload path’s existence probe report the hash absent, so identical bytes would upload cleanly and resurrect the identical URL — which is exactly the failure content-addressable keys make possible and the tombstone exists to prevent. The tombstone keeps the schema version, the hash, the original upload time, the recorded identity and its source, and the three purge audit fields. It drops the management token, the object key, the extension, the size, the original filename and both content types — the description of the content someone asked to have erased.

Known residual: the upload-path override does not restore anything

An upload of already-tombstoned bytes can carry an override flag with a reason, and three older artifacts describe that override as re-publishing from the retained original. It does not. The override branch returns a 200 carrying the public URL with the deduplication marker set, and performs no write at all. So after an unshare, the override hands the caller a URL that 404s.

Treat the override as an audit event, not as a restore. republish is the restore path, and it is the only one. This is a live defect being documented rather than a gap being filled; it is recorded here so the next reader who finds the older wording is not misled by it.

Rehearsing the three verbs

Rehearsal status: REHEARSED, 2026-08-16.

This procedure has been performed end to end against the live estate, on a disposable object uploaded for the purpose, and all three verbs touched a real bucket. The transcripts are 06-rehearsal-unshare-republish.observed.txt and 06-rehearsal-purge.observed.txt, both in .planning/milestones/ws-image-host-r2-2026-09-19/milestones/v1.5-phases/06-operations-documentation/. Read the stages below as the procedure to follow; read those two files for what happened when it was followed.

What made the run possible, since it was outstanding for most of this phase: 0.2.0 was cut and merged, CI published ghcr.io/fzymgc-house/hl-assets-uploader:0.2.0 as a multi-arch OCI index, the Deployment was re-pinned to that index digest and merged, and ArgoCD synced it. The running pod’s imageID then resolved to sha256:0bd2dc3e41b69fc9e789a2750d39a7e184c253caa155f150d236fea7fd34f86c — the pinned digest — and the binary inside answered -version with 0.2.0. Before that the pod did not carry the verbs at all and kubectl exec … uploader unshare fell through to the server path, which is why no earlier plan could run this. If you change this status line, a test refuses to let it drift from those two files in either direction — see “How the claim is kept honest”.

Rehearse on a disposable object you upload yourself. Never on a real asset. Upload a throwaway image through the normal client path so a genuine record, retained original and public object all exist, and record the URL and hash the upload prints. The purge arm destroys the object it names, so the object it names must be one nobody wants.

Stage	What to do
0	Upload the throwaway; record its hash and URL from the upload’s own output
1	Fetch the URL; record status and headers
2	unshare the hash; record stdout verbatim, including the emitted purge command with the token placeholder unexpanded
3	Read back the record’s state; confirm unshared
4	Fetch the URL; confirm it no longer serves — assert the transport succeeded first
5	republish the hash; record stdout
6	Read back the state; confirm live
7	Fetch the URL and confirm it serves the same bytes as stage 1 — compare a digest of the body, not just the status
8	Run each verb a second time; confirm exit 0, no transition, same output
9	Settle the disposable object’s identity with a read-only GetObject/HeadObject, recorded in the transcript
10	purge --confirm the disposable hash, then follow “The purge sequence, in order” above

Stage 7 is the one that earns the word “reversible”. A state read-back alone would pass for a republish that flipped the state and wrote no object — the exact shape of the known upload-path override defect documented above. Comparing the bytes is what makes “an unshare is reversible” a checked property rather than a promise.

Stage 9 is not optional, and it is not about suspicion. The rule is settle the identity of every key a destructive verb names, with a read-only read, immediately before it names it — not “check the key that looks doubtful”. It is cheap, it is the only step that cannot be recovered after the fact, and it is the step that caught a misattributed key in this very project: a planning artifact named an object as a disposable forged probe when it was in fact a frozen display baseline that four completed evidence artifacts cite. One GetObject disproved it. Apply the rule to the throwaway too, however certain you are of its key.

Stages 1, 3 and 10’s confirmations can be run as one gate, which records them in the gate’s own output rather than as a hand-assembled narrative:

# Run this AFTER the purge verb and BEFORE the cache purge — that window is the
# measurement, not an inconvenience. The object is already gone from R2 while the
# edge is still serving it from cache, which is what makes the later 404 mean
# something. --rehearsal-url is not derivable from the hash: the extension lives
# only in the record's `key`, and the tombstone drops that field, so take the URL
# from the purge verb's own first-run output.
#
# --rehearsal-state names the transition the verb you just ran makes: `purged` after
# a `purge`, `unshared` after an `unshare`. There is no default — the harness cannot
# observe which verb you ran, and a default would make one of the two rehearsals red
# on a correct run.
uv run python tools/hl-assets-verify/verify.py \
  --phase operations --only op-rehearsal \
  --rehearsal-hash <hash> --rehearsal-url <url> \
  --rehearsal-state purged

It emits four verdicts in order — op-prepurge-hit, op-state-readback, op-purge-response, op-postpurge-absent — and emits all four even when an earlier one fails, so the transcript shows which stage broke. Two behaviours worth knowing before you read its output:

• A cf-cache-status: MISS on the pre-purge fetch is a refusal, not a warning. Warm the object and re-fetch. Proceeding from a MISS produces a transcript that looks like evidence and is not.
• If the post-purge poll flakes, raise its attempt count. Never relax what it asserts. The poll breaks only on a real 404; a curl that cannot resolve returns 000, which would satisfy any absence-shaped check.

Expected purge behaviours — these are correct, not bugs. The scrubbed record survives at its own key carrying state=purged; the tombstone keeps the eight audit fields and drops the seven content-describing ones; and a second purge deliberately emits no URL and no curl line, because the extension lives only in the record’s key, which the tombstone drops.

unshare and republish on an already-purged hash both refuse by name, exiting non-zero with takedown: that hash was purged; …. That is the tombstone answering, not a corrupted record — a purge erased the content-describing fields, so there is nothing left to unshare and nothing to restore. Do not read it as a fault and do not go looking for the record: it is exactly where it was, saying what it should.

How the claim is kept honest

The status line above is not prose anyone can edit freely. A test (test_the_runbook_claims_a_rehearsal_only_when_both_transcripts_exist) reads it and asserts a biconditional against the two transcript files in the phase directory: claiming REHEARSED with either transcript missing fails, and leaving NOT YET REHEARSED once both transcripts exist fails too. Neither side can drift silently — a rehearsal that never ran cannot be documented as done, and a rehearsal that did run cannot leave the runbook telling an operator it is still outstanding.

The takedown window

Delete plus purge still does not reach every copy. The edge/browser TTL split, and the residual that nothing an operator does can reach a browser holding the bytes, are documented once in Cloudflare Operations → Takedown window. Read it before telling anyone the link is gone.

Monitoring

Two Uptime Kuma monitors watch this service, both routed to the cluster-edge Pushover app — the estate partitions Pushover apps by the alert speaker, and Uptime Kuma is the external prober.

Monitor	Target	Type	Cadence	What its failure means
hl_assets_canary	https://i.hl-assets.dev/_health/canary	keyword	60 s, 3 retries	the read path is broken — R2 unreachable, or answering 200 with the wrong bytes
hl_assets_uploader	https://drop.fzymgc.net/readyz	HTTP status	60 s, 3 retries	the write path is broken — the tunnel, the pod, or Access

The canary is a keyword monitor rather than a status check on purpose. The R2 custom domain is the component most likely to fail in a way that still answers 200: rebound to the wrong bucket, or fronted by a rule serving something else, the response looks perfectly healthy. A status-only check reads that as UP forever.

Three caveats, stated here rather than left to be discovered:

• Uptime Kuma checks response bodies, not response headers. If a caching change started serving the canary URL from the edge, the body would keep matching and this monitor would report UP straight through a total R2 outage. The only thing that prevents it is the edge rule’s /f/ path predicate, which keeps _health/ uncached, and no check guards that predicate: review any change to the cache rule against the canary URL.
• The canary reaches R2 on every poll, which is what makes it able to detect that R2 is unreachable. At a 60-second interval that is roughly 43,800 Class B operations a month in a perfectly healthy system. That number is a floor, not an anomaly — do not read it as one, and do not set a Class B threshold below it. It is not the whole floor. The uploader pod runs its own background reachability prober: one HeadObject against the private bucket per STATUS_PROBE_INTERVAL per replica, defaulting to 60s and not overridden by the Deployment. That is a second ~43,800 a month, and billing_usage_alert has no bucket predicate, so both count against the same limit. The real steady-state floor is 43,800 × (1 + replicas) — ~87,600 at replicas = 1. The prober’s key can never collide with a real record so it always 404s, and R2 bills that: the only operation Cloudflare documents as not charged is one answered 401 Unauthorized. See R2 usage alerting for the margin rule, which matters before scaling this Deployment — at four replicas the healthy floor is 219,000 and the 200,000 anomaly limit would fire every month on its own traffic.
• /readyz performs no network I/O. A 200 there means config parsed, salt present, spill directory writable and store constructed. It does not mean R2 is up. R2 reachability is the canary’s job, and that is why there are two monitors rather than one.

The uploader monitor authenticates with its own Access service token — a fourth client class on its own Vault path, so a scheduled health check never holds an uploader credential. An unauthenticated probe asserting the Access challenge shape would prove DNS and Access configuration and nothing else, because Access rejects at the edge before the tunnel is reached; that probe would stay green through a total outage of the tunnel, the pod and the store.

Accepted residual — the monitor’s Access client secret is not redacted. The Uptime Kuma provider does not mark its headers attribute as sensitive (measured, with a known-sensitive attribute as a control), so the secret is rendered unredacted in that workspace’s plan output and stored unredacted in its state. This is written as accepted, not mitigated: Cloudflare Access authenticates on request headers specifically and does not accept Basic authentication, and the provider ships no write-only variant. What bounds it is that the exposed credential is this monitor’s own client class and nothing else’s.

Volume alerting. 4 R2 usage notification policies watch R2 operation volume — the cross product of anomaly and cost-ceiling thresholds against Class A and Class B operations. The thresholds and the arithmetic behind them are documented once, in Cloudflare Operations → R2 usage alerting.

Standing hazard. Destroying and recreating the monitor’s service token rotates its credential, and the monitor reads DOWN until the uptime-kuma workspace re-applies — a false outage on a canary. Re-apply the consuming workspace in the same change.

Tracked operational items

Three facts that fail silently and are each fatal in their own way. Each is measured on a schedule rather than remembered.

Item	How it is measured	Warning window	Check id	Current known value
hl-assets.dev registration	RDAP expiration event, unauthenticated	90 days	op-domain-expiry	expires 2027-07-29
Cloudflare bootstrap token	account-owned token introspection	90 days	op-token-expiry	expires 2027-07-03
CAS salt (cluster/hl-assets/salt)	Vault KV metadata: current_version against a pinned 1, and created_time against a pinned timestamp	none — any change is a failure	op-salt-version	version 1 and creation time both unchanged

The first two dates are three weeks apart and older documentation conflates them. They are two different secrets with two different consequences: the domain lapsing kills every link ever shared, and the bootstrap token lapsing breaks every tf/cloudflare apply for every zone in the account. They are carried as separate constants in the harness and as separate rows here for that reason. If you find a single item claiming to track “the expiry”, it is wrong.

The domain check asserts the expiry moving forward, not an auto-renew setting. There is no auto-renew boolean to read — neither RDAP nor the Cloudflare Registrar API exposes one. Asserting the outcome is the stronger claim anyway: a setting reading on is still green on a domain three days from lapsing, whereas a date that has stopped moving is the failure itself.

The salt check reads metadata only, and cannot read the salt. That is a property of the credential rather than of the script: the policy grants the metadata path and deliberately omits the paired data path, so the read that would return the salt’s bytes cannot succeed for the identity the check runs as. The never-rotate rule was already written in five places; what was missing was any way to notice a rotation that happened anyway, and a rotation does not error — it silently stops deduplication.

Who runs it. The hl-assets operational items workflow (.github/workflows/hl-assets-operational-items.yml) runs weekly, Mondays 06:17 UTC, on the in-cluster runner, and can be triggered on demand with workflow_dispatch. It authenticates to Vault as the runner pod and posts failures to the cluster-infra Pushover app.

There are two signals, not one. The first is the page when a check trips. The second is the run history itself — a checker that stops firing silently would move the problem rather than solve it, so an absent weekly run is itself the finding. Check the workflow’s run history if you have not seen it report in over a week.

The token-expiry check uses a dedicated reader, not the bootstrap credential. tf/cloudflare/token_expiry_monitor.tf follows the account-token-to-Vault pattern in nodes_email.tf: it creates hl-assets-token-expiry-monitor with only Account API Tokens Read and writes its credential, account ID, and explicit bootstrap target ID to secret/fzymgc-house/cluster/hl-assets/token-expiry-monitor. This permission reads account-wide token metadata; it does not grant token mutation, DNS, or R2 access. The runner gets read-only access to this exact Vault path and its metadata, never to the bootstrap credential.

The harness fetches /accounts/{account_id}/tokens/{target_token_id} and checks the returned ID, active status, and expiry. It does not use /tokens/verify, which would measure the reader instead. Missing credentials, denied reads, an inactive target, and malformed responses remain op-token-expiry failures. No Cloudflare repository secret is supplied to this workflow.

Deployment and rotation. Both main-cluster-vault (reader grants) and main-cluster-cloudflare (token and Vault object) must apply through the normal VCS workflow before the scheduled check can pass. Their applies can run independently: the Cloudflare writer already has the paired cluster/hl-assets/* grants. Dispatch hl-assets-operational-items.yml after both applies and confirm all three measurement rows pass.

The monitoring credential has no automatic expiry, following the existing workload-token precedent. Revoke or replace it through Terraform when needed. When replacing the bootstrap token, update the non-secret cloudflare_bootstrap_token_id in the same Terraform change; an in-place value roll keeps the ID. The reader then monitors the replacement without receiving its credential. Do not seed the bootstrap value into GitHub to repair a failed check: the old repository credential produced HTTP 401 on 2026-09-07, while the current Vault-held bootstrap verified active. That was the reason for introducing this dedicated reader.

Troubleshooting

An upload returns a redirect to fzymgc.cloudflareaccess.com instead of JSON. This is the one genuinely confusing failure, because two different causes produce the same wire response: a service token that has expired, and a service token that was never in the policy. Both get a 302 to the Access login origin with an HTML body. You cannot tell them apart from the response, so do not guess — read the token’s expires_at from the Cloudflare API (GET /accounts/{account_id}/access/service_tokens).

If the expiry is in the future, the token is not in the policy: check the non_identity policy in tf/cloudflare/access.tf, which names its four token ids explicitly — the three client classes plus the uptime monitor — rather than admitting every service token in the account. If the client is the uptime monitor, check the standing hazard above first: its client_secret is readable only at creation, so a destroy-and-recreate rotates it silently (secrets reference). See Service-token expiry for the two controls and their lead times.

share.fish says it cannot read the Vault path. Run vault login. The script checks vault token lookup before it tries, so an authentication problem is reported as one rather than as a failed upload.

An upload returns 413. The file is over the service’s size limit. The limit is enforced at the origin, not at the edge.

The URL was printed but the clipboard is empty. The clipboard copy is deliberately not allowed to fail the upload: if neither pbcopy nor wl-copy is on PATH, the URL still goes to stdout and the exit code still reflects the upload.