hl-assets Uploader
Operational guide for the Access-protected upload endpoint at drop.fzymgc.net and the four
ways in.
What this is
Section titled “What this is”hl-assets-uploader accepts a file, stores it in the private R2 bucket, and returns a public
URL on i.hl-assets.dev. It sits behind Cloudflare Access on drop.fzymgc.net, reached through
the cloudflared-main tunnel. There is no public path to it that does not pass an Access policy.
There are four ways to upload: the browser drop page (visit https://drop.fzymgc.net and sign in
with SSO), an iOS Shortcut on the share sheet, scripts/share.fish from a terminal, and a curl
one-liner.
The Access application, its policies, the service tokens and the expiry controls are documented once, in Cloudflare Operations. This page covers the client side.
| Property | Value |
|---|---|
| Upload endpoint | https://drop.fzymgc.net/api/v1/upload |
| Public asset host | https://i.hl-assets.dev/f/<key> |
| Multipart field name | file (fixed server-side) |
| Size limit | MAX_UPLOAD_BYTES in argocd/app-configs/hl-assets/deployment.yaml, enforced at the origin |
| Credentials | secret/fzymgc-house/cluster/hl-assets/access-token (Vault), 1Password (iOS) |
| Terminal client | scripts/share.fish |
| Source | apps/hl-assets-uploader/ |
| Manifests | argocd/app-configs/hl-assets/ |
Credentials
Section titled “Credentials”Three client classes hold three separate service-token pairs, so revoking one family device’s
credential leaves the terminal clients working. All six fields live in one Vault secret at
secret/fzymgc-house/cluster/hl-assets/access-token, written by the same Terraform apply that
creates the tokens (tf/cloudflare/access.tf). The uptime monitor holds a fourth pair at
secret/fzymgc-house/cluster/hl-assets/monitor-token.
| Client class | Fields | Where the authoritative copy lives |
|---|---|---|
share.fish |
cli_client_id, cli_client_secret |
Vault, read at run time |
curl one-liner |
curl_client_id, curl_client_secret |
Vault, read at run time |
| iOS Shortcut | ios_client_id, ios_client_secret |
1Password, pasted into the Shortcut at setup |
The iOS arrangement buys custody and rotation bookkeeping, not runtime secrecy. Shortcuts cannot read 1Password at run time, so the pair lives inside the Shortcut body and syncs through iCloud with the secret in it. A Keychain lookup is the only shape where the Shortcut definition is safe to share, and it adds a prompt to a share-sheet flow that must be one tap. Treat the Shortcut body as sensitive. Do not screenshot it, and do not share it through the Shortcuts gallery.
Rotation is client_secret_version (increment it to mint a new secret) plus
previous_client_secret_expires_at, the overlap window during which the old secret still works.
To revoke a compromised credential at once, move that timestamp into the past. The terminal
clients pick up the new value on their next run. The iOS pair has to be re-pasted on every family
device, which is why the overlap window exists.
The curl one-liner
Section titled “The curl one-liner”# Read the curl client's pair from Vault. Nothing is written to disk and neither# half is ever echoed.creds=$(vault kv get -format=json secret/fzymgc-house/cluster/hl-assets/access-token)cf_id=$(jq -r '.data.data.curl_client_id' <<<"$creds")cf_secret=$(jq -r '.data.data.curl_client_secret' <<<"$creds")
# Upload. Both Access headers reach curl through a config on STDIN (-K -), never# as -H arguments.printf 'header = "CF-Access-Client-Id: %s"\nheader = "CF-Access-Client-Secret: %s"\n' \ "$cf_id" "$cf_secret" \ | curl -sS -f -K - \ -H 'Accept: text/plain' \ -F "file=@$1" \ https://drop.fzymgc.net/api/v1/uploadThe config-on-stdin form exists because a -H "CF-Access-Client-Secret: $cf_secret" argument
publishes the secret in curl’s process argument list, where ps and any local process
inspection can read it for the lifetime of the request. Shell-history hygiene does nothing about
that. printf is a shell builtin in both bash and fish, so no process is forked and the
expanded secret never becomes anyone’s argv. Do not write the config to a temp file instead.
That trades an argv exposure for an on-disk one. scripts/share.fish carries the same
mechanism.
Two things are fixed server-side. The request must be multipart/form-data (so -F, not
--data), and the part must be named file. Accept: text/plain returns the bare URL plus one
newline and nothing else, which is what makes … | pbcopy safe. Without it the response is the
JSON envelope (url, key, dedup, message).
share.fish
Section titled “share.fish”# One file: the URL is printed and copied to the clipboard../scripts/share.fish ~/Desktop/screenshot.png
# Several: one line of output per file, in order../scripts/share.fish *.heicIt reads the cli pair from Vault at run time (so vault login first), checks its dependencies
before doing anything, and exits non-zero with the HTTP status and the response error code on any
failure. It never prints either credential half and never runs curl in verbose mode, because
verbose mode prints the request headers.
The management token for an uploaded object is returned in the X-Token response header on a
fresh upload only. It is never reissued, because anyone holding the bytes would otherwise mint a
takedown capability for someone else’s object. share.fish therefore appends every token it
sees to ~/.local/state/hl-assets/tokens.tsv (mode 600) before printing the URL:
<upload time> https://i.hl-assets.dev/f/abc123.heic <management token>Lose that line and the object can no longer be taken down with a token. A deduplicated upload
returns 200 rather than 201 and carries no X-Token. That is not an error. It means those
exact bytes were already stored.
The iOS Shortcut
Section titled “The iOS Shortcut”Build it once per device, from these steps. The pair to paste comes from 1Password, not from Vault.
- Receive: Images and Files from the Share Sheet.
- Get Contents of URL:
https://drop.fzymgc.net/api/v1/upload- Method:
POST - Headers:
CF-Access-Client-IdandCF-Access-Client-Secret, values pasted from 1Password - Request Body:
Form - One field, type File, named
file, value Shortcut Input
- Method:
- Get Dictionary Value: key
url. - Copy to Clipboard.
Shortcuts sends no Accept header, which the server maps to JSON, so Get Dictionary Value on
url works with no header configuration. The response body is {"url", "key", "dedup", "message"}.
The management token is lost on this surface. X-Token is a response header, and the Shortcuts
Get Contents of URL action does not expose response headers. An object uploaded from a phone
is taken down by the operator, with the commands below. Moving the token into the response body
would change the response contract and put a takedown capability into a body a family member can
screenshot, so the token stays in the header.
Taking a link down
Section titled “Taking a link down”There are three verbs, and choosing between them is the whole job. Read the table before you type anything. Getting it wrong means either keeping bytes someone asked you to erase, or destroying an original because someone wanted a link to stop working.
| Verb | Deletes | Keeps | Writes | Reversible by | Choose it when |
|---|---|---|---|---|---|
unshare |
the public object f/<hash><ext> |
the retained original orig/<hash>, indefinitely |
state=unshared on the record; body untouched |
republish |
someone wants a link to stop working |
purge |
the public object and the retained original | nothing recoverable, only an audit tombstone | the record body replaced by a tombstone carrying state=purged |
nothing. This cannot be undone | the bytes must go |
republish |
nothing | everything | the public object written back from the retained original, then state=live |
unshare |
undoing an unshare |
unshare is reversible because the retained original survives it. A retained original never
expires, so a republish works a year later. purge deletes that original, which is what makes
it irreversible, and it is why purge carries a confirmation token that the other two do not.
An exec is not a kubectl apply, and it does not cross the repository’s no-direct-apply rule.
That rule is about who writes Kubernetes objects. These commands create, patch and delete nothing
in the cluster. They run a process inside a container that already exists, and the only things
they change are objects in R2. ArgoCD stays the sole writer of every manifest.
The three commands
Section titled “The three commands”Each runs on the deployed pod, which already holds the R2 credential, so the operator handles no secrets at any point.
# `<reason>` is the rest of the line and does not need quoting. `<actor>` and# `<reason>` are mandatory: the command exits non-zero with a usage line if# either is missing, before it constructs an R2 client or issues a single call.kubectl exec -n hl-assets deploy/hl-assets-uploader -- \ /usr/local/bin/uploader unshare <hash> <actor> <reason># `--confirm` is checked at that EXACT position, before the hash. It is not a# flag that can be moved or spelled differently, and it is deliberately not# accepted anywhere else in the line: the check runs in the pure argument parse,# so an unconfirmed purge is refused before any credential is even resolved.kubectl exec -n hl-assets deploy/hl-assets-uploader -- \ /usr/local/bin/uploader purge --confirm <hash> <actor> <reason># `republish` takes EXACTLY two operands and no reason. A stray third token is# rejected rather than ignored, so an operator who types a reason learns it was# not recorded instead of believing it was.kubectl exec -n hl-assets deploy/hl-assets-uploader -- \ /usr/local/bin/uploader republish <hash> <actor>All three are idempotent. A second run reports that the state was already reached and issues no
write. A second purge does not overwrite the first tombstone, which protects the original
takedown’s actor, reason and timestamp from a retry.
republish does not only flip the state back. It reads the retained original, re-runs the upload
path’s own normalisation, re-derives the public key, and refuses unless that key equals the one
the record stores. A restore that wrote to a different key would hand back a URL nobody has, and
a state flip with no object would hand back a URL that 404s.
The purge sequence, in order
Section titled “The purge sequence, in order”The binary prints step 3 under the header STEP 3 OF 4, matching the numbering below. A screen
that numbers the same action differently from the runbook is what makes an operator stop early.
-
Fetch the URL first and make sure that it answers
200withcf-cache-status: HIT. The Cloudflare purge API returns success for a well-formed request whose URL matches nothing cached, so a purge that did nothing looks the same as one that worked unless you know the object was cached to begin with. A lone post-purge404cannot tell “the purge worked” from “it was never cached”. -
Run the verb, then read back the record’s
state. The command reports<from> -> <to>. That line is the confirmation, not the absence of an error. -
Run the cache-purge command the verb printed. It arrives with the URL filled in. Until you run it, Cloudflare keeps serving the deleted object for up to 30 days, the edge TTL. The printed command looks like this:
Terminal window # $CF_API_TOKEN is the Cloudflare bootstrap token, read from Vault at# secret/fzymgc-house/infrastructure/cloudflare/bootstrap-token (field: token).# $CF_ZONE_ID is the hl-assets.dev zone id; read it from Terraform rather than# pasting it, so there is one copy: tf/cloudflare/r2.tf, local# hl_assets_dev_zone_id.# The token reaches curl on STDIN (-K -), never as a -H argument, for the same# reason the upload one-liner above does it that way.printf 'header = "Authorization: Bearer %s"\n' "$CF_API_TOKEN" \| curl -sS -K - -X POST \"https://api.cloudflare.com/client/v4/zones/$CF_ZONE_ID/purge_cache" \-H "Content-Type: application/json" \--data '{"files":["https://i.hl-assets.dev/f/<hash><ext>"]}' -
Re-fetch and make sure that the answer is
404. Make sure that the transport succeeded before believing the absence. Acurlthat cannot resolve the host returns000, which satisfies any absence-shaped check as well as a real404does.
Record the URL from the first purge run’s output. A second purge cannot re-emit step 3. The
public URL ends in the stored object’s extension, the extension survives only in the record’s
key field, and the tombstone drops that field rather than keeping a description of content
someone asked to have erased. The binary will not guess an extension. A guessed URL purges some
other object, or nothing.
The purge token is not on the pod. Putting a Cloudflare cache-purge credential in the uploader’s
environment would give a zone credential to the one workload that by design feeds
attacker-controlled bytes into C decoders (libheif, libde265, libwebp). The Cache Purge grant
sits on the bootstrap token instead (see
Cloudflare Operations).
republish prints no step 3. A cache purge removes a cached copy, and a restore needs none. The
edge can still serve a short-lived 404 for that URL, so a link that is not instantly warm is
not a failed restore.
The call order, and why it is what it is
Section titled “The call order, and why it is what it is”Each verb deletes or writes the object before it writes the record’s state. If a command is
interrupted between the two, unshare and purge leave a record still saying live while the
object is gone. The link 404s, which is visible, and republish repairs it. The reverse order
would leave a record saying unshared while R2 and the edge still serve the bytes, which nothing
in this system detects, because nothing lists the bucket. republish inverts the order for the
same reason. It writes the public object first, then sets the state back to live.
Who is allowed to do this
Section titled “Who is allowed to do this”Anyone who can kubectl exec into the hl-assets namespace can run all three verbs. The control
is Kubernetes RBAC on pods/exec, and nothing else. The binary performs no authentication of its
own. <actor> is a label you type, not an identity it verifies. Do not add a check to the
binary. An in-binary check is theatre against someone who already holds exec.
What is durably recorded, and what is not
Section titled “What is durably recorded, and what is not”<actor> and <reason> are mandatory arguments and are printed to stdout. Capture them in the
change record from there. They are not written into records/<hash>.json. That object is a
compatibility surface with no migration path, so a field added now would exist on takedowns from
now on and on nothing earlier. purge is different. It replaces the record body with a tombstone,
and that tombstone carries purged_by, purge_reason and purged_at.
The audit is therefore asymmetric. The “who and why” of unshare and republish lives in the
change record, and the “who and why” of purge lives in the object.
A purge keeps the record at its own key, carrying state=purged, and erases only its contents.
Deleting the key instead would make the upload path’s existence probe report the hash absent, so
identical bytes would upload cleanly and resurrect the identical URL. The tombstone keeps the
schema version, the hash, the original upload time, the recorded identity and its source, and the
three purge audit fields. It drops the management token, the object key, the extension, the size,
the original filename and both content types.
The upload-path override does not restore anything
Section titled “The upload-path override does not restore anything”An upload of already-tombstoned bytes can carry an override flag with a reason. The override
branch returns a 200 carrying the public URL with the deduplication marker set, and performs no
write. After an unshare, the override hands the caller a URL that 404s. Treat the override as
an audit event. republish is the restore path, and it is the only one.
Rehearsing the three verbs
Section titled “Rehearsing the three verbs”Rehearse the verbs against a disposable object before trusting a new image build, and never on a real asset. Upload a throwaway image through the normal client path so a genuine record, retained original and public object all exist, and record the URL and hash the upload prints. The purge arm destroys the object it names.
| Stage | What to do |
|---|---|
| 0 | Upload the throwaway; record its hash and URL from the upload’s own output |
| 1 | Fetch the URL; record status and headers |
| 2 | unshare the hash; record stdout verbatim, including the emitted purge command with the token placeholder unexpanded |
| 3 | Read back the record’s state; make sure that it is unshared |
| 4 | Fetch the URL; make sure that it no longer serves, and that the transport succeeded |
| 5 | republish the hash; record stdout |
| 6 | Read back the state; make sure that it is live |
| 7 | Fetch the URL and compare a digest of the body with stage 1, not only the status |
| 8 | Run each verb a second time; make sure of exit 0, no transition, same output |
| 9 | Settle the disposable object’s identity with a read-only GetObject or HeadObject |
| 10 | purge --confirm the disposable hash, then follow The purge sequence, in order above |
Stage 7 is the one that earns the word “reversible”. A state read-back alone would pass for a
republish that flipped the state and wrote no object. Stage 9 is not optional. Settle the
identity of every key a destructive verb names, with a read-only read, immediately before it
names it, however certain you are of the key.
Stages 1, 3 and 10 can run as one gate, which records them in the gate’s own output:
# Run this AFTER the purge verb and BEFORE the cache purge — that window is the# measurement, not an inconvenience. The object is already gone from R2 while the# edge is still serving it from cache, which is what makes the later 404 mean# something. --rehearsal-url is not derivable from the hash: the extension lives# only in the record's `key`, and the tombstone drops that field, so take the URL# from the purge verb's own first-run output.## --rehearsal-state names the transition the verb you just ran makes: `purged` after# a `purge`, `unshared` after an `unshare`. There is no default — the harness cannot# observe which verb you ran, and a default would make one of the two rehearsals red# on a correct run.uv run python tools/hl-assets-verify/verify.py \ --phase operations --only op-rehearsal \ --rehearsal-hash <hash> --rehearsal-url <url> \ --rehearsal-state purgedIt emits four verdicts in order, op-prepurge-hit, op-state-readback, op-purge-response and
op-postpurge-absent, and emits all four even when an earlier one fails. Two behaviours to know
before you read its output:
- A
cf-cache-status: MISSon the pre-purge fetch is a refusal, not a warning. Warm the object and re-fetch. - If the post-purge poll flakes, raise its attempt count. Never relax what it asserts. The poll
breaks only on a real
404, because acurlthat cannot resolve returns000.
Expected purge behaviours, which are correct and not bugs: the scrubbed record survives at its
own key carrying state=purged, and a second purge emits no URL and no curl line, because the
extension lives only in the record’s key, which the tombstone drops. unshare and republish
on an already-purged hash both refuse by name, exiting non-zero with
takedown: that hash was purged; …. That is the tombstone answering, not a corrupted record.
The takedown window
Section titled “The takedown window”Delete plus purge still does not reach every copy. The edge and browser TTL split, and the residual that nothing an operator does can reach a browser holding the bytes, are documented once in Cloudflare Operations, Takedown window. Read it before telling anyone the link is gone.
Monitoring
Section titled “Monitoring”Two Uptime Kuma monitors watch this service (tf/uptime-kuma/monitors-hl-assets.tf), both routed
to the cluster-edge Pushover app. The estate partitions Pushover apps by the alert speaker, and
Uptime Kuma is the external prober.
| Monitor | Target | Type | Cadence | What its failure means |
|---|---|---|---|---|
hl_assets_canary |
https://i.hl-assets.dev/_health/canary |
keyword | 60 s, 3 retries | the read path is broken: R2 unreachable, or answering 200 with the wrong bytes |
hl_assets_uploader |
https://drop.fzymgc.net/readyz |
HTTP status | 60 s, 3 retries | the write path is broken: the tunnel, the pod, or Access |
The canary is a keyword monitor rather than a status check on purpose. The R2 custom domain is
the component most likely to fail in a way that still answers 200. Rebound to the wrong bucket,
or fronted by a rule serving something else, the response looks healthy, and a status-only check
reads that as UP forever.
Three constraints:
- Uptime Kuma checks response bodies, not response headers. If a caching change started serving
the canary URL from the edge, the body would keep matching and this monitor would report UP
through a total R2 outage. The only thing that prevents it is the
/f/path predicate on the cache rule intf/cloudflare/rulesets-hl_assets_dev.tf, which keeps_health/uncached, and no check guards that predicate. Review any change to the cache rule against the canary URL. - The canary reaches R2 on every poll, which is what lets it detect that R2 is unreachable. At a
60-second interval that is about 43,800 Class B operations a month in a healthy system. The
uploader pod runs its own background reachability prober too: one
HeadObjectagainst the private bucket perSTATUS_PROBE_INTERVAL(60 s by default, not overridden by the Deployment) per replica. The prober’s key never collides with a real record, so it always404s, and R2 bills that. The healthy floor is therefore43,800 x (1 + replicas), about 87,600 atreplicas: 1, andbilling_usage_alerthas no bucket predicate, so both count against the same limit. Do not set a Class B threshold below the floor, and re-derive the floor before scaling the Deployment. See R2 usage alerting for the margin rule. /readyzperforms no network I/O. A200there means that the configuration parsed, the salt is present, the spill directory is writable and the store is constructed. It does not mean R2 is up. R2 reachability is the canary’s job, which is why there are two monitors.
The uploader monitor authenticates with its own Access service token, a fourth client class on its own Vault path, so a scheduled health check never holds an uploader credential. An unauthenticated probe asserting the Access challenge shape would prove DNS and Access configuration and nothing else, because Access rejects at the edge before the tunnel is reached.
Accepted residual: the monitor’s Access client secret is not redacted. The Uptime Kuma provider
does not mark its headers attribute as sensitive, so the secret is rendered unredacted in that
workspace’s plan output and stored unredacted in its state. Cloudflare Access authenticates on
request headers and does not accept Basic authentication, and the provider ships no write-only
variant. What bounds it is that the exposed credential is this monitor’s own client class and
nothing else’s.
Volume alerting: four R2 usage notification policies (tf/cloudflare/notifications-hl_assets.tf)
watch R2 operation volume, the cross product of anomaly and cost-ceiling thresholds against Class
A and Class B operations. The thresholds and the arithmetic behind them are documented once, in
Cloudflare Operations, R2 usage alerting.
Standing hazard: destroying and recreating the monitor’s service token rotates its credential, and the monitor reads DOWN until the uptime-kuma workspace re-applies, a false outage on a canary. Re-apply the consuming workspace in the same change.
Tracked operational items
Section titled “Tracked operational items”Three facts fail silently and are each fatal in their own way, so each is checked on a schedule
rather than remembered. The hl-assets operational items workflow
(.github/workflows/hl-assets-operational-items.yml) runs weekly, Mondays 06:17 UTC, on the
in-cluster runner, and can be triggered on demand with workflow_dispatch. It authenticates to
Vault as the runner pod and posts failures to the cluster-infra Pushover app.
| Item | How it is measured | Warning window | Check id |
|---|---|---|---|
hl-assets.dev registration |
RDAP expiration event, unauthenticated |
90 days | op-domain-expiry |
| Cloudflare bootstrap token | account-owned token introspection | 90 days | op-token-expiry |
CAS salt (cluster/hl-assets/salt) |
Vault KV metadata: current_version against a pinned 1, and created_time against a pinned timestamp |
none. Any change is a failure | op-salt-version |
The first two are two different secrets with two different consequences. The domain lapsing
kills every link ever shared. The bootstrap token lapsing breaks every tf/cloudflare apply for
every zone in the account. They are separate constants in the harness and separate rows here.
The domain check asserts the expiry moving forward, not an auto-renew setting. Neither RDAP nor
the Cloudflare Registrar API exposes such a setting, and the outcome is the stronger claim anyway.
A setting reading on is still green on a domain three days from lapsing.
The salt check reads metadata only and cannot read the salt. The policy grants the metadata path and omits the paired data path, so the read that would return the salt’s bytes cannot succeed for the identity the check runs as. A rotation does not error. It silently stops deduplication, which is why the check exists.
There are two signals, not one. The first is the page when a check trips. The second is the run history itself. A checker that stops firing silently would move the problem rather than solve it, so an absent weekly run is itself the finding. Check the workflow’s run history if it has not reported in over a week.
The token-expiry check uses a dedicated reader, not the bootstrap credential.
tf/cloudflare/token_expiry_monitor.tf creates hl-assets-token-expiry-monitor with only
Account API Tokens Read and writes its credential, the account ID and the bootstrap target ID to
secret/fzymgc-house/cluster/hl-assets/token-expiry-monitor. The runner gets read-only access to
that Vault path and its metadata, never to the bootstrap credential. The harness fetches
/accounts/{account_id}/tokens/{target_token_id} and checks the returned ID, active status and
expiry. It does not use /tokens/verify, which would measure the reader instead. Missing
credentials, denied reads, an inactive target and malformed responses are all op-token-expiry
failures. No Cloudflare repository secret is supplied to this workflow, and do not seed the
bootstrap value into GitHub to repair a failed check.
When replacing the bootstrap token, update the non-secret cloudflare_bootstrap_token_id in the
same Terraform change. An in-place value roll keeps the ID. The reader then monitors the
replacement without receiving its credential.
Open work on this service is tracked in the issue tracker:
gh issue list --state open --search hl-assetsTroubleshooting
Section titled “Troubleshooting”An upload returns a redirect to fzymgc.cloudflareaccess.com instead of JSON. Two different
causes produce the same wire response: a service token that has expired, and a service token that
is not in the policy. Both get a 302 to the Access login origin with an HTML body. Read the
token’s expires_at from the Cloudflare API
(GET /accounts/{account_id}/access/service_tokens) rather than guessing.
If the expiry is in the future, the token is not in the policy. Check the non_identity policy in
tf/cloudflare/access.tf, which names its four token ids explicitly, the three client classes
plus the uptime monitor, rather than admitting every service token in the account. If the client
is the uptime monitor, check the standing hazard above first. Its client_secret is readable
only at creation, so a destroy-and-recreate rotates it silently
(secrets reference). See
Service-token expiry
for the two controls and their lead times.
share.fish says it cannot read the Vault path. Run vault login. The script checks
vault token lookup before it tries, so an authentication problem is reported as one rather than
as a failed upload.
An upload returns 413. The file is over MAX_UPLOAD_BYTES. The limit is enforced at the
origin, not at the edge.
The URL was printed but the clipboard is empty. The clipboard copy is not allowed to fail the
upload. If neither pbcopy nor wl-copy is on PATH, the URL still goes to stdout and the exit
code still reflects the upload.