All 12 account zones under IaC (fzymgc.house, fzymgc.net, fzymgc.dev, fzymgc.email, plus bwcgroup.com, fovea-review.ai, fuzzymagic.com, holomush.dev, hl-assets.dev, muxserver.net, seanb4t.dev, specgraph.io)
Uses a single hand-created bootstrap token that Terraform authenticates as.
Account-owned workload tokens (e.g. the R2 S3 token in r2.tf) are created by
Terraform via cloudflare_account_token (works on provider 5.21); the bootstrap token
cannot bootstrap itself, so it stays manual.
Token
Purpose
Permissions
Created By
Bootstrap
Terraform auth + all operations
Account: Workers/R2/Tunnel/Access/API-Tokens; Zone (All zones in the account): DNS:Edit, Zone:Read, Zone Settings:Edit
Manual; 1-year expiry
Blast radius: since Terraform is the source of truth for every zone’s DNS +
settings, this token can edit DNS on all account domains (including personal ones
sharing the account). It lives only in Vault, never in git.
Three zone-level permission groups were added to the bootstrap token in the dashboard on
2026-07-31T13:05:35Z, taking it from 41 permission groups to 44 — exactly +3, with no
collateral widening. A second edit on 2026-08-14 added two account-level groups for the
v1.6 uploader, taking the token from 44 to 46 — again exactly +2, with no collateral
widening. The count was re-measured after each edit by set membership on the permission-group
id, never on a name substring, with the matcher shown able to report a fabricated id as
absent before its verdict on the real ones was believed:
cloudflare_ruleset.hl_assets_dev_cache_settings fails to apply without it
Zone Transform Rules Write (0ac90a90249747bca6b047d97f0803e9)
Phase 2
the response-header and URL-rewrite rulesets fail to apply without it
Cache Purge (e17beae8b8cb423a99b1730f21238bed)
Phase 6
takedown is not complete without a purge (see the takedown window below). It was granted early, in the same edit, so the takedown runbook is not blocked on a token edit later
Access: Service Tokens Write (a1c0fec57cf94af79479a6d827fa518c)
v1.6 Phase 5
the four cloudflare_zero_trust_access_service_token resources behind drop.fzymgc.net 403 at apply without it, and a 403 partway through fails the whole shared main-cluster-cloudflare workspace — taking unrelated DNS, tunnel and Access work down with it
cloudflare_notification_policy.hl_assets_service_token_expiry 403s at apply without it, with the same whole-workspace blast radius. Without the policy a service token expires in silence
Re-measured 2026-08-15 (milestone v1.6 Phase 6, plan 06-02): still 46 — no drift. The
Cache Purge id above was read off that run and is now pinned as
operations.CACHE_PURGE_GROUP_ID in tools/hl-assets-verify, so the next re-measure is a
set-membership check rather than a name grep. Until it was pinned the harness check FAILED
rather than passing: an unpinned expectation is satisfied by every account that has ever
existed, so it is not an assertion. Full id-to-name mapping and the fabricated-id positive
control: .planning/milestones/ws-image-host-r2-2026-09-19/milestones/v1.5-phases/06-operations-documentation/06-bootstrap-token.observed.txt.
The same run read the token’s expires_on as 2027-07-03T23:59:59Z (322 days out at
measurement).
R2 lifecycle authorisation is an OPEN ITEM, not a recorded grant. The token holds
Workers R2 Storage Write (bf7481a1826f439697cb59a20b22293e) and Workers R2 Storage Read
(b4992e1108244f5d8bfbd5744320c2e1); there is no group named Workers R2 Storage: Edit on
this account. What the permission-group set does not establish is whether that write group
authorises PutBucketLifecycleConfiguration specifically — the mapping carries no
per-operation detail. Any plan declaring cloudflare_r2_bucket_lifecycle must confirm this
from its own plan output first, because a 403 partway through fails the apply for the whole
shared main-cluster-cloudflare workspace.
Two groups are absent by deliberate choice, not by oversight. Do not add either without a
decision to go with it:
Absent group
Consequence of its absence
Managed headers Write
Cloudflare Managed Transforms cannot be read or declared on this zone (GET /zones/<id>/managed_headers returns request is not authorized). A managed transform that adds a header is undetectable from configuration
Bot Management Write
Cloudflare’s managed robots.txt module cannot be turned off. That is the disable-managed branch the edge-policy ADR rejected on cost
Editing a token’s permissions in place does not change its value. The Vault secret
at .../cloudflare/bootstrap-token was last written 2026-07-03 and was untouched by the
2026-07-31 permission edit — confirmed via vault kv metadata get. So a permission edit
needs no Vault write; an expiry does.
Standing hazard — the bootstrap token expires 2027-07-03T23:59:59Z. A lapse breaks
every apply in the shared main-cluster-cloudflare workspace, for every zone, not just
hl-assets.dev. Rotating it means minting a new token and writing the new value to Vault.
Terraform is the source of truth for every Cloudflare DNS record and per-zone
setting across all 11 account zones (adopted in hl-vzfh via cf-terraforming).
File
Contents
tunnel.tf
Tunnel CNAMEs driven by for_each variables (*-wh.fzymgc.net, auth/id.fzymgc.house, status.fzymgc.net)
dns-<zone>.tf
All other records per zone (one file per zone), one cloudflare_dns_record resource + import{} block each
zone-settings.tf
Per-zone cloudflare_zone_setting resources for settings explicitly changed from Cloudflare defaults
Tunnel-backed service: add to the relevant for_each variable in variables.tf (see webhook_services / internal_services).
Any other record: add a cloudflare_dns_record resource to the matching dns-<zone>.tf (no import{} block — that’s only for adopting pre-existing records).
Submit PR; the main-cluster-cloudflare HCP workspace plans, then applies on merge.
Generate faithful HCL + import blocks with cf-terraforming (provider v5), pointed at a
throwaway provider-schema dir so you don’t need HCP auth to init:
Verify zero-diff before merging by running terraform plan against live state with an
empty local state — it must report N to import, 0 to add, 0 to change, 0 to destroy.
R2 custom domains — two things a sweep must not touch.
The proxied CNAME that an R2 custom domain creates (currently i.hl-assets.dev) is
created and deleted by the R2 API, not the DNS API. cf-terraformingwill
offer it as a cloudflare_dns_record; do not adopt it. Nothing in the record’s
name marks it controller-owned, unlike _acme-challenge.* — which is exactly why it
is listed in tf/cloudflare/CLAUDE.md’s MUST-NOT-Import table.
The cloudflare_r2_custom_domain resource itself has no terraform import
support on provider 5.22.0 (upstream PR #7267 pending). A custom domain connected
in the dashboard is therefore permanent, unfixable drift with no remedy. Never create
or edit one in the dashboard — it must be born in Terraform on the first attempt.
When a new domain is onboarded to the Cloudflare account:
Token: no change needed. The bootstrap token is scoped All zones in the
account, so any domain in this Cloudflare account is covered automatically.
(Only relevant if the token were ever re-scoped to Specified Domains — then
you’d add the new domain to its policy first.)
Add records: create dns-<zone-slug>.tf (./- → _). For an existing zone
full of dashboard records, adopt them with cf-terraforming + import{} blocks
as above; for a greenfield zone, write cloudflare_dns_record resources directly.
Add changed settings to zone-settings.tf (only settings that differ from
Cloudflare defaults — check modified_on != null via
GET /zones/<id>/settings).
Open a PR; confirm the HCP plan is zero-diff (imports) or intended (new records),
then merge.
The zone itself is referenced as a data.cloudflare_zone (looked up by name), not
managed as a cloudflare_zone resource — Terraform owns the records and settings,
not zone creation/registration.
Exception — hl-assets.dev uses a literal zone id. This zone deliberately does
not follow the guidance above: its zone id is written as a literal in
tf/cloudflare/r2.tf, and terraform.tf gains nodata "cloudflare_zone" "hl_assets_dev" block. Reason: both open provider issues that
produce spurious cloudflare_r2_custom_domain replacement plans involve a zone_id
fed from a zone data source, and this zone’s acceptance gate is a clean plan
immediately after apply. A literal id removes that input entirely. Do not “fix” the
tree back to a data source — the divergence is intentional and this note is why.
The primary tunnel fzymgc-house-main is defined in tunnel.tf
(cloudflare_zero_trust_tunnel_cloudflared.main). Its ingress rules and the
matching DNS CNAMEs (<id>.cfargotunnel.com) are generated from for_each
variables in variables.tf — add a service by adding a map entry, not a raw record:
Variable
Hostname pattern
Protection
webhook_services
<name>-wh.fzymgc.net
Public (webhook receivers)
internal_services
<name>.fzymgc.net
Cloudflare Access
ssh_services
<name>.fzymgc.net
Access browser-rendered SSH
The tunnel token is written to Vault (.../cloudflared/tunnels/<name>) and consumed
by the cloudflared connector in argocd/app-configs/cloudflared-main/.
Adding a service to the existing tunnel is the common case (above). A new tunnel
is only needed for a separate connector / isolation boundary:
Add a cloudflare_zero_trust_tunnel_cloudflared resource + a
cloudflare_zero_trust_tunnel_cloudflared_config (ingress) in tunnel.tf, and a
data.cloudflare_zero_trust_tunnel_cloudflared_token to fetch its run token.
Persist the token to Vault via a vault_kv_secret_v2 (mirror the existing
tunnel_credentials resource).
Point DNS cloudflare_dns_record CNAMEs at <new-tunnel-id>.cfargotunnel.com.
Deploy a cloudflared connector (new argocd/app-configs/cloudflared-*/) that reads
the Vault token.
There is currently a second, un-IaC’d tunneld7ed1f27… backing
wh-argowf/wh-events.fzymgc.house (its DNS records are managed; the tunnel object
is not — tracked in hl-j4cv). Adopt it with cf-terraforming --resource-type cloudflare_zero_trust_tunnel_cloudflared when consolidating.
Access applications are declared in access.tf. Identity providers (Keycloak OIDC, GitHub
OAuth, Email one-time PIN) are shared; policies are not necessarily shared, and the difference
matters.
Application
Hostname
Session
Policies
Cluster Documentation
cluster-docs.docs.fzymgc.house
24h
docs_allow (four approved email domains)
Uptime Kuma Status Page
status.fzymgc.net
24h
docs_allow
Keycloak Admin Console
id.fzymgc.house/admin
4h
docs_allow; Keycloak itself excluded from allowed_idps (gating Keycloak’s admin console with Keycloak is circular)
hl-assets Uploader
drop.fzymgc.net
24h
two, in this order: scripted clients via four named service tokens (non_identity), then browser SSO limited to fzymgc.email and fuzzymagic.com
The uploader’s browser policy is deliberately NOT docs_allow.docs_allow fronts three
other applications and admits four domains; the uploader admits two. Keeping them separate is
the whole point — widening who can read the docs site or the status page must never silently
widen who can upload. Do not “tidy” this by pointing the uploader at the shared policy.
Two further deliberate choices on the uploader, recorded so a later reader does not undo them:
enable_binding_cookie = true, unlike the estate’s other three applications. It is the first
browser surface here that accepts multipart/form-data driven by a session cookie, and the
origin trusts the tunnel rather than validating the Access JWT — so the binding cookie is the
only CSRF lever in play.
The four service tokens — the three client classes (iOS Shortcut, share.fish, curl)
plus the uptime monitor, covered in The uptime monitor’s own service token below — are named
individually in the non_identity policy rather than admitting every service token in the
account. Revoking one family device’s credential must leave share.fish, the documented
curl one-liner and the monitor working.
Service-token expiry: two controls, neither redundant
Service tokens default to a one-year lifetime and their failure mode is quiet — an expired
token gets a 302 to the Access login, which on the wire is indistinguishable from a token
that was never in the policy at all.
every Access service token in the account — this alert type takes no filters
Harness gate
tools/hl-assets-verify
fails at now + 30 days
the uploader’s three tokens
Neither substitutes for the other: the notification is the human-facing control and gives a
week; the gate is the machine-facing one and gives a month. The notification’s account-wide
scope is a property of the alert type, not a misconfiguration — today it is a superset of one
group.
var.notification_email carries no default and no committed address. Its value is supplied
as the non-sensitive HCP Terraform workspace variable notification_email on the
main-cluster-cloudflare workspace. It is not a secret and does not belong in Vault; it is
also not Terraform’s to invent, which is why no address literal appears under tf/.
All three client credentials live at secret/fzymgc-house/cluster/hl-assets/access-token — one
Vault secret with six fields ({ios,cli,curl}_client_{id,secret}), written by the same apply
that creates the tokens. client_secret is readable only at creation; there is no recovery
read.
share.fish and the documented curl one-liner read that Vault path at run time, so
rotation costs them nothing beyond the apply.
The iOS pair is authoritative in 1Password and is pasted into the Shortcut by hand at
setup. That buys custody and rotation bookkeeping, not runtime secrecy — the Shortcut body
syncs through iCloud with the secret in it.
Rotation is client_secret_version (increment to trigger) plus
previous_client_secret_expires_at (the overlap window during which the old secret still
works). Moving that timestamp into the past is the immediate-revocation lever for a
compromised credential.
A Terraform plan JSON for main-cluster-cloudflare MUST NOT be downloaded to a tracked
path or attached to a PR. The artifact is workspace-scoped, not change-scoped: it carries
every credential in the whole estate’s workspace in cleartext, not just the ones your change
touches. Fetch to a temp dir, extract what you need, delete. PR #1750 exists because this was
once got wrong.
A fourth service token, hl_assets_uptime_monitor, exists alongside the three client
classes above, and its pair lives at secret/fzymgc-house/cluster/hl-assets/monitor-token —
a different Vault path, on purpose.
It is a fourth client class, not a fourth copy of an existing one. AU-03 is “one credential
per client class so that revoking one never revokes another”; giving the monitor its own pair
honours that rule rather than relaxing it.
The separate path keeps the terraform-uptime-kuma-admin grant narrow to that one path. A
scheduled health check must not hold every uploader credential in the estate.
It needs no new expiry alarm. hl_assets_service_token_expiry takes no filters and
already fires for every Access service token in the account, this one included.
Standing hazard:client_secret is readable only at creation, so destroying and
recreating the token rotates the credential and the monitor reads DOWN until
main-cluster-uptime-kuma re-applies. That is a false outage on a canary — re-apply the
consuming workspace in the same change.
Four cloudflare_notification_policy resources in tf/cloudflare/notifications-hl_assets.tf
watch R2 operation volume. Branch A is live — the alarms are declared in Terraform, not
recorded as a gap — and the runbook says so here so an operator learns the posture without
reading HCL.
They are the cross product of two axes that are not the same axis, and both survive:
Anomaly — is something wrong?
Cost ceiling — am I about to be billed?
Class A (writes)
10,000 ops/month
1,000,000 ops/month
Class B (reads)
200,000 ops/month
10,000,000 ops/month
Neither column substitutes for the other, and neither row does either. A single threshold
cannot answer both questions, and a single product literal cannot cover both operation classes.
Where the numbers come from:
The ceilings sit exactly at the R2 free tier for their own class — 1,000,000 Class A and
10,000,000 Class B operations per month — so the alarm fires no later than the first billable
operation of that class.
The Class B anomaly limit must clear a floor of 43,800 × (1 + replicas) operations a
month — roughly 87,600 at replicas = 1. Two in-house sources produce Class B
operations continuously in a perfectly healthy system, each about 43,800 a month at a
60-second cadence:
the canary monitor, whose path is deliberately left uncached so it can detect real
downtime, so every check at interval = 60 reaches R2;
the uploader’s own background reachability prober, one HeadObject against the
private bucket per STATUS_PROBE_INTERVAL per replica. The default is 60s and the
Deployment does not override it. The probe always 404s by design, and R2 bills that —
the only operation Cloudflare documents as not charged is one answered 401 Unauthorized.
Both count against the same limit: billing_usage_alert is account-scoped with a single
product filter and no bucket predicate. A threshold below the floor alarms on a healthy
system, forever. 200,000 is roughly 2.3× that floor at replicas = 1 — not the 4.5× this
page previously claimed, which came from counting only source 1 — and roughly 50× below the
ceiling. The margin rule is limit ≥ 2 × 43,800 × (1 + replicas), so a scale-up to two
replicas requires raising this limit to at least 262,800 in the same change; at four replicas
the healthy floor is 219,000 and 200,000 would fire every month on its own traffic.
tools/hl-assets-verify derives the floor from the monitor, the Deployment and config.go
and asserts the rule, so the coupling is a gate rather than a note.
The Class A anomaly limit is two orders of magnitude lower because nothing bounds it from
below — the monitor never writes, and only four authenticated ingest surfaces can produce a
Class A operation at all. The asymmetry between the two rows is deliberate.
A Class B spike means cache misses, not popularity: the public /f/ prefix is edge-cached
for thirty days, so a healthy month barely reaches the bucket. Read a spike as key enumeration
or as a caching regression undoing the edge policy.
The product literals were measured, not guessed..planning/.../06-available-alerts.observed.txt settled that billing_usage_alert is available
on this account but answers AvailableValues: null for both filter keys, so nothing is
enumerable from it. 06-usage-alert-products.observed.txt settled the literals by probing the
API’s own validator: r2_class_a_operations and r2_class_b_operations are accepted,
r2_storage is accepted but excluded (a storage product, not an operation class), and every
near-miss spelling is rejected with error 17106. Do not re-spell either literal — the
matcher is exact and case-sensitive, product and limit must each carry exactly one value
(error 17012), and a rejected value fails the whole shared twelve-zone workspace apply.
The account also carries a dashboard-created billing_budget_alert on
total_spend_dollars: ["200"]. It is untracked state, invisible to drift detection, and it
is a spend alarm rather than a volume one — it cannot tell you the read path is being
enumerated. It does not substitute for anything above, and it should not be presented as
tracked infrastructure.
hl-assets.dev (cc148ffdfdeaea99cd2f1aa72c5d6715) carries three cloudflare_ruleset
entry points, declared in tf/cloudflare/rulesets-hl_assets_dev.tf. These are the onlycloudflare_ruleset resources in the repository:
Ruleset phase
Id
Effect
http_request_cache_settings
231ea61aa69248b6b3b0e5d33de8f592
Cache Everything under /f/*; edge TTL 2592000 s, browser TTL 3600 s, 404 TTL 30 s
http_response_headers_transform
9b4fc274996f4baf96eff283f2935660
Five security headers on every response the phase reaches; Content-Disposition inline for the allowlisted extensions under /f/, attachment for everything else under /f/
http_request_transform
15851f44b8ca452a9de379183722c3a6
Blanks the query string under /f/ before the cache phase
Applied by VCS-triggered HCP runs run-GtWfmNcPwaXp26mV (1 added) and
run-BK9LU9AjR1hH2RQw (2 added). Change them the same way as everything else here: PR →
HCP plan → merge. Never in the dashboard.
The adopt-then-declare import{} idiom does NOT transfer to rulesets. Every DNS record
and zone setting in this repo is adopted from live state with an import{} block. Provider
5.22.0 has no entrypoint-adoption fallback for rulesets, so a ruleset that already
exists in a phase you are declaring is a hard apply failure in the shared workspace —
not a silent adoption. Before adding a ruleset for a new phase, read the entrypoint first:
Error code 10003 (could not find entrypoint ruleset…) means genuinely absent → declare
with noimport{}. A payload reading request is not authorized with no error code
is a permission denial, not an absence — do not read it as “nothing exists”. All three
phases above were measured empty this way before being declared, so zero import{} blocks
were written. The recovery import{} shape, for the day a read does return an id, is
recorded in tf/cloudflare/rulesets-hl_assets_dev.tf.
Objects in the public bucket that are NOT user content — do not delete
hl-assets-public holds five permanent objects placed by this project rather than by an
uploader. Four of them live under the /f/ user-content prefix. Deleting any of them is
not cleanup: it breaks a verification gate and leaves stale bytes serving from the edge.
Key
Served at
What it proves
Instruction
f/_probe/cacheprobe.heic
/f/_probe/cacheprobe.heic
.heic is cached at the edge (it is absent from Cloudflare’s default cacheable-extension allowlist)
Do not delete
f/_probe/cacheprobe.mov
/f/_probe/cacheprobe.mov
.mov is cached at the edge, same reason
Do not delete
f/_probe/dispositionprobe.svg
/f/_probe/dispositionprobe.svg
a non-allowlisted extension downloads rather than renders
Do not delete
f/_probe/dispositionprobe.html
/f/_probe/dispositionprobe.html
the same, for the format with the largest execution surface
Do not delete
robots.txt
/robots.txt
the crawl policy is served from a real bucket object, not from R2’s 404 page
Do not delete
Why they are permanent, in operator terms — this is the counter-intuitive part. The
/f/* cache rule sets a 2592000-second (30-day) edge TTL. A throwaway fixture placed
and then removed by the gate would keep being served from Cloudflare’s edge for up to a month
after deletion unless the gate also purged it. Exercising a purge on every gate run would
require the gate to hold Cache Purge, and a purge that silently fails leaves stale bytes
serving with nothing noticing. Permanent probes at fixed keys need no purge, leave no edge
residue, and are idempotent — the gate re-places only if the bytes differ.
The two dispositionprobe.* objects are permanent for exactly the same reason as the two
cacheprobe.* objects: they sit under the same /f/ prefix and therefore inherit the same
cache rule.
Fixture provenance, the inertness requirement for the two disposition probes, and the
accepted residuals on the camera captures are recorded in
tools/hl-assets-verify/fixtures/README.md. Do not re-encode, trim or “optimise” the
camera fixtures — that destroys the provenance the gate measures.
Removing an object from hl-assets-public does not immediately stop it being served.
Tier
TTL
Reachable by an operator?
Cloudflare edge
2592000 s (30 days)
Yes — a zone/prefix purge clears it. This is what Cache Purge on the bootstrap token exists for
Browser
3600 s (1 hour)
No. Nothing an operator does reaches a browser that already holds the bytes
So a takedown is delete + purge, and even then anyone who fetched the link inside the
last hour keeps their copy until their browser cache expires.
The procedure lives in
hl-assets Uploader → Taking a link down —
the three verbs, which one to choose, and the edge-purge command the verb hands you. It
depends on the Cache Purge grant recorded under API Token Pattern
above; do not remove that grant on the grounds that no current Terraform uses it. Nothing in
Terraform ever will: the purge is an operator action, and that grant is exactly what the
procedure runs on.
Account > Account API Tokens > Edit (workload tokens in r2.tf)
Account > Workers R2 Storage > Edit
Account > Access: Apps and Policies > Edit; Access: Organizations, IdPs, and Groups > Edit
Zone > DNS > Edit
Zone > Zone > Read
Zone > Zone Settings > Edit (for zone-settings.tf)
Zone Resources: All zones from an account → (the fzymgc account). Do not
use All Domains — an account-owned token rejects the all-zones wildcard with
Account tag in access policy must match tag in request uri.
Expiration: set a 1-year expiry (not never-expire) and a rotation reminder.
Cause: Editing this account-owned token with an All Domains (all-zones
wildcard) zone policy — Cloudflare requires the zone policy be pinned to the token’s
account.
Fix: Use Specified Domains and list every zone explicitly instead of All
Domains.