Skip to content

Cloudflare Operations

Terraform in tf/cloudflare/ owns every Cloudflare DNS record, zone setting, tunnel, Access application, service token, R2 bucket, notification policy and Worker that this estate uses. The HCP Terraform workspace main-cluster-cloudflare plans on every pull request that touches the directory and applies on merge. This page states what exists, where each part is defined, and the procedures that touch it. tf/cloudflare/CLAUDE.md carries the file layout and the import rules.

Property Value
Zones Twelve account zones. Eleven carry a dns-<zone>.tf file. fovea-review.ai carries zone settings only
Terraform module tf/cloudflare/
Workspace main-cluster-cloudflare
Provider credential secret/fzymgc-house/infrastructure/cloudflare/bootstrap-token
Tunnel fzymgc-house-main
Worker hcp-terraform-discord
Public asset host i.hl-assets.dev, an R2 custom domain with no tunnel, no Worker and no origin

Terraform authenticates as one hand-made bootstrap token. terraform.tf reads it from Vault as an ephemeral value, so it never enters state or plan output. Account-owned workload tokens are made by Terraform with cloudflare_account_token: the Email Sending tokens in nas_email.tf and nodes_email.tf, the bucket-scoped R2 token in r2.tf, and the read-only token-expiry monitor in token_expiry_monitor.tf. The bootstrap token cannot make itself, so it stays manual.

The bootstrap token is scoped to every zone in the account. Any zone added to the account is covered without a token edit. Because Terraform is the source of truth for every zone, the token can edit DNS on every domain in the account. It lives only in Vault.

The token’s grants are authoritative only in Cloudflare. Read them from the API, never from a comment or this page:

Terminal window
CF_TOKEN=$(vault kv get -field=token secret/fzymgc-house/infrastructure/cloudflare/bootstrap-token)
curl -sS -H "Authorization: Bearer $CF_TOKEN" \
"https://api.cloudflare.com/client/v4/accounts/<account_id>/tokens" \
| jq '.result[] | select(.name | test("bootstrap")) | {expires_on, policies}'

The grants that this repository depends on, and why each one is load-bearing:

Grant Needed by
DNS Edit, Zone Read, Zone Settings Edit, on all zones in the account dns-<zone>.tf, the data.cloudflare_zone lookups, zone-settings.tf
Cloudflare Tunnel Edit tunnel.tf
Access: Apps and Policies Edit, Access: Identity Providers Edit, Access: Service Tokens Write access.tf
Workers Scripts Edit workers.tf
Account API Tokens Edit every cloudflare_account_token resource
Email Sending Edit nas_email.tf, nodes_email.tf
Workers R2 Storage Write and Read r2.tf
Cache Settings Write, Zone Transform Rules Write rulesets-hl_assets_dev.tf
Notifications Write access.tf and notifications-hl_assets.tf
Cache Purge the takedown procedure. No Terraform resource uses it, and it must stay

Cloudflare lets a token grant only permission groups that it holds itself. Before an apply that mints a workload token with a new group, add that group to the bootstrap token in the dashboard. A missing grant fails at apply with a 403, and a failed apply blocks the whole shared workspace for every zone until a change merges.

Two groups are absent by decision, not by oversight. Without Managed headers Write, Cloudflare Managed Transforms cannot be read or declared on any zone. Without Bot Management Write, the managed robots.txt module cannot be turned off. The edge-policy ADR records why.

Editing a token’s grants in place does not change its value, so a grant edit needs no Vault write. An expiry does. The token carries an expiry date, and a lapse breaks every apply in the workspace. token_expiry_monitor.tf mints a read-only credential at secret/fzymgc-house/cluster/hl-assets/token-expiry-monitor, and the scheduled workflow .github/workflows/hl-assets-operational-items.yml reads the expiry with it and fails when fewer than TOKEN_WARNING_DAYS (90) remain. To renew, mint a new token with the same grants and write it to Vault. The provider reads the latest version, so no configuration changes.

Path Content
secret/fzymgc-house/infrastructure/cloudflare/bootstrap-token The bootstrap token, key token
secret/fzymgc-house/infrastructure/cloudflare/discord-webhook Discord webhook URL for the Worker
secret/fzymgc-house/infrastructure/cloudflare/hcp-terraform-hmac HMAC secret the Worker validates
secret/fzymgc-house/cluster/cloudflared/tunnels/fzymgc-house-main The tunnel token, written by tunnel.tf
secret/fzymgc-house/cluster/cloudflare-access/oidc The Keycloak OIDC client that Access uses
secret/fzymgc-house/cluster/hl-assets/access-token The three uploader service tokens, six fields
secret/fzymgc-house/cluster/hl-assets/monitor-token The uptime monitor’s service token
secret/fzymgc-house/cluster/hl-assets/r2 The bucket-scoped R2 token and endpoints
secret/fzymgc-house/cluster/hl-assets/token-expiry-monitor The read-only token-metadata credential
secret/fzymgc-house/infrastructure/nas/cloudflare-email The NAS Email Sending token
secret/fzymgc-house/infrastructure/nodes/cloudflare-email The cluster nodes’ Email Sending token

Terraform is the source of truth for every Cloudflare DNS record and per-zone setting.

File Contents
tunnel.tf Tunnel CNAMEs from the for_each maps in variables.tf: <name>-wh.fzymgc.net, id.fzymgc.house, <name>.fzymgc.net
dns-<zone>.tf Every other record, one file per zone, one cloudflare_dns_record resource per record
zone-settings.tf A cloudflare_zone_setting for each setting that differs from the Cloudflare default

A zone is a data.cloudflare_zone lookup by name, not a managed resource. Terraform owns the records and settings, not zone registration. hl-assets.dev is the exception: r2.tf writes its zone id as a literal, because a zone id fed from a data source produces spurious replacement plans for the R2 custom domain. Do not change it back.

Records NOT managed by Terraform (by design)

Section titled “Records NOT managed by Terraform (by design)”

A controller owns these. Importing one gives the record two owners:

Pattern Owner
_acme-challenge.* TXT cert-manager DNS-01 (argocd/app-configs/cert-manager-config/cloudflare-acme-issuer.yaml)
cf<year>-<n>._domainkey TXT Cloudflare Email Routing DKIM
The proxied CNAME for an R2 custom domain, today i.hl-assets.dev The R2 API, through cloudflare_r2_custom_domain in r2.tf
minify and development_mode zone settings Nobody. Deprecated and transient
  1. For a service behind the tunnel, add an entry to webhook_services, internal_services or ssh_services in variables.tf. The map key is the hostname label and the record name.
  2. For any other record, add a cloudflare_dns_record to the matching dns-<zone>.tf. Do not add an import{} block. That block is only for adopting a record that already exists.
  3. Open a pull request. Make sure that the HCP plan shows the intended add and nothing else, then merge.

Adopting an existing (dashboard-created) record

Section titled “Adopting an existing (dashboard-created) record”

cf-terraforming generates the HCL and the import{} block. Point it at a throwaway provider schema directory so that init needs no HCP credentials:

Terminal window
export CLOUDFLARE_API_TOKEN=$(vault kv get -mount=secret -field=token \
fzymgc-house/infrastructure/cloudflare/bootstrap-token)
cf-terraforming generate --zone <zone_id> --resource-type cloudflare_dns_record \
--terraform-install-path <schema_dir> --modern-import-block

Make sure that the plan is a pure import before you merge. It must read N to import, 0 to add, 0 to change, 0 to destroy. The HCP speculative plan on the pull request is the gate.

Do not adopt the R2 custom domain’s CNAME, even though cf-terraforming offers it. Do not create or edit an R2 custom domain in the dashboard: the pinned provider cannot import cloudflare_r2_custom_domain, so a dashboard-made one is drift with no remedy.

  1. The token needs no change. Its scope covers every zone in the account.

  2. Create dns-<zone-slug>.tf, with . and - written as _. For a zone full of dashboard records, adopt them as above. For an empty zone, write the resources directly.

  3. Add each setting that differs from the Cloudflare default to zone-settings.tf. The API marks a changed setting with modified_on set:

    Terminal window
    curl -sS -H "Authorization: Bearer $CF_TOKEN" \
    "https://api.cloudflare.com/client/v4/zones/<zone_id>/settings" \
    | jq '.result[] | select(.modified_on != null) | {id, value}'
  4. Open a pull request. Make sure that the HCP plan is a pure import or the intended adds, then merge.

Type Use Case
A / AAAA Direct IP mapping
CNAME Alias / tunnel target (<id>.cfargotunnel.com)
MX / TXT Mail routing, SPF / DKIM / DMARC
CAA Certificate authority authorization

The tunnel gives external access to cluster services without a public address.

Internet -> Cloudflare -> Tunnel -> Traefik or Service -> Pod

tunnel.tf declares the tunnel fzymgc-house-main, its ingress configuration and the matching CNAMEs. The configuration source is Cloudflare, so the cloudflared connector in argocd/app-configs/cloudflared-main/ carries no ingress rules of its own. Each map in variables.tf produces one ingress rule and one CNAME per key:

Variable Hostname pattern Protection
webhook_services <name>-wh.fzymgc.net Public. Each origin validates its own webhook signature. An optional path regex limits the paths the tunnel forwards
internal_services <name>.fzymgc.net Cloudflare Access
ssh_services <name>.fzymgc.net Access browser-rendered SSH. The map is empty

id.fzymgc.house is a fixed ingress rule to the Keycloak Service. The tunnel token is written to Vault by tunnel.tf and read by the connector.

Two tunnel objects exist in the account that Terraform does not manage. Their records are managed: wh-argowf.fzymgc.house and wh-events.fzymgc.house in dns-fzymgc_house.tf point at one, and a record in dns-holomush_dev.tf points at the other. To adopt one, generate it with cf-terraforming --resource-type cloudflare_zero_trust_tunnel_cloudflared.

To add a service to the tunnel, add a map entry. To add a service that needs Access, land the Access application in the same commit: a CNAME without the application is a public endpoint.

A new tunnel is only for a separate connector or an isolation boundary.

  1. Add a cloudflare_zero_trust_tunnel_cloudflared, a cloudflare_zero_trust_tunnel_cloudflared_config and a data.cloudflare_zero_trust_tunnel_cloudflared_token in tunnel.tf.
  2. Write the token to Vault with a vault_kv_secret_v2, in the shape of tunnel_credentials.
  3. Point the CNAMEs at <new-tunnel-id>.cfargotunnel.com.
  4. Deploy a connector under argocd/app-configs/cloudflared-<name>/ that reads the token.

For a break-glass apply while HCP Terraform is unavailable, follow tf/cloudflare/MANUAL_APPLY.md.

access.tf declares the identity providers, the policies and the applications. Three identity providers exist: Keycloak (OIDC, client from Vault), Email One-Time PIN, and GitHub, which exists only when the workspace variable github_oauth_client_id is set.

Application Hostname Session Policies
Cluster Documentation cluster-docs.docs.fzymgc.house 24h docs_allow: four email domains
Uptime Kuma Status Page status.fzymgc.net 24h docs_allow
Keycloak Admin Console id.fzymgc.house/admin and /realms/master 4h docs_allow. Keycloak is not an allowed provider here, because gating Keycloak’s admin console with Keycloak is circular
hl-assets Uploader drop.fzymgc.net 24h Two, in order: scripted clients by four named service tokens (non_identity), then browser SSO for fzymgc.email and fuzzymagic.com

docs_allow admits fzymgc.email, fzymgc.house, fzymgc.dev and fuzzymagic.com. The uploader’s browser policy admits two of those. Keep them separate. Widening who can read the docs site must never widen who can upload.

The uploader sets enable_binding_cookie and http_only_cookie_attribute. It is the one browser surface that accepts multipart/form-data under a session cookie, and the origin trusts the tunnel rather than validating the Access JWT, so the binding cookie is the only CSRF control in play.

The four service tokens are named one by one in the non_identity policy rather than admitting every service token in the account. Revoking one client’s credential leaves the others working.

Service-token expiry: two controls, neither redundant

Section titled “Service-token expiry: two controls, neither redundant”

Every service token in access.tf carries duration = "8760h". An expired token gets a 302 to the Access login page, which on the wire looks the same as a token that was never in the policy.

Control Where Lead time Scope
cloudflare_notification_policy.hl_assets_service_token_expiry access.tf, emails var.notification_email One week. The alert type has no other setting Every Access service token in the account. The alert type takes no filter
The uptime monitor tf/uptime-kuma/monitors-hl-assets.tf Its check interval The monitor’s own token. The canary sends the token on every check, so an expired token reads as DOWN

var.notification_email has no default and no committed value. It is the non-sensitive workspace variable notification_email on main-cluster-cloudflare.

The three client credentials live at secret/fzymgc-house/cluster/hl-assets/access-token in six fields, {ios,cli,curl}_client_{id,secret}, written by the apply that creates the tokens. client_secret is readable at creation only.

  • share.fish and the documented curl one-liner read the Vault path at run time. Rotation costs them nothing beyond the apply.
  • The iOS pair is authoritative in 1Password and is pasted into the Shortcut by hand. Update the Shortcut after a rotation.

To rotate, set client_secret_version on the resource and raise it. previous_client_secret_expires_at sets the overlap during which the old secret still works. Move that timestamp into the past to revoke a compromised credential at once. Open a pull request, merge, then make sure that the Vault secret carries the new value:

Terminal window
vault kv metadata get secret/fzymgc-house/cluster/hl-assets/access-token # current_version rose by one

Never download a plan JSON for main-cluster-cloudflare to a tracked path or attach it to a pull request. The artifact is workspace-scoped and carries every credential in the workspace in cleartext. Fetch to a temporary directory, read what you need, delete it.

A fourth token, hl_assets_uptime_monitor, lives at secret/fzymgc-house/cluster/hl-assets/monitor-token, on a different path on purpose. The path keeps the terraform-uptime-kuma-admin grant narrow: a scheduled check must not hold every uploader credential. It needs no expiry alarm of its own, because the notification policy covers every service token in the account.

Destroying and recreating this token rotates the credential, and the monitor reads DOWN until main-cluster-uptime-kuma applies again. Re-apply that workspace in the same change.

Four cloudflare_notification_policy resources in notifications-hl_assets.tf watch R2 operation volume. They are the cross product of two axes, and both survive:

Anomaly, is something wrong? Cost ceiling, is a bill coming?
Class A (writes) 10,000 ops/month 1,000,000 ops/month
Class B (reads) 200,000 ops/month 10,000,000 ops/month

Where the numbers come from:

  • Each ceiling sits at the R2 free tier for its class, so the alarm fires no later than the first billable operation.
  • The Class B anomaly limit must clear a floor of 43,800 × (1 + replicas) operations a month. Two in-house sources produce Class B operations in a healthy system, each about 43,800 a month at a 60-second cadence: the canary monitor, whose path is left uncached so that it can detect real downtime, and the uploader’s own reachability prober, one HeadObject against the private bucket per STATUS_PROBE_INTERVAL per replica. billing_usage_alert is account-scoped with no bucket predicate, so both count against the same limit. The rule is limit >= 2 × 43,800 × (1 + replicas). If you scale the uploader Deployment or shorten the probe interval, raise the limit in the same change.
  • The Class A anomaly limit is two orders of magnitude lower because nothing bounds writes from below. Only the four authenticated ingest surfaces can produce a Class A operation.
  • A Class B spike means cache misses, not popularity. The public /f/ prefix is edge-cached for thirty days. Read a spike as key enumeration or as a caching regression.

The product literals are r2_class_a_operations and r2_class_b_operations. The API matcher is exact and case-sensitive, and product and limit each take exactly one value. A rejected value fails the whole shared workspace apply, so do not re-spell either literal.

The account also carries dashboard-made alert policies that Terraform does not track. Read them with GET /accounts/<account_id>/alerting/v3/policies. A spend alarm there does not substitute for the volume alarms above.

Transforms HCP Terraform notification webhooks into Discord embeds.

Property Value
Code cloudflare/workers/hcp-terraform-discord/worker.js
Terraform tf/cloudflare/workers.tf
Secrets DISCORD_WEBHOOK_URL and HMAC_SECRET, from the two Vault paths above

When HMAC_SECRET is set, the Worker validates X-TFE-Notification-Signature and rejects an invalid signature with 401.

Serves the Tesla Fleet public key of Home Assistant at https://ha.fzymgc.net/.well-known/appspecific/com.tesla.3p.public-key.pem. Every other path answers 404. Tesla reads the key when Home Assistant registers the domain, and each vehicle pairs with it. The Worker runs on the Workers Free plan with no origin, so the key answers while the cluster and the tunnel are down.

Property Value
Code cloudflare/workers/tesla-fleet-key/worker.js
Key cloudflare/workers/tesla-fleet-key/com.tesla.3p.public-key.pem, read into the TESLA_PUBLIC_KEY binding
Terraform tf/cloudflare/workers.tf
Domain ha.fzymgc.net, a Worker custom domain in the fzymgc.net zone
Runbook docs/operations/home-assistant.md, section Tesla Fleet

A change to the key file publishes a new Worker version on merge. While the file is absent, the binding is empty and the key path answers 404. The workspace main-cluster-cloudflare carries the trigger prefix cloudflare, so a merge that touches only cloudflare/workers/ starts a run. A merge that reports Run not triggered applied nothing, and an operator starts the run by hand from the workspace page.

hl-assets.dev Edge Rules and Managed Objects

Section titled “hl-assets.dev Edge Rules and Managed Objects”

This section is operational. The reasoning lives in the edge-policy ADR.

rulesets-hl_assets_dev.tf declares the only cloudflare_ruleset resources in the repository, one entry point per phase:

Ruleset phase Effect
http_request_cache_settings Cache everything under /f/*. Edge TTL 2592000 s, browser TTL 3600 s, 404 TTL 30 s
http_response_headers_transform Five security headers on every response. Content-Disposition inline for the allowlisted extensions under /f/, attachment for everything else under /f/
http_request_transform Blank the query string under /f/ before the cache phase

Change them the same way as everything else here: pull request, HCP plan, merge. Never in the dashboard.

A zone holds exactly one entry point per phase, and the pinned provider does not adopt an entry point that already exists. A second resource for a phase that has one is an apply failure for the whole shared workspace. Add rules to the rules list of the existing resource. List order is evaluation order. Before you declare a ruleset for a new phase, read the entry point:

Terminal window
curl -sS -H "Authorization: Bearer $CF_TOKEN" \
"https://api.cloudflare.com/client/v4/zones/<zone_id>/rulesets/phases/<phase>/entrypoint"

Error code 10003 means the phase is empty. Declare the resource with no import{} block. A body that reads request is not authorized with no error code is a permission denial, not an absence. The recovery import{} shape, for a phase that does return an id, is recorded next to the resources in rulesets-hl_assets_dev.tf.

Objects in the public bucket that are not user content

Section titled “Objects in the public bucket that are not user content”

hl-assets-public holds five permanent objects placed by this project. Four sit under the /f/ user-content prefix. tools/hl-assets-verify places them and re-places one only if its bytes differ. Deleting one breaks a verification check and leaves stale bytes serving from the edge.

Key What it proves
f/_probe/cacheprobe.heic .heic is cached at the edge. It is absent from Cloudflare’s default cacheable-extension list
f/_probe/cacheprobe.mov .mov is cached at the edge, for the same reason
f/_probe/dispositionprobe.svg A non-allowlisted extension downloads rather than renders
f/_probe/dispositionprobe.html The same, for the format with the largest execution surface
robots.txt The crawl policy is served from a bucket object, not from R2’s 404 page

The probes are permanent because the /f/* cache rule sets a 30-day edge TTL. A fixture placed and removed on each run would keep serving from the edge for up to a month unless the check also purged it, and a purge that fails silently leaves stale bytes with nothing noticing. Fixed keys need no purge.

Removing an object from hl-assets-public does not stop it being served at once.

Tier TTL Reachable by an operator?
Cloudflare edge 2592000 s (30 days) Yes. A zone or prefix purge clears it. This is what the Cache Purge grant exists for
Browser 3600 s (1 hour) No. Nothing an operator does reaches a browser that already holds the bytes

A takedown is delete plus purge, and anyone who fetched the link inside the last hour keeps a copy until their browser cache expires. The procedure is in hl-assets Uploader → Taking a link down. It depends on the Cache Purge grant under API Token Pattern. No Terraform resource uses that grant, and the procedure does.

  1. In the Cloudflare dashboard, create a custom token with every grant in the table under API Token Pattern.

  2. Set the zone resource to “All zones from an account” and pick the account. Do not pick “All Domains”: an account-owned token rejects that wildcard with Account tag in access policy must match tag in request uri.

  3. Set an expiry of one year, not never.

  4. Write it to Vault:

    Terminal window
    vault kv put secret/fzymgc-house/infrastructure/cloudflare/bootstrap-token token="<token>"
  5. Merge a change under tf/cloudflare/ so that the workspace plans and applies with the new token. Make sure that the run reads the token: a wrong token fails at plan with a 403.

tf/vault -> tf/cloudflare -> tf/uptime-kuma

tf/vault creates the HMAC secret and the grants. tf/cloudflare deploys the Worker, mints the workload tokens and writes their Vault paths. tf/uptime-kuma reads the monitor token. The workspaces carry no run triggers between them, so a change that touches two of them queues two runs with no ordering guarantee. Land the grant first, alone.

r2-authentik-orphan.tf holds a one-shot removed block that forgets an orphaned bucket from state. Delete the file after the apply that consumes it succeeds.

  1. Read propagation: dig @1.1.1.1 <name>.
  2. Make sure that the record is in Terraform state and not only in the dashboard.

403 Forbidden - Authentication error means the token lacks a grant, or the target zone is outside the token’s scope.

  1. Read the token’s grants with the API call under API Token Pattern.

  2. If one zone fails while others work, the token was re-scoped to named domains. Set it back to all zones in the account.

  3. Make sure that Vault holds the token:

    Terminal window
    vault kv metadata get secret/fzymgc-house/infrastructure/cloudflare/bootstrap-token

Account tag in access policy must match tag in request uri

Section titled “Account tag in access policy must match tag in request uri”

The token’s zone policy was set to “All Domains”. An account-owned token needs “All zones from an account”.

Terminal window
kubectl -n cloudflared get pods
kubectl -n cloudflared logs -l app.kubernetes.io/name=cloudflared
vault kv metadata get secret/fzymgc-house/cluster/cloudflared/tunnels/fzymgc-house-main
Terminal window
curl -sS -H "Authorization: Bearer $CF_TOKEN" \
"https://api.cloudflare.com/client/v4/accounts/<account_id>/workers/scripts" | jq '.result[].id'
  • HCP Terraform Operations: webhook notifications
  • tf/cloudflare/CLAUDE.md: file layout, import rules and the drift check for the R2 custom domain
  • Tunnel connector: argocd/app-configs/cloudflared-main/
  • Worker code: cloudflare/workers/hcp-terraform-discord/