Outbound Email (Cloudflare Email Sending)

How applications in the fzymgc-house cluster send transactional email (verification, password reset, notifications). The cluster standard is Cloudflare Email Service — “Email Sending” over authenticated SMTP submission, with a distinct API token per app.

This page is the reusable recipe: follow it to onboard a new app. Keycloak is the reference implementation (hl-zih3).

The model

Property	Value
Host	smtp.mx.cloudflare.net
Port	465
Security	Implicit TLS (SMTPS) — not STARTTLS
SMTP username	the literal string api_token (identical for every app)
SMTP password	a Cloudflare API token with the Email Sending: Edit permission
Sender domain	hl.fzymgc.house (onboarded under Email Service → Email Sending)

Two facts drive every integration:

• The per-app credential is the password, not the username. The username is always the literal api_token; each app’s own Cloudflare API token goes in the password slot. “Each app gets its own key” means a distinct token per app.
• Email Sending: Edit is account-scoped. A token can send from any onboarded domain on the account — it cannot be locked to one domain. The per-app token buys independent rotation/revocation and per-token attribution in the Email Sending → Activity Log, not sender-domain isolation. Treat every token as a full send credential.

Prerequisites (one-time, account-level)

1. The sending domain is onboarded in the Cloudflare dashboard under Email Service → Email Sending (status Enabled, DNS Configured). The MX / SPF / DKIM records are auto-created and Locked (Cloudflare-managed); _dmarc is Unlocked (v=DMARC1; p=reject;). hl.fzymgc.house is already onboarded.
2. The envelope sender (MAIL FROM) must be on an onboarded domain, or Cloudflare rejects the message. Use an address like auth@hl.fzymgc.house.

Recipe: onboard a new app

1. Create a per-app Cloudflare API token

Create an account-owned token (survives the creator leaving) with the Email Sending: Edit permission. Two ways, pick per how the app’s Vault secret is owned (step 2):

• Terraform-provisioned (preferred when the app’s cluster/<app> secret is already Terraform-managed): clone the cloudflare_account_token + vault_kv_secret_v2 pattern in tf/cloudflare/r2.tf, swapping the R2 permission groups for Email Sending. Resolve the permission-group id by exact name (Email Sending Write) via cloudflare_account_api_token_permission_groups_list (fail-loud at plan if the name drifts). tf/cloudflare/nas_email.tf (hl-xz9j) is the worked example. One-time prereq: the bootstrap token must itself hold Email Sending: Edit before it can mint a token with that group (Cloudflare only grants groups the creating token holds) — add it in the dashboard, see tf/cloudflare/CLAUDE.md.
• Manually seeded (when cluster/<app> is operator-seeded, so a vault_kv_secret_v2 write would clobber it — this is Keycloak’s case): create the token in the dashboard (Manage Account → API Tokens) and seed it into Vault by hand, the same way GitHub OAuth secrets are seeded on cluster/keycloak.

2. Store the token on the app’s own Vault secret

Add the token as cf_email_token on the app’s existing secret/fzymgc-house/cluster/<app> path, alongside a email_from field set to an @hl.fzymgc.house address. Keeping both on the app’s own secret is the cluster convention — do not invent a per-app subpath for the token.

Register the new keys in the Secrets Reference.

3. Point the app at Cloudflare

Configure the app’s SMTP client with implicit TLS on 465, username api_token, password = cf_email_token, sender = email_from. The host, port, and username are non-secret and identical for every app, so hardcode them in the app’s config and read only the token (and sender) from Vault.

Per-app TLS knob — every app defaults to STARTTLS and must be flipped:

App / framework	STARTTLS knob → implicit-TLS knob
Keycloak (keycloak_realm.smtp_server)	starttls → ssl = true, port = "465"
Django apps	EMAIL_USE_TLS → EMAIL_USE_SSL ("1", not "true"), port 465
TrueNAS (midclt mail.update)	security: "TLS" (STARTTLS) → security: "SSL" (implicit TLS), port = 465
Postfix (satellite relay)	smtp_tls_security_level=may → =encrypt plus smtp_tls_wrappermode=yes; relayhost=[smtp.mx.cloudflare.net]:465

Postfix has one extra requirement the app frameworks handle internally: node system mail is root@<node>.fzymgc.house, so the envelope sender must be rewritten to the onboarded domain (smtp_generic_maps = regexp:… → nodes@hl.fzymgc.house) or Cloudflare rejects the MAIL FROM.

4. Grant Vault read (Terraform consumers only)

If the app reads Vault from a Terraform workspace (like tf/keycloak), ensure its policy grants read on secret/{data,metadata}/fzymgc-house/cluster/<app> — usually already present, since the token lives on the app’s existing secret. Apps that read via ExternalSecrets/ESO use the fzymgc-cluster-secret-reader wildcard and need no change.

Reference implementation — Keycloak (hl-zih3)

• Realm SMTP: tf/keycloak/main.tf smtp_server block — host = "smtp.mx.cloudflare.net", port = "465", ssl = true, auth.username = "api_token", auth.password = data.vault_kv_secret_v2.keycloak.data["cf_email_token"], from = ...["email_from"].
• Vault: cf_email_token + email_from (auth@hl.fzymgc.house) on cluster/keycloak (operator-seeded). The stale, unreferenced email_password key was removed in the same change (the Mailgun password lived on cluster/smtp, never here).
• Policy: tf/vault/policy-keycloak.tf already grants cluster/keycloak read; the former cluster/smtp grant was removed.

Second implementation — NAS (hl-xz9j)

The TrueNAS nas host sends its native alert email (SMART / scrub / pool-health) via this transport. It differs from Keycloak in two ways worth noting as a pattern:

• Terraform-provisioned token (the preferred path above): tf/cloudflare/nas_email.tf mints the per-app token and writes cf_email_token + email_from to secret/fzymgc-house/infrastructure/nas/cloudflare-email. The tf/vault policy grant for that path must apply first (main-cluster-vault before main-cluster-cloudflare).
• Appliance transport, not app config: the nas-mail Ansible role pushes the SMTP settings through the TrueNAS middleware (midclt call mail.update), setting security: "SSL" (implicit TLS). See docs/operations/nas.md.

Third implementation — Turing Pi nodes (hl-zejv.2)

The Turing Pi 2 cluster nodes relay system mail (unattended-upgrades, cron) via a postfix satellite (ansible/roles/tp2-bootstrap-node). Two differences from the app consumers:

• Terraform-provisioned token, read control-node-side — same as the NAS: tf/cloudflare/nodes_email.tf mints the per-app token and writes cf_email_token + email_from to secret/fzymgc-house/infrastructure/nodes/cloudflare-email (tf/vault grant must apply first). The tp2-bootstrap-node role reads it over the operator’s Vault session at play time (community.hashi_vault.vault_kv2_get) — the nodes never touch Vault, so no node-side vault-agent is needed for one static token. Cold bootstrap (Vault not yet up) is non-fatal: the fetch is wrapped in a block/rescue that skips the relay config with a warning; re-run --tags mail once Vault is reachable.
• Envelope-sender rewrite — postfix must rewrite root@<node>.fzymgc.house to nodes@hl.fzymgc.house (smtp_generic_maps) so MAIL FROM is on the onboarded domain; the app frameworks set their sender directly, postfix does not.

Verify a cutover

1. Trigger a send (Keycloak: Forgot password, or the realm’s Test connection in the admin console → Realm settings → Email).
2. Confirm delivery in the Cloudflare dashboard: Email Service → Email Sending → Activity Log (the per-app token makes the source attributable).
3. Cross-check the app’s own logs / ClickStack default.otel_logs for send errors. Common failures: username not the literal api_token, STARTTLS/587 instead of implicit-TLS/465, or a MAIL FROM domain that is not onboarded.

Retiring the old Mailgun transport

The former shared transport secret/fzymgc-house/cluster/smtp (Mailgun) is deprecated. Once every consumer has moved to Cloudflare, delete the cluster/smtp secret and drop any lingering Vault grants.

The NAS carried a separate Mailgun SMTP credential inline in its own mail.config (not cluster/smtp). After the nas-mail cutover verifies, revoke that Mailgun key in the Mailgun dashboard — it is no longer in the appliance config but remains a live send credential until revoked.