Skip to content

Design: Vault Agent certificate renewal

See proposal.md, Why, for the motivation. Each fact below was read on 2026-10-12 or 2026-10-13.

  • Six roles issue a certificate from the ICA1 mount fzymgc-house/v1/ica1/v1 at converge time: kea, dns-otel-collector, calgary-otel-collector, nas-support-otel-collector, nas-otel-collector and nas-rustfs. Each one reissues only when openssl x509 -checkend 604800 fails. The TTL is 720 hours, except 2160 hours for calgary.
  • The dns-node-worker already renews its own certificate with Vault Agent on sandiego and portland. The agent runs as its own user, authenticates with an AppRole per node (tf/vault/approle-dns-node-worker.tf, secret_id_ttl = 0), renders a pkiCert template, and sets lease_renewal_threshold = 0.5. The role installs the vault package from the HashiCorp apt source after a fingerprint check, at the pin dns_node_worker_vault_version (1.21.4-1). The control node mints the secret id once, when the node holds none.
  • The agent’s journal reaches ClickStack today. On 2026-10-12 and 2026-10-13, default.otel_logs held 151 rows with ServiceName = 'otelcol-dns' and _SYSTEMD_UNIT = 'dns-node-worker-pki.service'. The calgary collector ships the whole journal. The resolver and nas-support collectors ship a list of units.
  • Hosts reach Vault at https://vault.fzymgc.house through a Traefik TLS passthrough route. Vault sees the address of a Traefik pod, so an AppRole cannot bind a CIDR.
  • The collectors run otelcol-contrib 0.161.0. Its configtls has reload_interval, and its tlscheck receiver reads local files (file_path) and endpoints. Both were read from the source at the tag v0.161.0.
  • Kea’s HA listener sets require-client-certs: true. A probe without a client certificate fails the handshake.
  • On the NAS, otelcol-nas and rustfs are custom apps. The rustfs app runs an nginx terminator as uid 568 under cap_drop: [ALL]. The NAS cannot reach its own 192.168.20.203:443.
  • The AppRole otel-collector-nas exists in tf/vault and nothing uses it. Its policy also reads secret/data/fzymgc-house/cluster/clickstack.
  • The PKI roles kea-ha and otel-collector-dns-client serve both resolvers.
  • Network: docs/reference/network.md is the addressing authority. This change allocates no address. calgary is on Core at 192.168.40.21. The gateway puts Core and Main in its zone Internal.

Goals:

  • No certificate in scope depends on a person or a CI run to renew.
  • Each agent credential can do one thing: issue the certificates of its own host or app.
  • A failure of any part, the agent, Vault or the reload, shows as a falling tlscheck.time_left and a Pushover message at 7 days.

Non-Goals:

  • A shared certificate-renewal service. Each host or app runs its own agent.
  • Folding the dns-node-worker agent into the new role. A later change can do that.
  • Render rows in ClickStack for the NAS sidecars. Their logs stay in the container log, and the time-left alert covers their failures.

D1. One agent and one AppRole per host or app

Section titled “D1. One agent and one AppRole per host or app”
Agent Where AppRole PKI roles it can issue
vault-cert-agent sandiego cert-agent-sandiego otel-collector-dns-client, and kea-ha on the issuer of D10
vault-cert-agent portland cert-agent-portland otel-collector-dns-client, and kea-ha on the issuer of D10
vault-cert-agent calgary cert-agent-calgary otel-collector-calgary-client
vault-cert-agent nas-support cert-agent-nas-support the role of tf/vault/pki-otel-collector-nas-support.tf
sidecar vault-agent NAS app otelcol-nas cert-agent-nas-otelcol otel-collector-nas-client
sidecar vault-agent NAS app rustfs cert-agent-nas-rustfs nas-rustfs-s3-server

Each policy grants create and update on the issue path of its PKI roles and the two token self paths, as the dns-node-worker policy does. The issue response carries the CA chain, so the policy needs no read of ca_chain. It grants no secret path. Each AppRole sets secret_id_ttl = 0, secret_id_num_uses = 0, token_ttl 3600 and token_max_ttl 86400, as the dns-node-worker AppRole does, and binds no CIDR.

Both resolvers share otel-collector-dns-client, and both share kea-ha on the issuer of D10. A split into one role per node was planned and dropped on 2026-10-13, during task group 1. A Vault PKI role has only the switch allow_ip_sans and no list of allowed addresses (builtin/logical/pki/path_roles.go at v1.21.4), so a role per node cannot pin the HA address of its node. Kea checks the chain and the IP SAN of a peer, never its common name, and the OTel gateway checks only the chain. A split pins a name that nothing reads. The control that holds is the AppRole per node: an operator revokes the secret id of one node and leaves the other node alone.

An agent can issue anything its PKI role allows, so a role that allows IP SANs needs a reason. nas-rustfs-s3-server took the provider default allow_ip_sans = true, although its clients reach s3-direct.nas.fzymgc.house by name and its certificate carries no IP SAN. It sets allow_ip_sans = false. The tlscheck receiver dials with InsecureSkipVerify, so the probe of 192.168.20.203:443 still reads the expiry. kea-ha needs an IP SAN, because Kea rejects a hostname in a peer URL. No agent gets the ICA1 role kea-ha, and D10 gives the role an issuer whose name constraints permit only the two HA addresses.

A write of a PKI role or an AppRole replaces the whole object in Vault. The provider resources vault_pki_secret_backend_role and vault_approle_auth_backend_role model each field that these roles set, so Terraform owns every field.

The unused AppRole otel-collector-nas and its policy go away in the same pull request. Its policy reads the ingest token of ClickStack, which a certificate sidecar does not need.

Alternatives considered:

  • One agent for every certificate of the resolvers inside the dns-node-worker agent. That agent runs as the worker’s user with the worker’s grants. Mixing the grants of three consumers into it widens what one compromise reaches.
  • TLS certificate auth, where a host logs in with its current certificate. No cert auth backend exists, and the chain breaks for good when a host stays off longer than one TTL.
  • Response wrapping from the control node for each renewal. That puts a person back in the loop.

The role takes the dns-node-worker pattern: the HashiCorp apt source with its fingerprint check, the vault package, a system user vault-cert-agent, the AppRole files, and a systemd unit vault-cert-agent.service. The version of the vault package moves to one source pin that both roles read, in the same Renovate group as the image tag of D5.

A list of certificates drives the role. Each entry names the PKI role, the common name, the IP SANs, the destination directory, the consumer user and group, and the name of the bundle file. Each entry renders one template block. The block writes one bundle file: the certificate first, then the CA chain, then the key. The dns-node-worker agent renders the same shape. The agent sets lease_renewal_threshold = 0.5.

One pkiCert call for each file is a constraint of consul-template v0.41.1, the version in Vault 1.21.4. It keys a pkiCert dependency by the issue path and the destination. It reads the destination back to decide whether to renew. Three templates for three files therefore issue three certificates, and the key does not match the certificate. A file without the certificate also forces a new issue at each start of the agent. Go’s tls.LoadX509KeyPair, the tlscheck receiver and nginx read a certificate and a key from one bundle, and they skip the blocks of another type. The first resolver converge makes sure that Kea does too.

The agent trusts the server certificate of Vault through a copy of ansible/files/fzymgc-ca-bundle.pem, the estate root CA and ICA1. A host does not need that CA in its system store. The agent decides a renewal only from the time left on the bundle, so a changed common name reaches a host only after the bundle is removed. The role asserts the common name of each bundle.

The agent user joins the group of each consumer. Each destination directory is owned by the agent user and the consumer group, at mode 2750. The setgid bit gives each rendered file the consumer group, at mode 0640. If a consumer runs as root, as the calgary collector does, its role creates a group for the certificate directory. If the group of a consumer also holds secrets, its role creates a group only for the certificate directory and adds the consumer user to it. Kea is that case: _kea holds the HA and control credentials and the D2 TSIG secrets, so the Kea bundle has the group kea-ha-tls, in /etc/kea-ha-tls outside the 0750 root:_kea directory /etc/kea. The agent then reads no secret of the consumer. The role asserts the modes at the write. It reads each file as the consumer user, because a root-context check certifies a file that the consumer cannot read.

The control node mints the secret id only when the host holds none, after it destroys every earlier secret id of the AppRole. The secret id file is mode 0600 and owned by the agent user. To rotate it, remove the file and converge.

Consumer Reload
The four OpenTelemetry collectors reload_interval: 1h on the client TLS of the exporter
The RustFS terminator nginx -s reload every hour from a loop in the container command
Kea HA The guarded timer of D4

No agent runs a restart hook. On the Firewalla, an agent hook exited 0 for months and never restarted anything, because the agent held no tool to restart the collector. A consumer that reloads on its own schedule needs no privilege from the agent.

A timer kea-ha-cert-reload.timer runs every 15 minutes. Its service runs a script that the role installs, and the script exits 0 with no action unless every step below passes:

  1. The certificate file is newer than the start of isc-kea-dhcp4-server.service.
  2. The local Kea role from status-get is standby, or the role is primary and the certificate file is at least two hours old.
  3. Both nodes report hot-standby, and the partner is in touch with no interrupted communication. Kea keeps in-touch true after it loses the partner, so the script also reads communication-interrupted.

Then the script restarts isc-kea-dhcp4-server.service and waits up to five minutes for both nodes to report hot-standby. If they do not, the unit fails, and the journal row reaches ClickStack.

The two-hour wait on the primary separates the two restarts when both certificates renew together, and it puts the standby first. That order keeps DHCP served during the first restart. The standby and the primary come from status-get at run time, because the VIP holder does not decide them. On 2026-10-12 portland held the VIPs and sandiego was the Kea primary.

A timer and an idempotent script retry by themselves. A .path unit fires once, and a missed event leaves Kea on the old certificate. The tests of the script read status-get responses of the pair, and the same responses with only the state fields changed.

Each of the apps otelcol-nas and rustfs gains a compose service vault-agent. It runs the image hashicorp/vault at a pinned tag in the Renovate group of D2, as the uid of the consumer. It mounts the AppRole files read-only, the estate CA bundle, its agent configuration and a certificate directory that the consumer also mounts. The roles write the AppRole files as that uid at mode 0600. The image entrypoint changes to the user vault with su-exec, which needs CAP_SETUID, so the service runs /bin/vault agent directly with no capability. The consumer starts after a healthcheck of the sidecar finds the bundle. The terminator’s command becomes a loop that reloads nginx every hour and then runs nginx in the foreground.

D6. The roles stop issuing at converge time

Section titled “D6. The roles stop issuing at converge time”

In the pull request that moves a certificate to an agent, the role loses its checkend probe, its PKI issue task and its write of the certificate material. The consumer reads its certificate, key and CA from the bundle of D2, so its configuration changes in the same pull request. The role removes the old certificate, key and CA files. The control node still needs VAULT_TOKEN to mint a secret id and to read the other secrets of a role.

Each collector gains a tlscheck receiver with collection_interval: 5m in its metrics pipeline to ClickStack. Each receiver has file targets for the certificate files on its host, including the bundle of the dns-node-worker on the resolvers. The calgary collector also has the endpoint targets 192.168.20.203:443 and ha.fzymgc.house:443. The NAS collector cannot reach 192.168.20.203:443, so calgary holds that target. Kea gets file targets only, because its HA listener refuses a probe without a client certificate. The HA state check of D4 covers the channel itself.

bootstrap-script.yaml gains a dashboard Certificates grouped by ResourceAttributes['host.name'] and ResourceAttributes['tlscheck.target']. The resolvers report the same file paths, so the host keeps their targets apart. HyperDX 2.39.1 replaces the value expression of a gauge with the last value of each series, so a builder tile cannot show days. A raw SQL table therefore lists the days left for each target, lowest first. HyperDX evaluates a tile alert only on a line, stacked bar or number tile, so a builder line tile of min(tlscheck.time_left) carries the alert. The alert uses the threshold type below, the threshold 604800 seconds (7 days) and the interval 1h, and the existing Pushover webhook. Each series reduces to its last value in a bucket before the min(). The groupBy gives each host and target its own alert state, and the alert name Certificate expires in under 7 days: {{{group}}} puts both in the Pushover title. The message points to docs/operations/certificates.md.

The calgary collector has an optional canary target at /etc/otelcol-calgary/canary/canary.crt, off by default. To test the path to Pushover, an operator issues a certificate with a 24-hour life from the calgary collector role, writes it to the canary path, and converges with the canary on. The alert fires within one hour. Then the operator converges with the canary off and removes the file, and the alert returns to OK. The runbook holds the commands, so the drill can repeat.

A new PKI mount fzymgc-house/v1/kea-ha/v1 holds a sub-CA that ICA1 signs. ICA1 carries pathlen:1, so it can sign one CA below itself. The sub-CA carries:

  • max_path_length = 0, so it signs no CA of its own
  • permitted_ip_ranges 192.168.40.6/32 and 192.168.40.7/32, the Core addresses of sandiego and portland in docs/reference/network.md
  • permitted_dns_domains kea-ha-sandiego.fzymgc.house and kea-ha-portland.fzymgc.house
  • an end date of 2027-12-31 (not_after), before the ICA1 chain expires on 2028-01-06. ICA1 refuses a certificate that outlives it (leaf_not_after_behavior = err), so a life of two years cannot be issued

The role kea-ha on the new mount has the settings of the ICA1 role. The policies of the two resolver agents grant its issue path.

Both agents issue from the one sub-CA, so either node can issue a certificate for the HA address and name of the other node. One sub-CA per node closes that, at the cost of a second mount, and this design accepts the risk instead. A cross issue needs a compromised node, because the secret id is a 0600 file on the node. That node already serves DHCP and DNS and holds the shared HA basic-auth password. Only the two nodes use the HA channel. A CIDR bound to the AppRole does not help: Vault sees the address of a Traefik pod, not of the node, and a node logs in from its own address anyway. A client that enforces RFC 5280 name constraints rejects a certificate from the sub-CA for any other address or name. OpenSSL and Go both enforce them, and a task proves it with openssl verify. Vault 1.19 added permitted_ip_ranges, and the provider supports it on vault_pki_secret_backend_root_sign_intermediate from 4.7.0. The lock pins 5.12.0.

The workspace policy terraform-vault-admin grants the mount, the paths under it and the ICA1 path root/sign-intermediate. That policy refuses changes to itself, so the operator writes it by hand before the merge. The workspace can already write any policy and any AppRole, so the grant of the signature adds little reach.

The ICA1 role kea-ha stays until the resolvers move to their agents, and that pull request removes it. Kea’s trust anchor holds the ICA1 chain, so a certificate from the sub-CA verifies. The tlscheck file target of the Kea certificate sets scrape_all_certs, so the time left on the sub-CA reaches the dashboard and the alert as well.

Alternatives considered:

  • Accepting the breadth, because a compromised resolver already runs DHCP and DNS for every client. Sean chose the constrained issuer on 2026-10-13, after a security review of the agent policies.
  • Keeping the Kea certificate on converge-time issue by an operator. That brings back the manual renewal.
  • [An AppRole secret id is a long-lived credential on disk] → Its policy can only issue the certificates of its own host, which already holds their keys. The file is 0600 and owned by the agent user. Rotation is a file removal and a converge.
  • [A consumer cannot read a bundle] → The pilot or the first converge of each host makes sure that the consumer presents the certificate from the bundle, before the next host moves.
  • [A restart of the agent issues a new certificate each time] → The calgary pilot restarts the agent and checks that the serial stays the same.
  • [Vault is down for more than half a life] → The alert fires at 7 days. The runbook holds a manual issue with an operator token, written to the same paths, as the fallback.
  • [The Kea guard never sees a healthy partner] → Kea keeps the old certificate, which still has half its life. The unit failures reach ClickStack, and the alert fires at 7 days.
  • [An endpoint probe by address gets no SNI] → The terminator serves one certificate. The spec scenario compares the probed notAfter with openssl s_client.
  • [The vault package and the image drift apart] → One Renovate group moves both.
  • [The Kea sub-CA expires] → The tlscheck target reads every certificate of the Kea chain, so the sub-CA reaches the 7-day alert like a leaf.

Each step is one pull request in the order of the proposal. Until a step lands, its role issues at converge time as before.

Step Live change Rollback
1 main-cluster-vault creates the AppRoles and the policies, and removes otel-collector-nas git revert <merge sha> in a pull request. The next apply restores the old state
2 calgary runs the agent, the collector reloads and reports, and the alert exists git revert <merge sha>, then scripts/calgary-playbook.sh --tags calgary-otel-collector and systemctl disable --now vault-cert-agent on calgary
3 nas-support runs the agent git revert <merge sha>, then scripts/nas-playbook.sh --tags nas-support-otel-collector
4 otelcol-nas runs the sidecar git revert <merge sha>, then scripts/nas-playbook.sh --tags nas-otel
5 rustfs runs the sidecar and the reload loop git revert <merge sha>, then scripts/nas-playbook.sh --tags nas-rustfs
6 main-cluster-vault creates the Kea mount, its sub-CA and the role kea-ha on it, and grants it to the resolver agents git revert <merge sha> in a pull request. The revert removes the mount
7 Each resolver runs the agent and the Kea timer git revert <merge sha>, then for each node scripts/dns-playbook.sh --limit <node> --tags kea,dns-otel-collector, and systemctl disable --now vault-cert-agent kea-ha-cert-reload.timer

The design changes an operational pattern, so a copy goes to docs/engineering/specs/2026-10-13-vault-agent-cert-renewal-design.md with the first pull request.

  • Whether the dns-node-worker agent later moves into vault-cert-agent. Nothing in this change depends on the answer.
  • Whether Kea accepts a trust anchor that holds only the Kea sub-CA, so that the HA channel trusts no other ICA1 certificate. The first resolver converge can test it. If Kea needs the full chain, the trust anchor keeps the ICA1 chain, and the name constraints still hold.