Design: Vault Agent certificate renewal
Context
Section titled “Context”See proposal.md, Why, for the motivation. Each fact below was read on 2026-10-12 or 2026-10-13.
- Six roles issue a certificate from the ICA1 mount
fzymgc-house/v1/ica1/v1at converge time:kea,dns-otel-collector,calgary-otel-collector,nas-support-otel-collector,nas-otel-collectorandnas-rustfs. Each one reissues only whenopenssl x509 -checkend 604800fails. The TTL is 720 hours, except 2160 hours for calgary. - The dns-node-worker already renews its own certificate with Vault Agent on sandiego and portland.
The agent runs as its own user, authenticates with an AppRole per node
(
tf/vault/approle-dns-node-worker.tf,secret_id_ttl = 0), renders apkiCerttemplate, and setslease_renewal_threshold = 0.5. The role installs thevaultpackage from the HashiCorp apt source after a fingerprint check, at the pindns_node_worker_vault_version(1.21.4-1). The control node mints the secret id once, when the node holds none. - The agent’s journal reaches ClickStack today. On 2026-10-12 and 2026-10-13,
default.otel_logsheld 151 rows withServiceName = 'otelcol-dns'and_SYSTEMD_UNIT = 'dns-node-worker-pki.service'. The calgary collector ships the whole journal. The resolver and nas-support collectors ship a list of units. - Hosts reach Vault at
https://vault.fzymgc.housethrough a Traefik TLS passthrough route. Vault sees the address of a Traefik pod, so an AppRole cannot bind a CIDR. - The collectors run
otelcol-contrib0.161.0. Itsconfigtlshasreload_interval, and itstlscheckreceiver reads local files (file_path) and endpoints. Both were read from the source at the tagv0.161.0. - Kea’s HA listener sets
require-client-certs: true. A probe without a client certificate fails the handshake. - On the NAS,
otelcol-nasandrustfsare custom apps. Therustfsapp runs an nginx terminator as uid 568 undercap_drop: [ALL]. The NAS cannot reach its own192.168.20.203:443. - The AppRole
otel-collector-nasexists intf/vaultand nothing uses it. Its policy also readssecret/data/fzymgc-house/cluster/clickstack. - The PKI roles
kea-haandotel-collector-dns-clientserve both resolvers. - Network:
docs/reference/network.mdis the addressing authority. This change allocates no address. calgary is on Core at192.168.40.21. The gateway puts Core and Main in its zoneInternal.
Goals / Non-Goals
Section titled “Goals / Non-Goals”Goals:
- No certificate in scope depends on a person or a CI run to renew.
- Each agent credential can do one thing: issue the certificates of its own host or app.
- A failure of any part, the agent, Vault or the reload, shows as a falling
tlscheck.time_leftand a Pushover message at 7 days.
Non-Goals:
- A shared certificate-renewal service. Each host or app runs its own agent.
- Folding the dns-node-worker agent into the new role. A later change can do that.
- Render rows in ClickStack for the NAS sidecars. Their logs stay in the container log, and the time-left alert covers their failures.
Decisions
Section titled “Decisions”D1. One agent and one AppRole per host or app
Section titled “D1. One agent and one AppRole per host or app”| Agent | Where | AppRole | PKI roles it can issue |
|---|---|---|---|
vault-cert-agent |
sandiego | cert-agent-sandiego |
otel-collector-dns-client, and kea-ha on the issuer of D10 |
vault-cert-agent |
portland | cert-agent-portland |
otel-collector-dns-client, and kea-ha on the issuer of D10 |
vault-cert-agent |
calgary | cert-agent-calgary |
otel-collector-calgary-client |
vault-cert-agent |
nas-support | cert-agent-nas-support |
the role of tf/vault/pki-otel-collector-nas-support.tf |
sidecar vault-agent |
NAS app otelcol-nas |
cert-agent-nas-otelcol |
otel-collector-nas-client |
sidecar vault-agent |
NAS app rustfs |
cert-agent-nas-rustfs |
nas-rustfs-s3-server |
Each policy grants create and update on the issue path of its PKI roles and the two token
self paths, as the dns-node-worker policy does. The issue response carries the CA chain, so the
policy needs no read of ca_chain. It grants no secret path. Each AppRole sets secret_id_ttl = 0,
secret_id_num_uses = 0, token_ttl 3600 and token_max_ttl 86400, as the dns-node-worker
AppRole does, and binds no CIDR.
Both resolvers share otel-collector-dns-client, and both share kea-ha on the issuer of D10. A
split into one role per
node was planned and dropped on 2026-10-13, during task group 1. A Vault PKI role has only the
switch allow_ip_sans and no list of allowed addresses (builtin/logical/pki/path_roles.go at
v1.21.4), so a role per node cannot pin the HA address of its node. Kea checks the chain and the IP
SAN of a peer, never its common name, and the OTel gateway checks only the chain. A split pins a
name that nothing reads. The control that holds is the AppRole per node: an operator revokes
the secret id of one node and leaves the other node alone.
An agent can issue anything its PKI role allows, so a role that allows IP SANs needs a reason.
nas-rustfs-s3-server took the provider default allow_ip_sans = true, although its clients
reach s3-direct.nas.fzymgc.house by name and its certificate carries no IP SAN. It sets
allow_ip_sans = false. The tlscheck receiver dials with InsecureSkipVerify, so the probe of
192.168.20.203:443 still reads the expiry. kea-ha needs an IP SAN, because Kea rejects a
hostname in a peer URL. No agent gets the ICA1 role kea-ha, and D10 gives the role an issuer
whose name constraints permit only the two HA addresses.
A write of a PKI role or an AppRole replaces the whole
object in Vault. The provider resources vault_pki_secret_backend_role and
vault_approle_auth_backend_role model each field that these roles set, so Terraform owns every
field.
The unused AppRole otel-collector-nas and its policy go away in the same pull request. Its policy
reads the ingest token of ClickStack, which a certificate sidecar does not need.
Alternatives considered:
- One agent for every certificate of the resolvers inside the dns-node-worker agent. That agent runs as the worker’s user with the worker’s grants. Mixing the grants of three consumers into it widens what one compromise reaches.
- TLS certificate auth, where a host logs in with its current certificate. No cert auth backend exists, and the chain breaks for good when a host stays off longer than one TTL.
- Response wrapping from the control node for each renewal. That puts a person back in the loop.
D2. The role vault-cert-agent
Section titled “D2. The role vault-cert-agent”The role takes the dns-node-worker pattern: the HashiCorp apt source with its fingerprint check,
the vault package, a system user vault-cert-agent, the AppRole files, and a systemd unit
vault-cert-agent.service. The version of the vault package moves to one source pin that both
roles read, in the same Renovate group as the image tag of D5.
A list of certificates drives the role. Each entry names the PKI role, the common name, the IP SANs,
the destination directory, the consumer user and group, and the name of the bundle file. Each entry
renders one template block. The block writes one bundle file: the certificate first, then the CA
chain, then the key. The dns-node-worker agent renders the same shape. The agent sets
lease_renewal_threshold = 0.5.
One pkiCert call for each file is a constraint of consul-template v0.41.1, the version in Vault
1.21.4. It keys a pkiCert dependency by the issue path and the destination. It reads the
destination back to decide whether to renew. Three templates for three files therefore issue three
certificates, and the key does not match the certificate. A file without the certificate also forces
a new issue at each start of the agent. Go’s tls.LoadX509KeyPair, the tlscheck receiver and
nginx read a certificate and a key from one bundle, and they skip the blocks of another type. The
first resolver converge makes sure that Kea does too.
The agent trusts the server certificate of Vault through a copy of
ansible/files/fzymgc-ca-bundle.pem, the estate root CA and ICA1. A host does not need that CA in
its system store. The agent decides a renewal only from the time left on the bundle, so a changed
common name reaches a host only after the bundle is removed. The role asserts the common name of
each bundle.
The agent user joins the group of each consumer. Each destination directory is owned by the agent
user and the consumer group, at mode 2750. The setgid bit gives each rendered file the consumer
group, at mode 0640. If a consumer runs as root, as the calgary collector does, its role creates a
group for the certificate directory. If the group of a consumer also holds secrets, its role
creates a group only for the certificate directory and adds the consumer user to it. Kea is that
case: _kea holds the HA and control credentials and the D2 TSIG secrets, so the Kea bundle has the
group kea-ha-tls, in /etc/kea-ha-tls outside the 0750 root:_kea directory /etc/kea. The
agent then reads no secret of the consumer. The role asserts the modes at the write. It reads each file as
the consumer user, because a root-context check certifies a file that the consumer cannot read.
The control node mints the secret id only when the host holds none, after it destroys every earlier secret id of the AppRole. The secret id file is mode 0600 and owned by the agent user. To rotate it, remove the file and converge.
D3. Each consumer reloads by itself
Section titled “D3. Each consumer reloads by itself”| Consumer | Reload |
|---|---|
| The four OpenTelemetry collectors | reload_interval: 1h on the client TLS of the exporter |
| The RustFS terminator | nginx -s reload every hour from a loop in the container command |
| Kea HA | The guarded timer of D4 |
No agent runs a restart hook. On the Firewalla, an agent hook exited 0 for months and never restarted anything, because the agent held no tool to restart the collector. A consumer that reloads on its own schedule needs no privilege from the agent.
D4. The guarded Kea restart
Section titled “D4. The guarded Kea restart”A timer kea-ha-cert-reload.timer runs every 15 minutes. Its service runs a script that the role
installs, and the script exits 0 with no action unless every step below passes:
- The certificate file is newer than the start of
isc-kea-dhcp4-server.service. - The local Kea role from
status-getisstandby, or the role isprimaryand the certificate file is at least two hours old. - Both nodes report
hot-standby, and the partner is in touch with no interrupted communication. Kea keepsin-touchtrue after it loses the partner, so the script also readscommunication-interrupted.
Then the script restarts isc-kea-dhcp4-server.service and waits up to five minutes for both nodes
to report hot-standby. If they do not, the unit fails, and the journal row reaches ClickStack.
The two-hour wait on the primary separates the two restarts when both certificates renew together,
and it puts the standby first. That order keeps DHCP served during the first restart. The standby
and the primary come from status-get at run time, because the VIP holder does not decide them. On
2026-10-12 portland held the VIPs and sandiego was the Kea primary.
A timer and an idempotent script retry by themselves. A .path unit fires once, and a missed event
leaves Kea on the old certificate. The tests of the script read status-get responses of the
pair, and the same responses with only the state fields changed.
D5. The NAS sidecars
Section titled “D5. The NAS sidecars”Each of the apps otelcol-nas and rustfs gains a compose service vault-agent. It runs the image
hashicorp/vault at a pinned tag in the Renovate group of D2, as the uid of the consumer. It mounts
the AppRole files read-only, the estate CA bundle, its agent configuration and a certificate
directory that the consumer also mounts. The roles write the AppRole files as that uid at mode 0600.
The image entrypoint changes to the user vault with su-exec, which needs CAP_SETUID, so the
service runs /bin/vault agent directly with no capability. The consumer starts after a
healthcheck of the sidecar finds the bundle. The terminator’s command becomes a loop that reloads
nginx every hour and then runs nginx in the foreground.
D6. The roles stop issuing at converge time
Section titled “D6. The roles stop issuing at converge time”In the pull request that moves a certificate to an agent, the role loses its checkend probe, its
PKI issue task and its write of the certificate material. The consumer reads its certificate, key
and CA from the bundle of D2, so its configuration changes in the same pull request. The role
removes the old certificate, key and CA files. The control node still needs VAULT_TOKEN to mint
a secret id and to read the other secrets of a role.
D7. Time-left telemetry
Section titled “D7. Time-left telemetry”Each collector gains a tlscheck receiver with collection_interval: 5m in its metrics pipeline
to ClickStack. Each receiver has file targets for the certificate files on its host, including the
bundle of the dns-node-worker on the resolvers. The calgary collector also has the endpoint targets
192.168.20.203:443 and ha.fzymgc.house:443. The NAS collector cannot reach 192.168.20.203:443,
so calgary holds that target. Kea gets file targets only, because its HA listener refuses a probe
without a client certificate. The HA state check of D4 covers the channel itself.
D8. The dashboard and the alert
Section titled “D8. The dashboard and the alert”bootstrap-script.yaml gains a dashboard Certificates grouped by
ResourceAttributes['host.name'] and ResourceAttributes['tlscheck.target']. The resolvers report
the same file paths, so the host keeps their targets apart. HyperDX 2.39.1 replaces the value
expression of a gauge with the last value of each series, so a builder tile cannot show days. A raw
SQL table therefore lists the days left for each target, lowest first. HyperDX evaluates a tile
alert only on a line, stacked bar or number tile, so a builder line tile of min(tlscheck.time_left)
carries the alert. The alert uses the threshold type below, the threshold 604800 seconds (7 days)
and the interval 1h, and the existing Pushover webhook. Each series reduces to its last value in a
bucket before the min(). The groupBy gives each host and target
its own alert state, and the alert name Certificate expires in under 7 days: {{{group}}} puts both
in the Pushover title. The message points to docs/operations/certificates.md.
D9. The canary test
Section titled “D9. The canary test”The calgary collector has an optional canary target at /etc/otelcol-calgary/canary/canary.crt,
off by default. To test the path to Pushover, an operator issues a certificate with a 24-hour life
from the calgary collector role, writes it to the canary path, and converges with the canary on.
The alert fires within one hour. Then the operator converges with the canary off and removes the
file, and the alert returns to OK. The runbook holds the commands, so the drill can repeat.
D10. A constrained issuer for Kea HA
Section titled “D10. A constrained issuer for Kea HA”A new PKI mount fzymgc-house/v1/kea-ha/v1 holds a sub-CA that ICA1 signs. ICA1 carries
pathlen:1, so it can sign one CA below itself. The sub-CA carries:
max_path_length = 0, so it signs no CA of its ownpermitted_ip_ranges192.168.40.6/32and192.168.40.7/32, the Core addresses of sandiego and portland indocs/reference/network.mdpermitted_dns_domainskea-ha-sandiego.fzymgc.houseandkea-ha-portland.fzymgc.house- an end date of 2027-12-31 (
not_after), before the ICA1 chain expires on 2028-01-06. ICA1 refuses a certificate that outlives it (leaf_not_after_behavior = err), so a life of two years cannot be issued
The role kea-ha on the new mount has the settings of the ICA1 role. The policies of the two
resolver agents grant its issue path.
Both agents issue from the one sub-CA, so either node can issue a certificate for the HA address
and name of the other node. One sub-CA per node closes that, at the cost of a second mount, and
this design accepts the risk instead. A cross issue needs a compromised node, because the secret
id is a 0600 file on the node. That node already serves DHCP and DNS and holds the shared HA
basic-auth password. Only the two nodes use the HA channel. A CIDR bound to the AppRole does not
help: Vault sees the address of a Traefik pod, not of the node, and a node logs in from its own
address anyway. A client that enforces RFC 5280 name constraints rejects a
certificate from the sub-CA for any other address or name. OpenSSL and Go both enforce them, and a
task proves it with openssl verify. Vault 1.19 added permitted_ip_ranges, and the provider
supports it on vault_pki_secret_backend_root_sign_intermediate from 4.7.0. The lock pins 5.12.0.
The workspace policy terraform-vault-admin grants the mount, the paths under it and the ICA1
path root/sign-intermediate. That policy refuses changes to itself, so the operator writes it by
hand before the merge. The workspace can already write any policy and any AppRole, so the grant of
the signature adds little reach.
The ICA1 role kea-ha stays until the resolvers move to their agents, and that pull request
removes it. Kea’s trust anchor holds the ICA1 chain, so a certificate from the sub-CA verifies. The
tlscheck file target of the Kea certificate sets scrape_all_certs, so the time left on the
sub-CA reaches the dashboard and the alert as well.
Alternatives considered:
- Accepting the breadth, because a compromised resolver already runs DHCP and DNS for every client. Sean chose the constrained issuer on 2026-10-13, after a security review of the agent policies.
- Keeping the Kea certificate on converge-time issue by an operator. That brings back the manual renewal.
Risks / Trade-offs
Section titled “Risks / Trade-offs”- [An AppRole secret id is a long-lived credential on disk] → Its policy can only issue the certificates of its own host, which already holds their keys. The file is 0600 and owned by the agent user. Rotation is a file removal and a converge.
- [A consumer cannot read a bundle] → The pilot or the first converge of each host makes sure that the consumer presents the certificate from the bundle, before the next host moves.
- [A restart of the agent issues a new certificate each time] → The calgary pilot restarts the agent and checks that the serial stays the same.
- [Vault is down for more than half a life] → The alert fires at 7 days. The runbook holds a manual issue with an operator token, written to the same paths, as the fallback.
- [The Kea guard never sees a healthy partner] → Kea keeps the old certificate, which still has half its life. The unit failures reach ClickStack, and the alert fires at 7 days.
- [An endpoint probe by address gets no SNI] → The terminator serves one certificate. The spec
scenario compares the probed
notAfterwithopenssl s_client. - [The
vaultpackage and the image drift apart] → One Renovate group moves both. - [The Kea sub-CA expires] → The
tlschecktarget reads every certificate of the Kea chain, so the sub-CA reaches the 7-day alert like a leaf.
Migration Plan
Section titled “Migration Plan”Each step is one pull request in the order of the proposal. Until a step lands, its role issues at converge time as before.
| Step | Live change | Rollback |
|---|---|---|
| 1 | main-cluster-vault creates the AppRoles and the policies, and removes otel-collector-nas |
git revert <merge sha> in a pull request. The next apply restores the old state |
| 2 | calgary runs the agent, the collector reloads and reports, and the alert exists | git revert <merge sha>, then scripts/calgary-playbook.sh --tags calgary-otel-collector and systemctl disable --now vault-cert-agent on calgary |
| 3 | nas-support runs the agent | git revert <merge sha>, then scripts/nas-playbook.sh --tags nas-support-otel-collector |
| 4 | otelcol-nas runs the sidecar |
git revert <merge sha>, then scripts/nas-playbook.sh --tags nas-otel |
| 5 | rustfs runs the sidecar and the reload loop |
git revert <merge sha>, then scripts/nas-playbook.sh --tags nas-rustfs |
| 6 | main-cluster-vault creates the Kea mount, its sub-CA and the role kea-ha on it, and grants it to the resolver agents |
git revert <merge sha> in a pull request. The revert removes the mount |
| 7 | Each resolver runs the agent and the Kea timer | git revert <merge sha>, then for each node scripts/dns-playbook.sh --limit <node> --tags kea,dns-otel-collector, and systemctl disable --now vault-cert-agent kea-ha-cert-reload.timer |
The design changes an operational pattern, so a copy goes to
docs/engineering/specs/2026-10-13-vault-agent-cert-renewal-design.md with the first pull request.
Open Questions
Section titled “Open Questions”- Whether the dns-node-worker agent later moves into
vault-cert-agent. Nothing in this change depends on the answer. - Whether Kea accepts a trust anchor that holds only the Kea sub-CA, so that the HA channel trusts no other ICA1 certificate. The first resolver converge can test it. If Kea needs the full chain, the trust anchor keeps the ICA1 chain, and the name constraints still hold.