Host certificates (Vault Agent)
Vault issues a certificate to each host that sends telemetry or serves TLS outside the cluster. A
Vault Agent on that host renews the certificate. No converge and no person renews it. The role
ansible/roles/vault-cert-agent installs the agent. On the NAS, the agent is a container of the
app that uses the certificate. The design is in
docs/engineering/specs/2026-10-13-vault-agent-cert-renewal-design.md.
The agents and their AppRoles
Section titled “The agents and their AppRoles”tf/vault/approle-cert-agents.tf holds one AppRole for each agent, and
tf/vault/policy-cert-agents.tf holds its policy. A policy allows the issue path of the PKI roles
of its own host or app, and the renewal and the read of its own token. It allows no secret path.
| Host | Agent | AppRole | PKI role | Bundle | Consumer |
|---|---|---|---|---|---|
| sandiego, portland | vault-cert-agent.service |
cert-agent-<node> |
otel-collector-dns-client |
/etc/otelcol-dns/certs/tls.pem |
otelcol-dns.service, as root |
| sandiego, portland | vault-cert-agent.service |
cert-agent-<node> |
kea-ha on fzymgc-house/v1/kea-ha/v1 |
/etc/kea-ha-tls/kea-ha.pem |
isc-kea-dhcp4-server.service, as _kea |
| calgary | vault-cert-agent.service |
cert-agent-calgary |
otel-collector-calgary-client |
/etc/otelcol-calgary/certs/tls.pem |
otelcol-calgary.service, as root |
| nas-support | vault-cert-agent.service |
cert-agent-nas-support |
otel-collector-nas-support-client |
/etc/otelcol-nas-support/certs/tls.pem |
otelcol-nas-support.service, as root |
| nas | container vault-agent of the app otelcol-nas |
cert-agent-nas-otelcol |
otel-collector-nas-client |
/mnt/apps/otel-collector/certs/tls.pem |
container otelcol of the same app, as root |
| nas | container vault-agent of the app rustfs |
cert-agent-nas-rustfs |
nas-rustfs-s3-server |
/mnt/main/rustfs/certs/tls.pem |
container rustfs-tls (nginx) of the same app, as uid 568 |
The agent runs as the system user vault-cert-agent. Its configuration is
/etc/vault-cert-agent/agent.hcl. The AppRole files are in /etc/vault-cert-agent/approle, at
mode 0600 for the agent user. The agent keeps its Vault token in memory and writes no token file.
It trusts the server certificate of Vault through /etc/vault-cert-agent/ca.pem, a copy of
ansible/files/fzymgc-ca-bundle.pem.
On the NAS, each app that uses a certificate has a container vault-agent. It runs the image
hashicorp/vault at the version of the vault package on the hosts (vault_agent_version in
ansible/inventory/group_vars/all.yml), with no capability and a read-only root file system. It
runs as the owner of its files: uid 0 in otelcol-nas and uid 568 (apps) in rustfs. Its
configuration, its AppRole files and its CA are in the directory vault-agent of the app:
/mnt/apps/otel-collector/vault-agent or /mnt/main/rustfs/vault-agent. The role of the app
writes them through vault-cert-agent/tasks/container.yml and restarts the app when one of them
changes. The container logs to Docker, not to the journal, so ClickStack holds no render rows for
it.
How a renewal works
Section titled “How a renewal works”The agent renders one bundle file for each certificate: the certificate first, then the CA chain, then the key. The file is 0640, owned by the agent user and the group of the consumer. The consumer reads its certificate, its key and its CA from that one file.
The agent renews a certificate at half of its life, with a random shift of 3 percent of the life. The calgary certificate lives 90 days, so it renews at about day 45. The certificates of the resolver pair live 30 days and renew at about day 15. A restart of the agent issues no new certificate. The agent reads the bundle and keeps it until the renewal time.
Each certificate has one template in agent.hcl. Do not split a bundle into three templates for
the certificate, the key and the CA. Vault Agent issues one certificate for each template, so the
key file then holds the key of another certificate.
An OpenTelemetry collector reads its bundle again every hour (reload_interval: 1h). The RustFS
terminator reloads nginx every hour from a loop in its command. Neither needs a restart after a
renewal.
Kea reads its HA certificate only at start. On each resolver node, kea-ha-cert-reload.timer
restarts Kea after a renewal, the Kea standby first. The DNS runbook describes the checks of the
timer (docs/operations/dns.md, The certificates of the pair).
The bundle directory of a consumer has a group of its own when the user of the consumer has a
group with secrets. The agent user joins the group of the bundle directory. On the resolvers, that
group is kea-ha-tls with the member _kea, so the agent cannot read the Kea credentials and the
TSIG secrets, which have the group _kea.
The agent decides a renewal only from the time left on the bundle. A change of the common name, the IP SANs or the PKI role of an entry therefore reaches the host only after you remove the bundle on the host and run the role. The role fails when the common name in the bundle differs from its entry.
The Kea HA issuer
Section titled “The Kea HA issuer”The Kea HA certificate needs an IP SAN, because Kea rejects a hostname in a peer URL. A PKI role cannot limit the value of an IP SAN. So no agent gets an ICA1 role that allows IP SANs, and the Kea HA certificate comes from its own issuer instead.
The mount fzymgc-house/v1/kea-ha/v1 (tf/vault/pki-kea-ha-issuer.tf) holds one issuer, the
sub-CA fzymgc-house Kea HA CA v1, which ICA1 signs. The sub-CA has pathlen:0. Its name
constraints permit only 192.168.40.6/32, 192.168.40.7/32, kea-ha-sandiego.fzymgc.house and
kea-ha-portland.fzymgc.house. A client that enforces RFC 5280, such as OpenSSL or Go, rejects any
other certificate of this issuer. The role kea-ha on the mount issues the certificate, and the
policies cert-agent-sandiego and cert-agent-portland grant its issue path. No ICA1
role issues the Kea HA certificate.
A certificate of the sub-CA carries the sub-CA as its chain. A client verifies it with the ICA1
chain. Kea’s trust anchor /etc/kea/certs/ca.crt is a copy of the ICA1 chain that the role
dns-common installs from Vault. To read the constraints and the end date of the sub-CA:
vault read -field=certificate fzymgc-house/v1/kea-ha/v1/issuer/default \ | openssl x509 -noout -subject -enddate -ext basicConstraints,nameConstraintsThe sub-CA expires on 2027-12-31, before the ICA1 chain expires on 2028-01-06. Vault refuses a
certificate that outlives its issuer. From 2027-12-01, a 30-day certificate outlives the sub-CA,
so the renewals of the Kea HA certificate fail. The sub-CA cannot outlive ICA1. Replace it in a
pull request to tf/vault/pki-kea-ha-issuer.tf after ICA1 renews, and make sure that the new
sub-CA is the default issuer of the mount.
Read the state of a certificate
Section titled “Read the state of a certificate”Each collector reports tlscheck.time_left, the seconds until the certificate expires. The
collector of each resolver node also reports the Kea HA bundle, with the Kea HA sub-CA, and the
bundle of the node worker. The calgary collector also reports the certificates that 192.168.20.203:443 (RustFS) and
ha.fzymgc.house:443 (Home Assistant) serve. The HyperDX dashboard Certificates shows the days
left for each host and target, lowest first, and the seconds left over time. To read the days with
a query, run this on the ClickStack server:
SELECT ResourceAttributes['host.name'] AS host, ResourceAttributes['tlscheck.target'] AS target, round(min(Value) / 86400, 1) AS days_leftFROM default.otel_metrics_gaugeWHERE MetricName = 'tlscheck.time_left' AND TimeUnix > now() - INTERVAL 15 MINUTEGROUP BY host, target ORDER BY days_leftTo read a bundle on calgary, run these commands as root. The second command prints nothing when
the certificate and the key match. On nas-support, run the same commands through
scripts/nas-adhoc.sh nas-support -b -m shell -a '<command>' with the path of its bundle:
openssl x509 -noout -serial -enddate -in /etc/otelcol-calgary/certs/tls.pemdiff <(openssl x509 -noout -pubkey -in /etc/otelcol-calgary/certs/tls.pem) \ <(openssl pkey -pubout -in /etc/otelcol-calgary/certs/tls.pem)The agent logs each render to the journal, and the calgary collector sends the whole journal to ClickStack. To read the render rows of the last day:
SELECT Timestamp, Body FROM default.otel_logsWHERE ResourceAttributes['host.name'] = 'calgary' AND LogAttributes['_SYSTEMD_UNIT'] = 'vault-cert-agent.service' AND Timestamp > now() - INTERVAL 1 DAYORDER BY Timestamp DESC LIMIT 20A certificate expires in under 7 days
Section titled “A certificate expires in under 7 days”The alert Certificate expires in under 7 days: {{{group}}} names the host and the target in its
title. A target is a bundle file or, on calgary, an endpoint. The alert compares seconds, and its
threshold is 604800 seconds, which is 7 days. HyperDX ignores a value expression on a gauge, so
the alert cannot compare days.
- If the target is a bundle file, read the agent on that host:
journalctl -u vault-cert-agent.service --since -1d. On the NAS, runscripts/nas-adhoc.sh nas -b -m command -a 'docker logs --since 24h ix-otelcol-nas-vault-agent-1'. For the RustFS certificate, the container isix-rustfs-vault-agent-1. - If the agent logs a login failure, the secret id is gone or destroyed. Rotate it (Rotate a secret id).
- If the agent logs a failed issue, read the policy of its AppRole and the PKI role in
tf/vault. - If Vault is down and the certificate expires before Vault returns, issue it by hand (Issue a certificate by hand).
- If the target is
ha.fzymgc.house:443, Home Assistant renews it. Read Home Assistant.
Rotate a secret id
Section titled “Rotate a secret id”The secret id does not expire. To replace it, remove the file on the host and converge the host. The converge destroys every secret id of the AppRole and creates a new one. For calgary:
-
Log in to calgary as
fzymgc(Home Assistant, Connection model) and runsudo rm -f /etc/vault-cert-agent/approle/secret_id. -
On the control node, run the converge:
Terminal window export VAULT_TOKEN="$(vault print token)"scripts/calgary-playbook.sh --tags vault-cert-agent
For the NAS app otelcol-nas, remove the file and run its tag:
scripts/nas-adhoc.sh nas -b -m command -a 'rm -f /mnt/apps/otel-collector/vault-agent/approle/secret_id'export VAULT_TOKEN="$(vault print token)"scripts/nas-playbook.sh --tags nas-otelFor the NAS app rustfs, remove /mnt/main/rustfs/vault-agent/approle/secret_id the same way and
run scripts/nas-playbook.sh --tags nas-rustfs.
To cut off one agent at once, destroy the secret ids of its AppRole on the control node. The agent
keeps its current token for at most 24 hours, the token_max_ttl of the AppRole:
role=cert-agent-calgaryfor a in $(vault list -format=json "auth/approle/role/$role/secret-id" | jq -r '.[]'); do vault write "auth/approle/role/$role/secret-id-accessor/destroy" secret_id_accessor="$a"doneIssue a certificate by hand
Section titled “Issue a certificate by hand”Use this only when Vault Agent cannot renew before the certificate expires. The control node issues the certificate with your token and writes the bundle to the same path, in the same shape. The agent reads the bundle and keeps it until half of its life is gone.
For calgary:
d=$(mktemp -d) && chmod 700 "$d"vault write -format=json fzymgc-house/v1/ica1/v1/issue/otel-collector-calgary-client \ common_name=otelcol-calgary.fzymgc.house > "$d/issue.json"jq -r '.data.certificate, .data.ca_chain[], .data.private_key' "$d/issue.json" > "$d/tls.pem"vault kv get -mount=secret -field=private_key fzymgc-house/infrastructure/calgary/automation-ssh > "$d/key"chmod 600 "$d/key"ssh -F /dev/null -o IdentitiesOnly=yes -o IdentityAgent=none -i "$d/key" fzymgc@calgary.fzymgc.house \ 'sudo install -o vault-cert-agent -g otelcol-calgary -m 0640 /dev/stdin /etc/otelcol-calgary/certs/tls.pem' \ < "$d/tls.pem"rm -rf "$d"On the NAS, issue with the PKI role otel-collector-nas-client and the common name
otelcol-nas.fzymgc.house, and copy the bundle with the wrapper before you remove $d:
scripts/nas-adhoc.sh nas -b -m ansible.builtin.copy \ -a "src=$d/tls.pem dest=/mnt/apps/otel-collector/certs/tls.pem owner=root group=root mode=0640"For RustFS, issue with the PKI role nas-rustfs-s3-server and the common name
s3-direct.nas.fzymgc.house, and copy the bundle as uid 568:
scripts/nas-adhoc.sh nas -b -m ansible.builtin.copy \ -a "src=$d/tls.pem dest=/mnt/main/rustfs/certs/tls.pem owner=apps group=apps mode=0640"For the Kea HA certificate of a resolver node, issue from the Kea HA mount with the HA address of
the node as the IP SAN. For portland, use kea-ha-portland.fzymgc.house and 192.168.40.7. Copy
the bundle with the DNS wrapper:
vault write -format=json fzymgc-house/v1/kea-ha/v1/issue/kea-ha \ common_name=kea-ha-sandiego.fzymgc.house ip_sans=192.168.40.6 > "$d/issue.json"jq -r '.data.certificate, .data.ca_chain[], .data.private_key' "$d/issue.json" > "$d/tls.pem"scripts/dns-adhoc.sh sandiego -b -m ansible.builtin.copy \ -a "src=$d/tls.pem dest=/etc/kea-ha-tls/kea-ha.pem owner=vault-cert-agent group=kea-ha-tls mode=0640"The timer kea-ha-cert-reload.timer restarts Kea with the new bundle within 15 minutes, when the
checks of the timer pass. For the collector of a resolver node, issue with the PKI role
otel-collector-dns-client and the common name otelcol-dns.fzymgc.house, and copy the bundle to
/etc/otelcol-dns/certs/tls.pem with the group otelcol-dns.
A collector and the RustFS terminator read the new bundle within one hour. Read the bundle again (Read the state of a certificate) to make sure that the serial changed and that the certificate and the key match.
The canary drill
Section titled “The canary drill”The drill makes sure that a short certificate life reaches Pushover. It puts a certificate with a 24-hour life at the canary path on calgary, and the calgary collector reports it. Run it after a change to the alert, the webhook or the calgary collector.
-
Issue the canary certificate and copy it to calgary. Only the certificate goes to calgary:
Terminal window d=$(mktemp -d) && chmod 700 "$d"vault write -format=json fzymgc-house/v1/ica1/v1/issue/otel-collector-calgary-client \common_name=otelcol-calgary.fzymgc.house ttl=24h | jq -r '.data.certificate' > "$d/canary.crt"vault kv get -mount=secret -field=private_key fzymgc-house/infrastructure/calgary/automation-ssh > "$d/key"chmod 600 "$d/key"ssh -F /dev/null -o IdentitiesOnly=yes -o IdentityAgent=none -i "$d/key" fzymgc@calgary.fzymgc.house \'sudo install -D -m 0644 /dev/stdin /etc/otelcol-calgary/canary/canary.crt' < "$d/canary.crt"rm -rf "$d" -
Converge the collector with the canary target on:
Terminal window export VAULT_TOKEN="$(vault print token)"scripts/calgary-playbook.sh --tags calgary-otel-collector -e calgary_otel_tlscheck_canary=true -
Wait for a Pushover message whose title starts with
Certificate expires in under 7 daysand namescalgaryand/etc/otelcol-calgary/canary/canary.crt. It arrives within one hour. -
Converge with the canary off, and remove the file on calgary:
Terminal window scripts/calgary-playbook.sh --tags calgary-otel-collectorLog in to calgary and run
sudo rm -rf /etc/otelcol-calgary/canary. -
Make sure that the alert state is
OKwithin two hours. At the start of each hour, HyperDX evaluates the readings of the hour before. The hour that holds the last canary reading keeps the alert inALERTfor one more evaluation. HyperDX sends no resolve message for the canary, because a target that stops reporting gets no new state.