Skip to content

Host certificates (Vault Agent)

Vault issues a certificate to each host that sends telemetry or serves TLS outside the cluster. A Vault Agent on that host renews the certificate. No converge and no person renews it. The role ansible/roles/vault-cert-agent installs the agent. On the NAS, the agent is a container of the app that uses the certificate. The design is in docs/engineering/specs/2026-10-13-vault-agent-cert-renewal-design.md.

tf/vault/approle-cert-agents.tf holds one AppRole for each agent, and tf/vault/policy-cert-agents.tf holds its policy. A policy allows the issue path of the PKI roles of its own host or app, and the renewal and the read of its own token. It allows no secret path.

Host Agent AppRole PKI role Bundle Consumer
sandiego, portland vault-cert-agent.service cert-agent-<node> otel-collector-dns-client /etc/otelcol-dns/certs/tls.pem otelcol-dns.service, as root
sandiego, portland vault-cert-agent.service cert-agent-<node> kea-ha on fzymgc-house/v1/kea-ha/v1 /etc/kea-ha-tls/kea-ha.pem isc-kea-dhcp4-server.service, as _kea
calgary vault-cert-agent.service cert-agent-calgary otel-collector-calgary-client /etc/otelcol-calgary/certs/tls.pem otelcol-calgary.service, as root
nas-support vault-cert-agent.service cert-agent-nas-support otel-collector-nas-support-client /etc/otelcol-nas-support/certs/tls.pem otelcol-nas-support.service, as root
nas container vault-agent of the app otelcol-nas cert-agent-nas-otelcol otel-collector-nas-client /mnt/apps/otel-collector/certs/tls.pem container otelcol of the same app, as root
nas container vault-agent of the app rustfs cert-agent-nas-rustfs nas-rustfs-s3-server /mnt/main/rustfs/certs/tls.pem container rustfs-tls (nginx) of the same app, as uid 568

The agent runs as the system user vault-cert-agent. Its configuration is /etc/vault-cert-agent/agent.hcl. The AppRole files are in /etc/vault-cert-agent/approle, at mode 0600 for the agent user. The agent keeps its Vault token in memory and writes no token file. It trusts the server certificate of Vault through /etc/vault-cert-agent/ca.pem, a copy of ansible/files/fzymgc-ca-bundle.pem.

On the NAS, each app that uses a certificate has a container vault-agent. It runs the image hashicorp/vault at the version of the vault package on the hosts (vault_agent_version in ansible/inventory/group_vars/all.yml), with no capability and a read-only root file system. It runs as the owner of its files: uid 0 in otelcol-nas and uid 568 (apps) in rustfs. Its configuration, its AppRole files and its CA are in the directory vault-agent of the app: /mnt/apps/otel-collector/vault-agent or /mnt/main/rustfs/vault-agent. The role of the app writes them through vault-cert-agent/tasks/container.yml and restarts the app when one of them changes. The container logs to Docker, not to the journal, so ClickStack holds no render rows for it.

The agent renders one bundle file for each certificate: the certificate first, then the CA chain, then the key. The file is 0640, owned by the agent user and the group of the consumer. The consumer reads its certificate, its key and its CA from that one file.

The agent renews a certificate at half of its life, with a random shift of 3 percent of the life. The calgary certificate lives 90 days, so it renews at about day 45. The certificates of the resolver pair live 30 days and renew at about day 15. A restart of the agent issues no new certificate. The agent reads the bundle and keeps it until the renewal time.

Each certificate has one template in agent.hcl. Do not split a bundle into three templates for the certificate, the key and the CA. Vault Agent issues one certificate for each template, so the key file then holds the key of another certificate.

An OpenTelemetry collector reads its bundle again every hour (reload_interval: 1h). The RustFS terminator reloads nginx every hour from a loop in its command. Neither needs a restart after a renewal.

Kea reads its HA certificate only at start. On each resolver node, kea-ha-cert-reload.timer restarts Kea after a renewal, the Kea standby first. The DNS runbook describes the checks of the timer (docs/operations/dns.md, The certificates of the pair).

The bundle directory of a consumer has a group of its own when the user of the consumer has a group with secrets. The agent user joins the group of the bundle directory. On the resolvers, that group is kea-ha-tls with the member _kea, so the agent cannot read the Kea credentials and the TSIG secrets, which have the group _kea.

The agent decides a renewal only from the time left on the bundle. A change of the common name, the IP SANs or the PKI role of an entry therefore reaches the host only after you remove the bundle on the host and run the role. The role fails when the common name in the bundle differs from its entry.

The Kea HA certificate needs an IP SAN, because Kea rejects a hostname in a peer URL. A PKI role cannot limit the value of an IP SAN. So no agent gets an ICA1 role that allows IP SANs, and the Kea HA certificate comes from its own issuer instead.

The mount fzymgc-house/v1/kea-ha/v1 (tf/vault/pki-kea-ha-issuer.tf) holds one issuer, the sub-CA fzymgc-house Kea HA CA v1, which ICA1 signs. The sub-CA has pathlen:0. Its name constraints permit only 192.168.40.6/32, 192.168.40.7/32, kea-ha-sandiego.fzymgc.house and kea-ha-portland.fzymgc.house. A client that enforces RFC 5280, such as OpenSSL or Go, rejects any other certificate of this issuer. The role kea-ha on the mount issues the certificate, and the policies cert-agent-sandiego and cert-agent-portland grant its issue path. No ICA1 role issues the Kea HA certificate.

A certificate of the sub-CA carries the sub-CA as its chain. A client verifies it with the ICA1 chain. Kea’s trust anchor /etc/kea/certs/ca.crt is a copy of the ICA1 chain that the role dns-common installs from Vault. To read the constraints and the end date of the sub-CA:

Terminal window
vault read -field=certificate fzymgc-house/v1/kea-ha/v1/issuer/default \
| openssl x509 -noout -subject -enddate -ext basicConstraints,nameConstraints

The sub-CA expires on 2027-12-31, before the ICA1 chain expires on 2028-01-06. Vault refuses a certificate that outlives its issuer. From 2027-12-01, a 30-day certificate outlives the sub-CA, so the renewals of the Kea HA certificate fail. The sub-CA cannot outlive ICA1. Replace it in a pull request to tf/vault/pki-kea-ha-issuer.tf after ICA1 renews, and make sure that the new sub-CA is the default issuer of the mount.

Each collector reports tlscheck.time_left, the seconds until the certificate expires. The collector of each resolver node also reports the Kea HA bundle, with the Kea HA sub-CA, and the bundle of the node worker. The calgary collector also reports the certificates that 192.168.20.203:443 (RustFS) and ha.fzymgc.house:443 (Home Assistant) serve. The HyperDX dashboard Certificates shows the days left for each host and target, lowest first, and the seconds left over time. To read the days with a query, run this on the ClickStack server:

SELECT ResourceAttributes['host.name'] AS host, ResourceAttributes['tlscheck.target'] AS target,
round(min(Value) / 86400, 1) AS days_left
FROM default.otel_metrics_gauge
WHERE MetricName = 'tlscheck.time_left' AND TimeUnix > now() - INTERVAL 15 MINUTE
GROUP BY host, target ORDER BY days_left

To read a bundle on calgary, run these commands as root. The second command prints nothing when the certificate and the key match. On nas-support, run the same commands through scripts/nas-adhoc.sh nas-support -b -m shell -a '<command>' with the path of its bundle:

Terminal window
openssl x509 -noout -serial -enddate -in /etc/otelcol-calgary/certs/tls.pem
diff <(openssl x509 -noout -pubkey -in /etc/otelcol-calgary/certs/tls.pem) \
<(openssl pkey -pubout -in /etc/otelcol-calgary/certs/tls.pem)

The agent logs each render to the journal, and the calgary collector sends the whole journal to ClickStack. To read the render rows of the last day:

SELECT Timestamp, Body FROM default.otel_logs
WHERE ResourceAttributes['host.name'] = 'calgary'
AND LogAttributes['_SYSTEMD_UNIT'] = 'vault-cert-agent.service'
AND Timestamp > now() - INTERVAL 1 DAY
ORDER BY Timestamp DESC LIMIT 20

The alert Certificate expires in under 7 days: {{{group}}} names the host and the target in its title. A target is a bundle file or, on calgary, an endpoint. The alert compares seconds, and its threshold is 604800 seconds, which is 7 days. HyperDX ignores a value expression on a gauge, so the alert cannot compare days.

  1. If the target is a bundle file, read the agent on that host: journalctl -u vault-cert-agent.service --since -1d. On the NAS, run scripts/nas-adhoc.sh nas -b -m command -a 'docker logs --since 24h ix-otelcol-nas-vault-agent-1'. For the RustFS certificate, the container is ix-rustfs-vault-agent-1.
  2. If the agent logs a login failure, the secret id is gone or destroyed. Rotate it (Rotate a secret id).
  3. If the agent logs a failed issue, read the policy of its AppRole and the PKI role in tf/vault.
  4. If Vault is down and the certificate expires before Vault returns, issue it by hand (Issue a certificate by hand).
  5. If the target is ha.fzymgc.house:443, Home Assistant renews it. Read Home Assistant.

The secret id does not expire. To replace it, remove the file on the host and converge the host. The converge destroys every secret id of the AppRole and creates a new one. For calgary:

  1. Log in to calgary as fzymgc (Home Assistant, Connection model) and run sudo rm -f /etc/vault-cert-agent/approle/secret_id.

  2. On the control node, run the converge:

    Terminal window
    export VAULT_TOKEN="$(vault print token)"
    scripts/calgary-playbook.sh --tags vault-cert-agent

For the NAS app otelcol-nas, remove the file and run its tag:

Terminal window
scripts/nas-adhoc.sh nas -b -m command -a 'rm -f /mnt/apps/otel-collector/vault-agent/approle/secret_id'
export VAULT_TOKEN="$(vault print token)"
scripts/nas-playbook.sh --tags nas-otel

For the NAS app rustfs, remove /mnt/main/rustfs/vault-agent/approle/secret_id the same way and run scripts/nas-playbook.sh --tags nas-rustfs.

To cut off one agent at once, destroy the secret ids of its AppRole on the control node. The agent keeps its current token for at most 24 hours, the token_max_ttl of the AppRole:

Terminal window
role=cert-agent-calgary
for a in $(vault list -format=json "auth/approle/role/$role/secret-id" | jq -r '.[]'); do
vault write "auth/approle/role/$role/secret-id-accessor/destroy" secret_id_accessor="$a"
done

Use this only when Vault Agent cannot renew before the certificate expires. The control node issues the certificate with your token and writes the bundle to the same path, in the same shape. The agent reads the bundle and keeps it until half of its life is gone.

For calgary:

Terminal window
d=$(mktemp -d) && chmod 700 "$d"
vault write -format=json fzymgc-house/v1/ica1/v1/issue/otel-collector-calgary-client \
common_name=otelcol-calgary.fzymgc.house > "$d/issue.json"
jq -r '.data.certificate, .data.ca_chain[], .data.private_key' "$d/issue.json" > "$d/tls.pem"
vault kv get -mount=secret -field=private_key fzymgc-house/infrastructure/calgary/automation-ssh > "$d/key"
chmod 600 "$d/key"
ssh -F /dev/null -o IdentitiesOnly=yes -o IdentityAgent=none -i "$d/key" fzymgc@calgary.fzymgc.house \
'sudo install -o vault-cert-agent -g otelcol-calgary -m 0640 /dev/stdin /etc/otelcol-calgary/certs/tls.pem' \
< "$d/tls.pem"
rm -rf "$d"

On the NAS, issue with the PKI role otel-collector-nas-client and the common name otelcol-nas.fzymgc.house, and copy the bundle with the wrapper before you remove $d:

Terminal window
scripts/nas-adhoc.sh nas -b -m ansible.builtin.copy \
-a "src=$d/tls.pem dest=/mnt/apps/otel-collector/certs/tls.pem owner=root group=root mode=0640"

For RustFS, issue with the PKI role nas-rustfs-s3-server and the common name s3-direct.nas.fzymgc.house, and copy the bundle as uid 568:

Terminal window
scripts/nas-adhoc.sh nas -b -m ansible.builtin.copy \
-a "src=$d/tls.pem dest=/mnt/main/rustfs/certs/tls.pem owner=apps group=apps mode=0640"

For the Kea HA certificate of a resolver node, issue from the Kea HA mount with the HA address of the node as the IP SAN. For portland, use kea-ha-portland.fzymgc.house and 192.168.40.7. Copy the bundle with the DNS wrapper:

Terminal window
vault write -format=json fzymgc-house/v1/kea-ha/v1/issue/kea-ha \
common_name=kea-ha-sandiego.fzymgc.house ip_sans=192.168.40.6 > "$d/issue.json"
jq -r '.data.certificate, .data.ca_chain[], .data.private_key' "$d/issue.json" > "$d/tls.pem"
scripts/dns-adhoc.sh sandiego -b -m ansible.builtin.copy \
-a "src=$d/tls.pem dest=/etc/kea-ha-tls/kea-ha.pem owner=vault-cert-agent group=kea-ha-tls mode=0640"

The timer kea-ha-cert-reload.timer restarts Kea with the new bundle within 15 minutes, when the checks of the timer pass. For the collector of a resolver node, issue with the PKI role otel-collector-dns-client and the common name otelcol-dns.fzymgc.house, and copy the bundle to /etc/otelcol-dns/certs/tls.pem with the group otelcol-dns.

A collector and the RustFS terminator read the new bundle within one hour. Read the bundle again (Read the state of a certificate) to make sure that the serial changed and that the certificate and the key match.

The drill makes sure that a short certificate life reaches Pushover. It puts a certificate with a 24-hour life at the canary path on calgary, and the calgary collector reports it. Run it after a change to the alert, the webhook or the calgary collector.

  1. Issue the canary certificate and copy it to calgary. Only the certificate goes to calgary:

    Terminal window
    d=$(mktemp -d) && chmod 700 "$d"
    vault write -format=json fzymgc-house/v1/ica1/v1/issue/otel-collector-calgary-client \
    common_name=otelcol-calgary.fzymgc.house ttl=24h | jq -r '.data.certificate' > "$d/canary.crt"
    vault kv get -mount=secret -field=private_key fzymgc-house/infrastructure/calgary/automation-ssh > "$d/key"
    chmod 600 "$d/key"
    ssh -F /dev/null -o IdentitiesOnly=yes -o IdentityAgent=none -i "$d/key" fzymgc@calgary.fzymgc.house \
    'sudo install -D -m 0644 /dev/stdin /etc/otelcol-calgary/canary/canary.crt' < "$d/canary.crt"
    rm -rf "$d"
  2. Converge the collector with the canary target on:

    Terminal window
    export VAULT_TOKEN="$(vault print token)"
    scripts/calgary-playbook.sh --tags calgary-otel-collector -e calgary_otel_tlscheck_canary=true
  3. Wait for a Pushover message whose title starts with Certificate expires in under 7 days and names calgary and /etc/otelcol-calgary/canary/canary.crt. It arrives within one hour.

  4. Converge with the canary off, and remove the file on calgary:

    Terminal window
    scripts/calgary-playbook.sh --tags calgary-otel-collector

    Log in to calgary and run sudo rm -rf /etc/otelcol-calgary/canary.

  5. Make sure that the alert state is OK within two hours. At the start of each hour, HyperDX evaluates the readings of the hour before. The hour that holds the last canary reading keeps the alert in ALERT for one more evaluation. HyperDX sends no resolve message for the canary, because a target that stops reporting gets no new state.