Skip to content

NATS Operations

NATS is the message broker for the cluster. This page states what exists, where each part is defined, and the procedures that touch it. Commands assume --context fzymgc-house.

Property Value
Namespace nats
Client address nats.nats.svc.cluster.local:4222, TLS with the nats-client-tls certificate
MQTT listener nats.nats.svc:1883, TLS with handshake_first: true
Monitoring port 8222, no TLS
Metrics port 7777, the Prometheus exporter
Chart nats from https://nats-io.github.io/k8s/helm/charts/, pinned in argocd/cluster-app/templates/nats.yaml
Values argocd/app-configs/nats/values.yaml
Cluster 3 replicas, routes over TLS with the nats-cluster-tls certificate
JetStream file store 10Gi per node on longhorn-encrypted, memory store 256Mi, domain fzymgc
Vault path secret/fzymgc-house/cluster/nats, read by the nats-credentials ExternalSecret
Streams in git DNS_EVENTS in the account DNS, for the events of dns-control. See The stream DNS_EVENTS
Telemetry The monitoring-otel-scraper collector scrapes port 7777 on each of the three pods. The ClickStack tiles are in argocd/app-configs/clickstack-alerts/bootstrap-script.yaml

Both certificates come from the fzymgc-house-issuer ClusterIssuer (argocd/app-configs/nats/certificate.yaml), last one year and renew 30 days before expiry.

NATS runs in operator mode. The operator is fzymgc-house. Four accounts exist:

Account Purpose JetStream How it reaches the server
SYS System monitoring and the push user Off Preloaded through resolver_preload in values.yaml, because the server needs it at start
SERVICES Cluster services Off. The account JWT has no JetStream limits Pushed by the PostSync job
IOT The MQTT bridge and IoT devices On Pushed by the PostSync job
DNS dns-control events on the stream DNS_EVENTS On. 1 GB of disk storage, 64 MB of memory storage Pushed by the PostSync job

An account is the NATS boundary for subjects and for the JetStream quota. A service that needs JetStream gets its own account, as dns-control does with DNS.

The resolver is type: full with dir: /data/jwt and allow_delete: false. The PostSync job nats-account-push (argocd/app-configs/nats/account-push-job.yaml) waits for the server and JetStream. It imports the operator and the SYS account with its seed, and it creates the push-admin user under SYS. It then pushes SERVICES, IOT and DNS with nsc push --system-user push-admin. The job runs after every sync and deletes itself on success.

values.yaml carries the SYS account public key twice, as system_account and as the resolver_preload key. Both must equal sys_account_public in Vault.

Terminal window
vault kv get secret/fzymgc-house/cluster/nats
Key Description
operator_jwt Operator JWT
operator_public Operator public key
operator_seed Operator seed. It signs the account JWTs. Only the nsc procedures read it, and the ExternalSecret does not map it
sys_account_seed, sys_account_public, sys_account_jwt The SYS account
services_account_seed, services_account_public, services_account_jwt The SERVICES account
iot_account_seed, iot_account_public, iot_account_jwt The IOT account
dns_account_seed, dns_account_public, dns_account_jwt The DNS account

The ExternalSecret also maps bridge_username and bridge_password from secret/fzymgc-house/cluster/mosquitto into the same Kubernetes Secret.

No NetworkPolicy in the repository restricts NATS clients. The mosquitto namespace carries the label nats-client: "true", and nothing reads it. A client needs the CA chain for fzymgc-house-issuer and credentials for its account.

nats-box runs in the namespace. It mounts every key of nats-credentials at /etc/nats-nkeys/default. Its default context gives services_account_seed to --nkey, and that context does not work. The file holds an account seed, not a user seed, so the nats CLI stops with Not a valid nkey user seed.

To run a nats command, sign a user that expires after one hour, and connect with its credentials file. The pod has no copy of the CA, so copy it in first. nsc import account accepts only a file that ends in .jwt, so the script copies the account JWT under that name. Replace DNS and dns_account_* with the account that you need:

Terminal window
kubectl get secret nats-client-tls -n nats -o jsonpath='{.data.ca\.crt}' | base64 -d \
| kubectl exec -i deploy/nats-box -n nats -- sh -c 'cat > /tmp/ca.crt'
kubectl exec -i deploy/nats-box -n nats -- sh -s <<'EOF'
set -eu
umask 077
K=/etc/nats-nkeys/default
W=$(mktemp -d)
export NKEYS_PATH="$W/nkeys" XDG_DATA_HOME="$W/data" XDG_CONFIG_HOME="$W/config"
mkdir -p "$W/keys"
cp "$K/dns_account_seed" "$W/keys/account.nk"
cp "$K/dns_account_jwt" "$W/dns.jwt"
nsc add operator -u "$K/operator_jwt"
nsc import account --file "$W/dns.jwt"
nsc import keys --dir "$W/keys"
nsc add user --account DNS --name admin --expiry 1h
nsc generate creds --account DNS --name admin > /tmp/admin.creds
rm -rf "$W"
EOF

The nats commands on this page leave out the connection flags. Add them after nats:

Terminal window
kubectl exec -it deploy/nats-box -n nats -- nats \
--server tls://nats.nats.svc.cluster.local:4222 --tlsca /tmp/ca.crt --creds /tmp/admin.creds \
stream ls

The nats server commands need a user in the SYS account.

Mosquitto (argocd/app-configs/mosquitto/) bridges to the NATS MQTT listener. The bridge configuration is rendered by the mosquitto-secrets ExternalSecret:

Property Value
Bridge address nats.nats.svc:1883
CA bundle /mosquitto/nats-ca/ca.crt from the fzymgc-ica1-ca ConfigMap
Credentials bridge_username and bridge_password from secret/fzymgc-house/cluster/mosquitto
Topic mapping topic # both 0 mqtt/: every Mosquitto topic under mqtt/ maps to the same NATS subject without the prefix, in both directions

The NATS MQTT listener runs in operator mode, so the bridge password is a bearer user JWT from the IOT account, and the username is any non-empty string. The listener sets handshake_first: true, so a client must start TLS at once and cannot use STARTTLS.

The bridge address must not carry a trailing dot. Go’s TLS stack rejects a trailing dot in the SNI name, and the bridge then fails with tlsv1 alert decode error.

Error Cause Fix
tlsv1 alert decode error Trailing dot in the bridge address Write nats.nats.svc
Connection Refused: not authorised Invalid or expired bearer JWT Generate a new JWT (see Create the bridge credential)
JetStream not enabled for account The IOT account JWT lacks JetStream Enable it with nsc edit account IOT --js-enable and push again
unacceptable protocol version A race on the first connect Transient. The bridge retries

Every procedure that signs keys needs the nsc CLI and a temporary nsc environment loaded from Vault:

Terminal window
brew install nats-io/nats-tools/nsc
export VAULT_ADDR=https://vault.fzymgc.house
vault login -method=oidc
export NKEYS_PATH=$(mktemp -d)
export NSC_HOME=$(mktemp -d)
vault kv get -field=operator_jwt secret/fzymgc-house/cluster/nats > "$NSC_HOME/operator.jwt"
nsc add operator -u "$NSC_HOME/operator.jwt"
# Only for a procedure that signs an account JWT:
mkdir -m 700 "$NSC_HOME/opkeys"
vault kv get -field=operator_seed secret/fzymgc-house/cluster/nats > "$NSC_HOME/opkeys/operator.nk"
nsc import keys --dir "$NSC_HOME/opkeys"

Remove both directories when you are done. They hold private keys.

The Namespace CLI also installs a binary named nsc. If nsc --version does not print nsc version 2.x, run the NATS nsc from the natsio/nats-box:0.20.0 image with both directories mounted.

Terminal window
vault kv get -field=operator_public secret/fzymgc-house/cluster/nats
for acct in sys services iot dns
do
echo "$acct: $(vault kv get -field=${acct}_account_public secret/fzymgc-house/cluster/nats)"
done
kubectl exec -it deploy/nats-box -n nats -- nats account list # the accounts the resolver holds
Terminal window
nsc add user --account IOT --name mqtt-bridge --bearer --conn-type MQTT
nsc generate creds --account IOT --name mqtt-bridge | grep -A1 'BEGIN.*JWT'

Store the JWT in Vault at secret/fzymgc-house/cluster/mosquitto as bridge_password. The mosquitto-secrets ExternalSecret picks it up at its next refresh.

  1. Create the account and write its keys to files. If the account needs JetStream, set its limits in the same step:

    Terminal window
    nsc add account NEW_ACCOUNT
    nsc edit account NEW_ACCOUNT --js-disk-storage 1G --js-mem-storage 64M
    OUT=$(mktemp -d)
    nsc describe account NEW_ACCOUNT --raw | tr -d '\n' > "$OUT/jwt"
    nsc describe account NEW_ACCOUNT --json | jq -r .sub | tr -d '\n' > "$OUT/public"
    tr -d '\n' < "$(find "$NKEYS_PATH" -name "$(cat "$OUT/public").nk")" > "$OUT/seed"

    A stream with num_replicas 3 reserves three times its max_bytes of the account disk limit. For example, a 256 MiB stream reserves 768 MiB, so the DNS account has 1 GB.

  2. Make sure that the keys are not in Vault yet. Then add the three values from the files. Always use patch. A put replaces every key on the path:

    Terminal window
    vault kv get -format=json secret/fzymgc-house/cluster/nats | jq '.data.data | keys'
    vault kv patch secret/fzymgc-house/cluster/nats \
    new_account_seed=@"$OUT/seed" \
    new_account_public=@"$OUT/public" \
    new_account_jwt=@"$OUT/jwt"
    rm -rf "$OUT"
  3. Add the JWT key to argocd/app-configs/nats/external-secret.yaml:

    data:
    - secretKey: new_account_jwt
    remoteRef:
    key: fzymgc-house/cluster/nats
    property: new_account_jwt
  4. Add the account to argocd/app-configs/nats/account-push-job.yaml, as an environment variable and as an import and push step in the script, in the shape of the SERVICES block:

    env:
    - name: NEW_ACCOUNT_JWT
    valueFrom:
    secretKeyRef:
    name: nats-credentials
    key: new_account_jwt
    Terminal window
    echo "$NEW_ACCOUNT_JWT" > /tmp/new_account.jwt
    nsc import account --file /tmp/new_account.jwt
    nsc push -a NEW_ACCOUNT -u "$NATS_URL" --ca-cert "$CA_FILE" --system-user push-admin
  5. Open a PR. After the merge, ArgoCD syncs and the PostSync job pushes the account.

  6. Make sure that the resolver holds it:

    Terminal window
    kubectl exec -it deploy/nats-box -n nats -- nats account info NEW_ACCOUNT

A user authenticates with a .creds file that holds its JWT and its seed:

Terminal window
nsc add user -a SERVICES myservice
nsc generate creds -a SERVICES -n myservice > myservice.creds

dns-control publishes its events as the user dns-control in the account DNS. The user can publish on dns.> and subscribe on _INBOX.> only. Its credentials file is the property nats_creds of secret/fzymgc-house/cluster/dns-control. dns-control reads the file at /etc/dns-control/nats/dns-control.creds, the default of its key nats_creds_file.

  1. Load the nsc environment of Procedures. The operator seed is not needed for a user.

  2. Import the DNS account and its seed:

    Terminal window
    vault kv get -field=dns_account_jwt secret/fzymgc-house/cluster/nats > "$NSC_HOME/dns.jwt"
    mkdir -m 700 "$NSC_HOME/dnskeys"
    vault kv get -field=dns_account_seed secret/fzymgc-house/cluster/nats > "$NSC_HOME/dnskeys/account.nk"
    nsc import account --file "$NSC_HOME/dns.jwt"
    nsc import keys --dir "$NSC_HOME/dnskeys"
  3. Add the user and write its credentials file:

    Terminal window
    nsc add user -a DNS dns-control --allow-pub 'dns.>' --allow-sub '_INBOX.>'
    (umask 077 && nsc generate creds -a DNS -n dns-control > "$NSC_HOME/dns-control.creds")
  4. Read the property names at the path, then write the file to the property nats_creds. The path holds other properties, so always use patch. A put replaces every property:

    Terminal window
    vault kv get -mount=secret -format=json fzymgc-house/cluster/dns-control | jq '.data.data | keys'
    vault kv patch -mount=secret fzymgc-house/cluster/dns-control nats_creds=@"$NSC_HOME/dns-control.creds"
  5. Remove both directories:

    Terminal window
    rm -rf "$NKEYS_PATH" "$NSC_HOME"

No command in this procedure prints the credentials. The user needs no $JS.API.> permission, because dns-control only publishes. If dns-control calls another JetStream API later, for example a stream lookup, the user also needs $JS.API.STREAM.INFO.DNS_EVENTS.

Rotate the user when its seed leaks, for example into a log or a transcript. The user JWT has no expiry, so a new credentials file does not stop the old one from working. The DNS account JWT must revoke the public key of the old user. The operator signs the account JWT, so this procedure needs the operator seed.

Reloader restarts dns-control when the Secret dns-control changes, and the new pod reads the new file. Write the new credentials before you revoke the old key. If you revoke first, the server disconnects dns-control before the new file reaches the pod.

  1. Load the nsc environment of Procedures, with the operator seed.

  2. Import the DNS account and its seed, as in step 2 of Create the dns-control user.

  3. Read the public key of the current user from Vault. The function prints the sub claim of the user JWT and does not print the seed:

    Terminal window
    user_key() {
    awk '/BEGIN NATS USER JWT/ { getline; print; exit }' \
    | jq -rR 'split(".")[1] | gsub("-"; "+") | gsub("_"; "/")
    | . + ("=" * ((4 - length % 4) % 4)) | @base64d | fromjson | .sub'
    }
    OLD=$(vault kv get -mount=secret -field=nats_creds fzymgc-house/cluster/dns-control | user_key)
    echo "$OLD"
  4. Add the new user and write its credentials file, as in step 3 of Create the dns-control user. Then read its public key:

    Terminal window
    NEW=$(user_key < "$NSC_HOME/dns-control.creds")
    echo "$NEW"
  5. Write the file to Vault, as in step 4 of Create the dns-control user.

  6. Refresh the ExternalSecret, or wait for its 5-minute interval:

    Terminal window
    kubectl annotate externalsecret dns-control -n dns-control force-sync=$(date +%s) --overwrite
  7. Make sure that dns-control connects with the new key. One server prints $NEW, and the other two print nothing:

    Terminal window
    for p in nats-0 nats-1 nats-2
    do
    kubectl exec -n nats "$p" -c nats -- wget -qO- 'http://localhost:8222/connz?auth=true&limit=256' \
    | jq -r '.connections[]? | select(.name == "dns-control") | .authorized_user'
    done
  8. Revoke the old key and export the account JWT:

    Terminal window
    nsc revocations add-user -a DNS -u "$OLD" --at 0
    nsc describe account DNS --raw | tr -d '\n' > "$NSC_HOME/dns-account.jwt"

    nsc prints unable to push, because the operator has no account server URL. The push happens in step 11.

  9. Write the account JWT to Vault. The path holds other properties, so use patch:

    Terminal window
    vault kv patch secret/fzymgc-house/cluster/nats dns_account_jwt=@"$NSC_HOME/dns-account.jwt"
  10. Refresh the ExternalSecret of NATS, or wait for its 15-minute interval:

    Terminal window
    kubectl annotate externalsecret nats-credentials -n nats force-sync=$(date +%s) --overwrite
  11. Push the DNS account with Push an account by hand.

  12. Make sure that each server holds the revocation. Each line prints $OLD with a timestamp:

    Terminal window
    ACCOUNT=$(vault kv get -field=dns_account_public secret/fzymgc-house/cluster/nats)
    for p in nats-0 nats-1 nats-2
    do
    kubectl exec -n nats "$p" -c nats -- wget -qO- "http://localhost:8222/accountz?acc=$ACCOUNT" \
    | jq -c '.account_detail.decoded_jwt.nats.revocations'
    done
  13. Remove both directories:

    Terminal window
    rm -rf "$NKEYS_PATH" "$NSC_HOME"
  1. Edit and export the account:

    Terminal window
    nsc edit account SERVICES
    nsc describe account SERVICES --raw > services.jwt
  2. Write the new JWT to Vault:

    Terminal window
    vault kv patch secret/fzymgc-house/cluster/nats services_account_jwt="$(cat services.jwt)"
  3. Refresh the ExternalSecret, or wait for its 15-minute interval:

    Terminal window
    kubectl annotate externalsecret nats-credentials -n nats force-sync=$(date +%s) --overwrite
  4. Sync the application, so that the PostSync job pushes the new JWT:

    Terminal window
    argocd app sync nats --force

    If the sync result lists no nats-account-push hook, use Push an account by hand.

  5. Make sure that the resolver holds the new JWT:

    Terminal window
    kubectl exec -it deploy/nats-box -n nats -- nats account info SERVICES

The PostSync job nats-account-push pushes every account after a sync. A self-heal sync of nats-credentials can replace that sync before the job runs. The self-heal sync is partial and runs no hooks, so the job never starts and the resolver keeps the old JWT. The controller log then shows skipHooks=true for a sync of the ExternalSecret alone.

These steps run the commands of the job in nats-box, which mounts the same Secret. The pushed JWT is the one in Vault, so the next PostSync job pushes the same JWT.

  1. Make sure that nats-box holds the new JWT. Replace dns with the account that you push. The two hashes must be equal:

    Terminal window
    vault kv get -field=dns_account_jwt secret/fzymgc-house/cluster/nats | tr -d '\n' | shasum -a 256
    kubectl exec -n nats deploy/nats-box -- sh -c \
    'tr -d "\n" < /etc/nats-nkeys/default/dns_account_jwt | sha256sum'
  2. Copy the CA into the pod, and push the account as a SYS user that expires after 15 minutes. Replace DNS and dns_account_jwt with the account that you push:

    Terminal window
    kubectl get secret nats-client-tls -n nats -o jsonpath='{.data.ca\.crt}' | base64 -d \
    | kubectl exec -i deploy/nats-box -n nats -- sh -c 'cat > /tmp/ca.crt'
    kubectl exec -i deploy/nats-box -n nats -- sh -s <<'EOF'
    set -eu
    umask 077
    K=/etc/nats-nkeys/default
    W=$(mktemp -d)
    export NKEYS_PATH="$W/nkeys" XDG_DATA_HOME="$W/data" XDG_CONFIG_HOME="$W/config"
    mkdir -p "$W/keys"
    cp "$K/sys_account_seed" "$W/keys/sys_account.nk"
    cp "$K/sys_account_jwt" "$W/sys.jwt"
    cp "$K/dns_account_jwt" "$W/account.jwt"
    nsc add operator -u "$K/operator_jwt"
    nsc import account --file "$W/sys.jwt"
    nsc import keys --dir "$W/keys"
    nsc add user --account SYS --name push-admin --expiry 15m
    nsc import account --file "$W/account.jwt"
    nsc push -a DNS -u nats://nats.nats.svc.cluster.local:4222 --ca-cert /tmp/ca.crt --system-user push-admin
    rm -rf "$W" /tmp/ca.crt
    EOF

    The push prints jwt updated once for each of the three servers.

Every account JWT is signed by the operator, so an operator rotation re-signs all of them and every client credential.

  1. Generate the new operator key pair and JWT.
  2. Re-sign every account JWT with the new operator.
  3. Write operator_jwt, operator_public and every *_account_jwt to Vault.
  4. Sync the application. NATS restarts with the new operator, and the PostSync job pushes the re-signed accounts.
  5. Issue new credentials to every client.

If the SYS account gets new keys as well, update system_account and the resolver_preload key in values.yaml in the same change.

Terminal window
kubectl exec -it deploy/nats-box -n nats -- nats server check
kubectl exec -it deploy/nats-box -n nats -- nats server report jetstream

The JetStream report lists the three servers in cluster nats and domain fzymgc, with an API error count of zero. The starred server is the meta leader.

A stream that a service depends on is declared in git. The stream DNS_EVENTS in the DNS account is the example:

Property Value
Configuration The ConfigMap dns-events-stream, key dns-events.json, in argocd/app-configs/nats/stream-dns-events.yaml
Subjects dns.>
Limits max_bytes 256 MiB, max_age 7 days, max_msg_size 64 KiB, discard: old, 3 replicas
Declared by The PostSync job nats-stream-dns-events, in sync wave 1 after nats-account-push

The job signs a user in the DNS account that expires after one hour. That user can only call the JetStream API. The job adds the stream when it is absent. When the stream exists, the job edits it to the ConfigMap, so a manual change goes back at the next sync. A second run prints No difference in configuration. To change the stream, change the ConfigMap in a PR.

dns-control publishes the events of its operations actions on DNS_EVENTS. The stream carries events only. The record of each action is the audit store of dns-control.

Property Value
Subjects in use dns.ops.<action>.<outcome>, for example dns.ops.flush.succeeded, and dns.ops.drain.overdue
Message id The header Nats-Msg-Id: the audit id of the action, or drain-overdue:<node>:<drain start in unix seconds>
Duplicate window 2 minutes. The stream drops a repeat of a message id inside it
Storage File, 3 replicas. The stream reserves 768 MiB of the 1 GB disk limit of the DNS account
Publisher The user dns-control. See Create the dns-control user

Read the stream with the connection flags of Client access:

Terminal window
kubectl exec -it deploy/nats-box -n nats -- nats \
--server tls://nats.nats.svc.cluster.local:4222 --tlsca /tmp/ca.crt --creds /tmp/admin.creds \
stream info DNS_EVENTS

The output shows the subjects dns.>, the limits of the ConfigMap, and the message count and size. Read the newest messages with stream view DNS_EVENTS in place of stream info DNS_EVENTS. The events and their payload are in the dns-control runbook, Operations actions.

The other commands for a stream, with a stream EVENTS as the example:

Terminal window
kubectl exec -it deploy/nats-box -n nats -- nats stream ls
kubectl exec -it deploy/nats-box -n nats -- nats stream add EVENTS \
--subjects="events.>" --storage=file --replicas=3 --retention=limits \
--max-msgs=-1 --max-bytes=1073741824 --max-age=168h --discard=old
kubectl exec -it deploy/nats-box -n nats -- nats stream info EVENTS
kubectl exec -it deploy/nats-box -n nats -- nats consumer ls EVENTS
kubectl exec -it deploy/nats-box -n nats -- nats stream backup EVENTS /path/to/backup
kubectl exec -it deploy/nats-box -n nats -- nats stream restore EVENTS /path/to/backup

A publish and subscribe test, in two terminals:

Terminal window
kubectl exec -it deploy/nats-box -n nats -- nats sub "test.>"
kubectl exec -it deploy/nats-box -n nats -- nats pub test.hello "Hello NATS"
Terminal window
kubectl port-forward svc/nats -n nats 7777:7777
curl localhost:7777/metrics

The same series reach ClickStack through the scraper as nats_varz_* gauges. The bootstrap script builds tiles for JetStream storage ratio, connections and slow consumers from them.

Terminal window
kubectl get pods -n nats -l app.kubernetes.io/name=nats
kubectl describe pod nats-0 -n nats
kubectl logs nats-0 -n nats
Terminal window
kubectl exec -it nats-0 -n nats -- nats server report connections
kubectl exec -it nats-0 -n nats -- nslookup nats-headless.nats.svc.cluster.local
Terminal window
kubectl exec -it deploy/nats-box -n nats -- nats server check jetstream
kubectl get pvc -n nats
Terminal window
kubectl get externalsecret nats-credentials -n nats
kubectl get secret nats-credentials -n nats -o jsonpath='{.data}' | jq -r 'keys'
kubectl exec -it nats-0 -n nats -- cat /etc/nats-config/nats.conf | grep -A5 resolver
Terminal window
kubectl get jobs -n nats -l argocd.argoproj.io/hook=PostSync
kubectl logs -n nats job/nats-account-push
kubectl get pods -n nats -l job-name=nats-account-push
Error Cause Fix
NATS not ready after 4 minutes The server pods did not start Read the pod logs and events
JetStream not ready after 2 minutes JetStream did not initialize Make sure that the PVCs are bound
Failed to import operator JWT The operator JWT in Vault is invalid Write a valid operator JWT to Vault
Failed to import account JWT The account JWT is not signed by the current operator Re-sign the account
system account ... not found The SYS account did not import Make sure that SYS_ACCOUNT_JWT is set in the job
Authorization Violation The push user has no credentials Make sure that the SYS seed imports and push-admin is created
unknown flag: --creds Wrong nsc push flag Use --system-user
unknown flag: --file on a key import Wrong nsc import keys flag Use --dir with a directory that holds the key
set an operator No operator context Run nsc env -o fzymgc-house first
scheme "tls" is not supported Wrong URL scheme for nsc Use nats:// and pass --ca-cert

The nsc and nats CLIs differ: nsc takes nats:// with --ca-cert, and nats takes tls:// with --tlsca.

To run the job again:

Terminal window
kubectl delete job nats-account-push -n nats
argocd app sync nats --force
Terminal window
kubectl logs -n nats job/nats-stream-dns-events
Error Cause Fix
Cannot connect to the DNS account after 1 minute The resolver does not hold the DNS account Read the log of nats-account-push. Make sure that dns_account_jwt is in nats-credentials
JetStream is not enabled for the DNS account The account JWT has no JetStream limits (err_code 10039) Run nsc edit account DNS --js-disk-storage 1G, write the new JWT to Vault, and sync
insufficient storage resources from stream add The account disk limit is smaller than 3 x max_bytes Raise --js-disk-storage on the account, or lower max_bytes
Terminal window
kubectl get certificate -n nats
kubectl get secret nats-client-tls -n nats
kubectl exec -it deploy/nats-box -n nats -- nats server info --tlsca /etc/nats-certs/ca.crt
  1. Make sure that the PVCs exist: kubectl get pvc -n nats.
  2. Delete the StatefulSet and keep the pods and PVCs: kubectl delete sts nats -n nats --cascade=orphan. ArgoCD recreates it.
  3. Make sure that the cluster forms again with nats server report jetstream.

Restore from a Velero backup if one covers the namespace, or recreate the streams from their definitions. A client with a durable consumer resumes from its last acknowledgement.

  • The operator JWT is the root of trust. Never commit a seed or a JWT to git.
  • Each trust level gets its own account.
  • Authentication failures are visible in the server logs. Read them after a rotation.