NATS Operations
NATS is the message broker for the cluster. This page states what exists, where each part is
defined, and the procedures that touch it. Commands assume --context fzymgc-house.
What exists
Section titled “What exists”| Property | Value |
|---|---|
| Namespace | nats |
| Client address | nats.nats.svc.cluster.local:4222, TLS with the nats-client-tls certificate |
| MQTT listener | nats.nats.svc:1883, TLS with handshake_first: true |
| Monitoring port | 8222, no TLS |
| Metrics port | 7777, the Prometheus exporter |
| Chart | nats from https://nats-io.github.io/k8s/helm/charts/, pinned in argocd/cluster-app/templates/nats.yaml |
| Values | argocd/app-configs/nats/values.yaml |
| Cluster | 3 replicas, routes over TLS with the nats-cluster-tls certificate |
| JetStream | file store 10Gi per node on longhorn-encrypted, memory store 256Mi, domain fzymgc |
| Vault path | secret/fzymgc-house/cluster/nats, read by the nats-credentials ExternalSecret |
| Streams in git | DNS_EVENTS in the account DNS, for the events of dns-control. See The stream DNS_EVENTS |
| Telemetry | The monitoring-otel-scraper collector scrapes port 7777 on each of the three pods. The ClickStack tiles are in argocd/app-configs/clickstack-alerts/bootstrap-script.yaml |
Both certificates come from the fzymgc-house-issuer ClusterIssuer
(argocd/app-configs/nats/certificate.yaml), last one year and renew 30 days before expiry.
Accounts
Section titled “Accounts”NATS runs in operator mode. The operator is fzymgc-house. Four accounts exist:
| Account | Purpose | JetStream | How it reaches the server |
|---|---|---|---|
SYS |
System monitoring and the push user | Off | Preloaded through resolver_preload in values.yaml, because the server needs it at start |
SERVICES |
Cluster services | Off. The account JWT has no JetStream limits | Pushed by the PostSync job |
IOT |
The MQTT bridge and IoT devices | On | Pushed by the PostSync job |
DNS |
dns-control events on the stream DNS_EVENTS |
On. 1 GB of disk storage, 64 MB of memory storage | Pushed by the PostSync job |
An account is the NATS boundary for subjects and for the JetStream quota. A service that needs
JetStream gets its own account, as dns-control does with DNS.
The resolver is type: full with dir: /data/jwt and allow_delete: false. The PostSync job
nats-account-push (argocd/app-configs/nats/account-push-job.yaml) waits for the server and
JetStream. It imports the operator and the SYS account with its seed, and it creates the
push-admin user under SYS. It then pushes SERVICES, IOT and DNS with
nsc push --system-user push-admin. The job runs after every sync and deletes itself on success.
values.yaml carries the SYS account public key twice, as system_account and as the
resolver_preload key. Both must equal sys_account_public in Vault.
Vault keys
Section titled “Vault keys”vault kv get secret/fzymgc-house/cluster/nats| Key | Description |
|---|---|
operator_jwt |
Operator JWT |
operator_public |
Operator public key |
operator_seed |
Operator seed. It signs the account JWTs. Only the nsc procedures read it, and the ExternalSecret does not map it |
sys_account_seed, sys_account_public, sys_account_jwt |
The SYS account |
services_account_seed, services_account_public, services_account_jwt |
The SERVICES account |
iot_account_seed, iot_account_public, iot_account_jwt |
The IOT account |
dns_account_seed, dns_account_public, dns_account_jwt |
The DNS account |
The ExternalSecret also maps bridge_username and bridge_password from
secret/fzymgc-house/cluster/mosquitto into the same Kubernetes Secret.
Client access
Section titled “Client access”No NetworkPolicy in the repository restricts NATS clients. The mosquitto namespace carries the
label nats-client: "true", and nothing reads it. A client needs the CA chain for
fzymgc-house-issuer and credentials for its account.
nats-box runs in the namespace. It mounts every key of nats-credentials at
/etc/nats-nkeys/default. Its default context gives services_account_seed to --nkey, and
that context does not work. The file holds an account seed, not a user seed, so the nats CLI
stops with Not a valid nkey user seed.
To run a nats command, sign a user that expires after one hour, and connect with its
credentials file. The pod has no copy of the CA, so copy it in first. nsc import account
accepts only a file that ends in .jwt, so the script copies the account JWT under that name.
Replace DNS and dns_account_* with the account that you need:
kubectl get secret nats-client-tls -n nats -o jsonpath='{.data.ca\.crt}' | base64 -d \ | kubectl exec -i deploy/nats-box -n nats -- sh -c 'cat > /tmp/ca.crt'kubectl exec -i deploy/nats-box -n nats -- sh -s <<'EOF'set -euumask 077K=/etc/nats-nkeys/defaultW=$(mktemp -d)export NKEYS_PATH="$W/nkeys" XDG_DATA_HOME="$W/data" XDG_CONFIG_HOME="$W/config"mkdir -p "$W/keys"cp "$K/dns_account_seed" "$W/keys/account.nk"cp "$K/dns_account_jwt" "$W/dns.jwt"nsc add operator -u "$K/operator_jwt"nsc import account --file "$W/dns.jwt"nsc import keys --dir "$W/keys"nsc add user --account DNS --name admin --expiry 1hnsc generate creds --account DNS --name admin > /tmp/admin.credsrm -rf "$W"EOFThe nats commands on this page leave out the connection flags. Add them after nats:
kubectl exec -it deploy/nats-box -n nats -- nats \ --server tls://nats.nats.svc.cluster.local:4222 --tlsca /tmp/ca.crt --creds /tmp/admin.creds \ stream lsThe nats server commands need a user in the SYS account.
The MQTT bridge
Section titled “The MQTT bridge”Mosquitto (argocd/app-configs/mosquitto/) bridges to the NATS MQTT listener. The bridge
configuration is rendered by the mosquitto-secrets ExternalSecret:
| Property | Value |
|---|---|
| Bridge address | nats.nats.svc:1883 |
| CA bundle | /mosquitto/nats-ca/ca.crt from the fzymgc-ica1-ca ConfigMap |
| Credentials | bridge_username and bridge_password from secret/fzymgc-house/cluster/mosquitto |
| Topic mapping | topic # both 0 mqtt/: every Mosquitto topic under mqtt/ maps to the same NATS subject without the prefix, in both directions |
The NATS MQTT listener runs in operator mode, so the bridge password is a bearer user JWT from the
IOT account, and the username is any non-empty string. The listener sets handshake_first: true, so a client must start TLS at once and cannot use STARTTLS.
The bridge address must not carry a trailing dot. Go’s TLS stack rejects a trailing dot in the
SNI name, and the bridge then fails with tlsv1 alert decode error.
| Error | Cause | Fix |
|---|---|---|
tlsv1 alert decode error |
Trailing dot in the bridge address | Write nats.nats.svc |
Connection Refused: not authorised |
Invalid or expired bearer JWT | Generate a new JWT (see Create the bridge credential) |
JetStream not enabled for account |
The IOT account JWT lacks JetStream |
Enable it with nsc edit account IOT --js-enable and push again |
unacceptable protocol version |
A race on the first connect | Transient. The bridge retries |
Procedures
Section titled “Procedures”Every procedure that signs keys needs the nsc CLI and a temporary nsc environment loaded
from Vault:
brew install nats-io/nats-tools/nscexport VAULT_ADDR=https://vault.fzymgc.housevault login -method=oidcexport NKEYS_PATH=$(mktemp -d)export NSC_HOME=$(mktemp -d)vault kv get -field=operator_jwt secret/fzymgc-house/cluster/nats > "$NSC_HOME/operator.jwt"nsc add operator -u "$NSC_HOME/operator.jwt"# Only for a procedure that signs an account JWT:mkdir -m 700 "$NSC_HOME/opkeys"vault kv get -field=operator_seed secret/fzymgc-house/cluster/nats > "$NSC_HOME/opkeys/operator.nk"nsc import keys --dir "$NSC_HOME/opkeys"Remove both directories when you are done. They hold private keys.
The Namespace CLI also installs a binary named nsc. If nsc --version does not print
nsc version 2.x, run the NATS nsc from the natsio/nats-box:0.20.0 image with both
directories mounted.
Read the current keys
Section titled “Read the current keys”vault kv get -field=operator_public secret/fzymgc-house/cluster/natsfor acct in sys services iot dnsdo echo "$acct: $(vault kv get -field=${acct}_account_public secret/fzymgc-house/cluster/nats)"donekubectl exec -it deploy/nats-box -n nats -- nats account list # the accounts the resolver holdsCreate the bridge credential
Section titled “Create the bridge credential”nsc add user --account IOT --name mqtt-bridge --bearer --conn-type MQTTnsc generate creds --account IOT --name mqtt-bridge | grep -A1 'BEGIN.*JWT'Store the JWT in Vault at secret/fzymgc-house/cluster/mosquitto as bridge_password. The
mosquitto-secrets ExternalSecret picks it up at its next refresh.
Add an account
Section titled “Add an account”-
Create the account and write its keys to files. If the account needs JetStream, set its limits in the same step:
Terminal window nsc add account NEW_ACCOUNTnsc edit account NEW_ACCOUNT --js-disk-storage 1G --js-mem-storage 64MOUT=$(mktemp -d)nsc describe account NEW_ACCOUNT --raw | tr -d '\n' > "$OUT/jwt"nsc describe account NEW_ACCOUNT --json | jq -r .sub | tr -d '\n' > "$OUT/public"tr -d '\n' < "$(find "$NKEYS_PATH" -name "$(cat "$OUT/public").nk")" > "$OUT/seed"A stream with
num_replicas3 reserves three times itsmax_bytesof the account disk limit. For example, a 256 MiB stream reserves 768 MiB, so theDNSaccount has 1 GB. -
Make sure that the keys are not in Vault yet. Then add the three values from the files. Always use
patch. Aputreplaces every key on the path:Terminal window vault kv get -format=json secret/fzymgc-house/cluster/nats | jq '.data.data | keys'vault kv patch secret/fzymgc-house/cluster/nats \new_account_seed=@"$OUT/seed" \new_account_public=@"$OUT/public" \new_account_jwt=@"$OUT/jwt"rm -rf "$OUT" -
Add the JWT key to
argocd/app-configs/nats/external-secret.yaml:data:- secretKey: new_account_jwtremoteRef:key: fzymgc-house/cluster/natsproperty: new_account_jwt -
Add the account to
argocd/app-configs/nats/account-push-job.yaml, as an environment variable and as an import and push step in the script, in the shape of theSERVICESblock:env:- name: NEW_ACCOUNT_JWTvalueFrom:secretKeyRef:name: nats-credentialskey: new_account_jwtTerminal window echo "$NEW_ACCOUNT_JWT" > /tmp/new_account.jwtnsc import account --file /tmp/new_account.jwtnsc push -a NEW_ACCOUNT -u "$NATS_URL" --ca-cert "$CA_FILE" --system-user push-admin -
Open a PR. After the merge, ArgoCD syncs and the PostSync job pushes the account.
-
Make sure that the resolver holds it:
Terminal window kubectl exec -it deploy/nats-box -n nats -- nats account info NEW_ACCOUNT
Create user credentials
Section titled “Create user credentials”A user authenticates with a .creds file that holds its JWT and its seed:
nsc add user -a SERVICES myservicensc generate creds -a SERVICES -n myservice > myservice.credsCreate the dns-control user
Section titled “Create the dns-control user”dns-control publishes its events as the user dns-control in the account DNS. The user can
publish on dns.> and subscribe on _INBOX.> only. Its credentials file is the property
nats_creds of secret/fzymgc-house/cluster/dns-control. dns-control reads the file at
/etc/dns-control/nats/dns-control.creds, the default of its key nats_creds_file.
-
Load the
nscenvironment of Procedures. The operator seed is not needed for a user. -
Import the
DNSaccount and its seed:Terminal window vault kv get -field=dns_account_jwt secret/fzymgc-house/cluster/nats > "$NSC_HOME/dns.jwt"mkdir -m 700 "$NSC_HOME/dnskeys"vault kv get -field=dns_account_seed secret/fzymgc-house/cluster/nats > "$NSC_HOME/dnskeys/account.nk"nsc import account --file "$NSC_HOME/dns.jwt"nsc import keys --dir "$NSC_HOME/dnskeys" -
Add the user and write its credentials file:
Terminal window nsc add user -a DNS dns-control --allow-pub 'dns.>' --allow-sub '_INBOX.>'(umask 077 && nsc generate creds -a DNS -n dns-control > "$NSC_HOME/dns-control.creds") -
Read the property names at the path, then write the file to the property
nats_creds. The path holds other properties, so always usepatch. Aputreplaces every property:Terminal window vault kv get -mount=secret -format=json fzymgc-house/cluster/dns-control | jq '.data.data | keys'vault kv patch -mount=secret fzymgc-house/cluster/dns-control nats_creds=@"$NSC_HOME/dns-control.creds" -
Remove both directories:
Terminal window rm -rf "$NKEYS_PATH" "$NSC_HOME"
No command in this procedure prints the credentials. The user needs no $JS.API.> permission,
because dns-control only publishes. If dns-control calls another JetStream API later, for example a
stream lookup, the user also needs $JS.API.STREAM.INFO.DNS_EVENTS.
Rotate the dns-control user
Section titled “Rotate the dns-control user”Rotate the user when its seed leaks, for example into a log or a transcript. The user JWT has no
expiry, so a new credentials file does not stop the old one from working. The DNS account JWT
must revoke the public key of the old user. The operator signs the account JWT, so this procedure
needs the operator seed.
Reloader restarts dns-control when the Secret dns-control changes, and the new pod reads the new
file. Write the new credentials before you revoke the old key. If you revoke first, the server
disconnects dns-control before the new file reaches the pod.
-
Load the
nscenvironment of Procedures, with the operator seed. -
Import the
DNSaccount and its seed, as in step 2 of Create the dns-control user. -
Read the public key of the current user from Vault. The function prints the
subclaim of the user JWT and does not print the seed:Terminal window user_key() {awk '/BEGIN NATS USER JWT/ { getline; print; exit }' \| jq -rR 'split(".")[1] | gsub("-"; "+") | gsub("_"; "/")| . + ("=" * ((4 - length % 4) % 4)) | @base64d | fromjson | .sub'}OLD=$(vault kv get -mount=secret -field=nats_creds fzymgc-house/cluster/dns-control | user_key)echo "$OLD" -
Add the new user and write its credentials file, as in step 3 of Create the dns-control user. Then read its public key:
Terminal window NEW=$(user_key < "$NSC_HOME/dns-control.creds")echo "$NEW" -
Write the file to Vault, as in step 4 of Create the dns-control user.
-
Refresh the ExternalSecret, or wait for its 5-minute interval:
Terminal window kubectl annotate externalsecret dns-control -n dns-control force-sync=$(date +%s) --overwrite -
Make sure that dns-control connects with the new key. One server prints
$NEW, and the other two print nothing:Terminal window for p in nats-0 nats-1 nats-2dokubectl exec -n nats "$p" -c nats -- wget -qO- 'http://localhost:8222/connz?auth=true&limit=256' \| jq -r '.connections[]? | select(.name == "dns-control") | .authorized_user'done -
Revoke the old key and export the account JWT:
Terminal window nsc revocations add-user -a DNS -u "$OLD" --at 0nsc describe account DNS --raw | tr -d '\n' > "$NSC_HOME/dns-account.jwt"nscprintsunable to push, because the operator has no account server URL. The push happens in step 11. -
Write the account JWT to Vault. The path holds other properties, so use
patch:Terminal window vault kv patch secret/fzymgc-house/cluster/nats dns_account_jwt=@"$NSC_HOME/dns-account.jwt" -
Refresh the ExternalSecret of NATS, or wait for its 15-minute interval:
Terminal window kubectl annotate externalsecret nats-credentials -n nats force-sync=$(date +%s) --overwrite -
Push the
DNSaccount with Push an account by hand. -
Make sure that each server holds the revocation. Each line prints
$OLDwith a timestamp:Terminal window ACCOUNT=$(vault kv get -field=dns_account_public secret/fzymgc-house/cluster/nats)for p in nats-0 nats-1 nats-2dokubectl exec -n nats "$p" -c nats -- wget -qO- "http://localhost:8222/accountz?acc=$ACCOUNT" \| jq -c '.account_detail.decoded_jwt.nats.revocations'done -
Remove both directories:
Terminal window rm -rf "$NKEYS_PATH" "$NSC_HOME"
Rotate an account JWT
Section titled “Rotate an account JWT”-
Edit and export the account:
Terminal window nsc edit account SERVICESnsc describe account SERVICES --raw > services.jwt -
Write the new JWT to Vault:
Terminal window vault kv patch secret/fzymgc-house/cluster/nats services_account_jwt="$(cat services.jwt)" -
Refresh the ExternalSecret, or wait for its 15-minute interval:
Terminal window kubectl annotate externalsecret nats-credentials -n nats force-sync=$(date +%s) --overwrite -
Sync the application, so that the PostSync job pushes the new JWT:
Terminal window argocd app sync nats --forceIf the sync result lists no
nats-account-pushhook, use Push an account by hand. -
Make sure that the resolver holds the new JWT:
Terminal window kubectl exec -it deploy/nats-box -n nats -- nats account info SERVICES
Push an account by hand
Section titled “Push an account by hand”The PostSync job nats-account-push pushes every account after a sync. A self-heal sync of
nats-credentials can replace that sync before the job runs. The self-heal sync is partial and
runs no hooks, so the job never starts and the resolver keeps the old JWT. The controller log then
shows skipHooks=true for a sync of the ExternalSecret alone.
These steps run the commands of the job in nats-box, which mounts the same Secret. The pushed JWT
is the one in Vault, so the next PostSync job pushes the same JWT.
-
Make sure that
nats-boxholds the new JWT. Replacednswith the account that you push. The two hashes must be equal:Terminal window vault kv get -field=dns_account_jwt secret/fzymgc-house/cluster/nats | tr -d '\n' | shasum -a 256kubectl exec -n nats deploy/nats-box -- sh -c \'tr -d "\n" < /etc/nats-nkeys/default/dns_account_jwt | sha256sum' -
Copy the CA into the pod, and push the account as a
SYSuser that expires after 15 minutes. ReplaceDNSanddns_account_jwtwith the account that you push:Terminal window kubectl get secret nats-client-tls -n nats -o jsonpath='{.data.ca\.crt}' | base64 -d \| kubectl exec -i deploy/nats-box -n nats -- sh -c 'cat > /tmp/ca.crt'kubectl exec -i deploy/nats-box -n nats -- sh -s <<'EOF'set -euumask 077K=/etc/nats-nkeys/defaultW=$(mktemp -d)export NKEYS_PATH="$W/nkeys" XDG_DATA_HOME="$W/data" XDG_CONFIG_HOME="$W/config"mkdir -p "$W/keys"cp "$K/sys_account_seed" "$W/keys/sys_account.nk"cp "$K/sys_account_jwt" "$W/sys.jwt"cp "$K/dns_account_jwt" "$W/account.jwt"nsc add operator -u "$K/operator_jwt"nsc import account --file "$W/sys.jwt"nsc import keys --dir "$W/keys"nsc add user --account SYS --name push-admin --expiry 15mnsc import account --file "$W/account.jwt"nsc push -a DNS -u nats://nats.nats.svc.cluster.local:4222 --ca-cert /tmp/ca.crt --system-user push-adminrm -rf "$W" /tmp/ca.crtEOFThe push prints
jwt updatedonce for each of the three servers.
Rotate the operator key
Section titled “Rotate the operator key”Every account JWT is signed by the operator, so an operator rotation re-signs all of them and every client credential.
- Generate the new operator key pair and JWT.
- Re-sign every account JWT with the new operator.
- Write
operator_jwt,operator_publicand every*_account_jwtto Vault. - Sync the application. NATS restarts with the new operator, and the PostSync job pushes the re-signed accounts.
- Issue new credentials to every client.
If the SYS account gets new keys as well, update system_account and the resolver_preload
key in values.yaml in the same change.
Check the cluster
Section titled “Check the cluster”kubectl exec -it deploy/nats-box -n nats -- nats server checkkubectl exec -it deploy/nats-box -n nats -- nats server report jetstreamThe JetStream report lists the three servers in cluster nats and domain fzymgc, with an API
error count of zero. The starred server is the meta leader.
Streams and consumers
Section titled “Streams and consumers”A stream that a service depends on is declared in git. The stream DNS_EVENTS in the DNS
account is the example:
| Property | Value |
|---|---|
| Configuration | The ConfigMap dns-events-stream, key dns-events.json, in argocd/app-configs/nats/stream-dns-events.yaml |
| Subjects | dns.> |
| Limits | max_bytes 256 MiB, max_age 7 days, max_msg_size 64 KiB, discard: old, 3 replicas |
| Declared by | The PostSync job nats-stream-dns-events, in sync wave 1 after nats-account-push |
The job signs a user in the DNS account that expires after one hour. That user can only call
the JetStream API. The job adds the stream when it is absent. When the stream exists, the job
edits it to the ConfigMap, so a manual change goes back at the next sync. A second run prints
No difference in configuration. To change the stream, change the ConfigMap in a PR.
The stream DNS_EVENTS
Section titled “The stream DNS_EVENTS”dns-control publishes the events of its operations actions on DNS_EVENTS. The stream carries
events only. The record of each action is the audit store of dns-control.
| Property | Value |
|---|---|
| Subjects in use | dns.ops.<action>.<outcome>, for example dns.ops.flush.succeeded, and dns.ops.drain.overdue |
| Message id | The header Nats-Msg-Id: the audit id of the action, or drain-overdue:<node>:<drain start in unix seconds> |
| Duplicate window | 2 minutes. The stream drops a repeat of a message id inside it |
| Storage | File, 3 replicas. The stream reserves 768 MiB of the 1 GB disk limit of the DNS account |
| Publisher | The user dns-control. See Create the dns-control user |
Read the stream with the connection flags of Client access:
kubectl exec -it deploy/nats-box -n nats -- nats \ --server tls://nats.nats.svc.cluster.local:4222 --tlsca /tmp/ca.crt --creds /tmp/admin.creds \ stream info DNS_EVENTSThe output shows the subjects dns.>, the limits of the ConfigMap, and the message count and
size. Read the newest messages with stream view DNS_EVENTS in place of stream info DNS_EVENTS.
The events and their payload are in the dns-control runbook,
Operations actions.
The other commands for a stream, with a stream EVENTS as the example:
kubectl exec -it deploy/nats-box -n nats -- nats stream lskubectl exec -it deploy/nats-box -n nats -- nats stream add EVENTS \ --subjects="events.>" --storage=file --replicas=3 --retention=limits \ --max-msgs=-1 --max-bytes=1073741824 --max-age=168h --discard=oldkubectl exec -it deploy/nats-box -n nats -- nats stream info EVENTSkubectl exec -it deploy/nats-box -n nats -- nats consumer ls EVENTSkubectl exec -it deploy/nats-box -n nats -- nats stream backup EVENTS /path/to/backupkubectl exec -it deploy/nats-box -n nats -- nats stream restore EVENTS /path/to/backupA publish and subscribe test, in two terminals:
kubectl exec -it deploy/nats-box -n nats -- nats sub "test.>"kubectl exec -it deploy/nats-box -n nats -- nats pub test.hello "Hello NATS"Read the metrics
Section titled “Read the metrics”kubectl port-forward svc/nats -n nats 7777:7777curl localhost:7777/metricsThe same series reach ClickStack through the scraper as nats_varz_* gauges. The bootstrap
script builds tiles for JetStream storage ratio, connections and slow consumers from them.
Troubleshooting
Section titled “Troubleshooting”A pod does not start
Section titled “A pod does not start”kubectl get pods -n nats -l app.kubernetes.io/name=natskubectl describe pod nats-0 -n natskubectl logs nats-0 -n natsThe cluster does not form
Section titled “The cluster does not form”kubectl exec -it nats-0 -n nats -- nats server report connectionskubectl exec -it nats-0 -n nats -- nslookup nats-headless.nats.svc.cluster.localJetStream is not available
Section titled “JetStream is not available”kubectl exec -it deploy/nats-box -n nats -- nats server check jetstreamkubectl get pvc -n natsAuthentication fails
Section titled “Authentication fails”kubectl get externalsecret nats-credentials -n natskubectl get secret nats-credentials -n nats -o jsonpath='{.data}' | jq -r 'keys'kubectl exec -it nats-0 -n nats -- cat /etc/nats-config/nats.conf | grep -A5 resolverThe account push job fails
Section titled “The account push job fails”kubectl get jobs -n nats -l argocd.argoproj.io/hook=PostSynckubectl logs -n nats job/nats-account-pushkubectl get pods -n nats -l job-name=nats-account-push| Error | Cause | Fix |
|---|---|---|
NATS not ready after 4 minutes |
The server pods did not start | Read the pod logs and events |
JetStream not ready after 2 minutes |
JetStream did not initialize | Make sure that the PVCs are bound |
Failed to import operator JWT |
The operator JWT in Vault is invalid | Write a valid operator JWT to Vault |
Failed to import account JWT |
The account JWT is not signed by the current operator | Re-sign the account |
system account ... not found |
The SYS account did not import |
Make sure that SYS_ACCOUNT_JWT is set in the job |
Authorization Violation |
The push user has no credentials | Make sure that the SYS seed imports and push-admin is created |
unknown flag: --creds |
Wrong nsc push flag |
Use --system-user |
unknown flag: --file on a key import |
Wrong nsc import keys flag |
Use --dir with a directory that holds the key |
set an operator |
No operator context | Run nsc env -o fzymgc-house first |
scheme "tls" is not supported |
Wrong URL scheme for nsc |
Use nats:// and pass --ca-cert |
The nsc and nats CLIs differ: nsc takes nats:// with --ca-cert, and nats takes
tls:// with --tlsca.
To run the job again:
kubectl delete job nats-account-push -n natsargocd app sync nats --forceThe stream job fails
Section titled “The stream job fails”kubectl logs -n nats job/nats-stream-dns-events| Error | Cause | Fix |
|---|---|---|
Cannot connect to the DNS account after 1 minute |
The resolver does not hold the DNS account |
Read the log of nats-account-push. Make sure that dns_account_jwt is in nats-credentials |
JetStream is not enabled for the DNS account |
The account JWT has no JetStream limits (err_code 10039) |
Run nsc edit account DNS --js-disk-storage 1G, write the new JWT to Vault, and sync |
insufficient storage resources from stream add |
The account disk limit is smaller than 3 x max_bytes |
Raise --js-disk-storage on the account, or lower max_bytes |
TLS fails
Section titled “TLS fails”kubectl get certificate -n natskubectl get secret nats-client-tls -n natskubectl exec -it deploy/nats-box -n nats -- nats server info --tlsca /etc/nats-certs/ca.crtRecovery
Section titled “Recovery”Rebuild the cluster on its data
Section titled “Rebuild the cluster on its data”- Make sure that the PVCs exist:
kubectl get pvc -n nats. - Delete the StatefulSet and keep the pods and PVCs:
kubectl delete sts nats -n nats --cascade=orphan. ArgoCD recreates it. - Make sure that the cluster forms again with
nats server report jetstream.
Recover lost JetStream data
Section titled “Recover lost JetStream data”Restore from a Velero backup if one covers the namespace, or recreate the streams from their definitions. A client with a durable consumer resumes from its last acknowledgement.
Constraints
Section titled “Constraints”- The operator JWT is the root of trust. Never commit a seed or a JWT to git.
- Each trust level gets its own account.
- Authentication failures are visible in the server logs. Read them after a rotation.