NATS JetStream Design

Date: 2026-01-04 Status: Approved Issue: #497

Summary

Deploy NATS with JetStream as a general messaging backbone for the cluster, with IoT/Home Assistant integration as a future consumer.

Goals

Messaging foundation — Central pub/sub and persistence layer for cluster services
High availability — 3-node cluster with JetStream replication
Security — NKey authentication, TLS encryption, Vault-managed credentials
Observability — Prometheus metrics, Grafana dashboard, Loki logs

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    NATS 3-Node Cluster                      │
│  ┌─────────┐    ┌─────────┐    ┌─────────┐                 │
│  │ nats-0  │◄──►│ nats-1  │◄──►│ nats-2  │  (Raft consensus)│
│  │         │    │         │    │         │                  │
│  │JetStream│    │JetStream│    │JetStream│                  │
│  └────┬────┘    └────┬────┘    └────┬────┘                  │
│       │              │              │                       │
│  ┌────┴────┐    ┌────┴────┐    ┌────┴────┐                 │
│  │Longhorn │    │Longhorn │    │Longhorn │  (encrypted)     │
│  │  PVC    │    │  PVC    │    │  PVC    │                  │
│  └─────────┘    └─────────┘    └─────────┘                  │
└─────────────────────────────────────────────────────────────┘
          │
          ▼
┌─────────────────────┐     ┌─────────────────────┐
│  nats.nats.svc      │     │  Vault              │
│  (ClusterIP)        │     │  - NKey seeds       │
│  Port 4222 (client) │     │  - Account JWTs     │
│  Port 6222 (cluster)│     │  - Server config    │
│  Port 7777 (metrics)│     │                     │
└─────────────────────┘     └─────────────────────┘

Components

Component	Choice
Namespace	`nats`
Helm chart	`nats/nats` v2.12.3 (latest stable)
Replicas	3 (HA cluster)
Storage	10GB `longhorn-encrypted` per node
Auth	NKey with NATS Accounts
TLS	cert-manager, `cluster-ca-issuer` (internal CA)
Secrets	Vault + ExternalSecret

Authentication & Security

NATS Account Structure

OPERATOR: fzymgc-house
└── ACCOUNT: SYS (system account - monitoring)
└── ACCOUNT: SERVICES (internal cluster services)
└── ACCOUNT: IOT (future: Home Assistant, MQTT bridge)

Vault Secret Structure

Single secret with properties (matches repository conventions):

secret/fzymgc-house/cluster/nats
├── operator_seed           # Operator NKey seed (private)
├── operator_public         # Operator public key
├── sys_account_seed        # SYS account NKey seed
├── sys_account_jwt         # SYS account signed JWT
├── services_account_seed   # SERVICES account NKey seed
├── services_account_jwt    # SERVICES account signed JWT
├── iot_account_seed        # IOT account NKey seed
├── iot_account_jwt         # IOT account signed JWT
└── resolver_config         # Account resolver config

Vault Policy

# Policy: nats-server
path "secret/data/fzymgc-house/cluster/nats" {
  capabilities = ["read"]
}
path "secret/metadata/fzymgc-house/cluster/nats" {
  capabilities = ["read"]
}

Initial NKey Setup

Generate credentials using the nsc CLI (one-time setup):

# Install nsc
brew install nats-io/nats-tools/nsc  # or go install github.com/nats-io/nsc/v2@latest

# Create operator and accounts
nsc init --name fzymgc-house
nsc add account SYS
nsc add account SERVICES
nsc add account IOT

# Export credentials for Vault
nsc describe operator -J > /tmp/operator.jwt
nsc describe account SYS -J > /tmp/sys.jwt
# ... repeat for other accounts

# Store in Vault (manual or via tf/vault module)
vault kv put secret/fzymgc-house/cluster/nats \
  operator_seed="$(nsc keys --operator --private)" \
  operator_public="$(nsc keys --operator)" \
  sys_account_seed="$(nsc keys --account SYS --private)" \
  sys_account_jwt="$(cat /tmp/sys.jwt)" \
  # ... additional properties

Note: Operator seed is root of trust. Storing in Vault is acceptable for automation but consider security implications for production environments.

Network Policies

Default deny ingress to nats namespace
Allow from namespaces with label nats-client: "true"
Allow cluster-internal traffic between NATS pods

Initial client namespaces: None (no consumers yet per YAGNI)

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: nats-ingress
  namespace: nats
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: nats
  policyTypes:
    - Ingress
  ingress:
    # Allow NATS cluster traffic (pod-to-pod)
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: nats
      ports:
        - port: 6222
    # Allow clients from labeled namespaces
    - from:
        - namespaceSelector:
            matchLabels:
              nats-client: "true"
      ports:
        - port: 4222
    # Allow Prometheus scraping
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - port: 7777

Kubernetes Resources

ArgoCD Application Structure

Application manifest: argocd/cluster-app/templates/nats.yaml (References the app-config below, follows app-of-apps pattern)

App configuration:

argocd/app-configs/nats/
├── kustomization.yaml
├── namespace.yaml
├── helm-release.yaml         # Helm chart reference
├── values.yaml               # Helm values
├── external-secret.yaml      # NKeys from Vault
├── certificate.yaml          # TLS cert via cert-manager
├── network-policy.yaml       # Ingress rules
└── prometheus-rules.yaml     # Alert rules

Helm Values

cluster:
  enabled: true
  replicas: 3

jetstream:
  enabled: true
  domain: fzymgc
  memoryStore:
    maxSize: 256Mi
  fileStore:
    pvc:
      size: 10Gi
      storageClassName: longhorn-encrypted

natsBox:
  enabled: true  # Debugging/admin pod

config:
  cluster:
    tls:
      enabled: true

Resource Requirements

Resource	Per Pod	Total (3 pods)
CPU request	100m	300m
CPU limit	500m	1500m
Memory request	256Mi	768Mi
Memory limit	512Mi	1536Mi
Storage	10Gi	30Gi

Observability

Metrics

NATS exposes Prometheus metrics on port 7777
ServiceMonitor scrapes /metrics endpoint
Key metrics: connections, messages in/out, JetStream stream/consumer stats

Grafana Dashboard

Use official NATS dashboard (ID: 14605) as base
Deploy as ConfigMap to argocd/app-configs/grafana/dashboards/nats.yaml
Dashboard will be auto-imported via Grafana sidecar

Alert Rules

Deploy as PrometheusRule CRD in argocd/app-configs/nats/prometheus-rules.yaml:

Critical:

JetStream storage > 80% capacity
Cluster lost quorum (< 2 nodes healthy)
No leader elected for > 30s

Warning:

Message delivery failures
Slow consumers (pending > threshold)
Connection errors spike

Logging

NATS logs collected by Alloy (existing infrastructure)
Label: app=nats

Not Included (YAGNI)

MQTT bridge — Add when Home Assistant deploys (#496)
External ingress — Internal-only for now
Leafnodes — Single cluster sufficient
WebSocket gateway — No browser clients yet

Execution Order

Verify storage class: kubectl get sc longhorn-encrypted
Generate NKey credentials using nsc CLI (see Initial NKey Setup)
Create Vault secrets (operator, accounts, server config)
Create Vault policy and Kubernetes auth role
Update docs/reference/secrets.md with NATS secret path
Deploy ArgoCD application
Verify cluster health and JetStream (see Verification table)
Deploy Grafana dashboard ConfigMap
Deploy PrometheusRule for alerts
Update docs/reference/services.md with NATS entry

Verification

Test	Command/Action	Expected
Storage class	`kubectl get sc longhorn-encrypted`	StorageClass exists
Cluster healthy	`nats-box: nats server check`	All nodes connected
Cluster quorum	`nats-box: nats server report jetstream`	3/3 nodes, leader elected
JetStream enabled	`nats stream ls`	Empty list, no errors
TLS working	`nats server info --tls`	TLS connection succeeds
Auth working	Connect without creds	Connection refused
Account list	`nats-box: nats account info`	Shows configured account
Metrics exposed	`curl :7777/metrics`	Prometheus metrics
ServiceMonitor	Check Prometheus targets	NATS target is Up
Persistence	Create stream, delete pod, verify	Stream retained after pod restart

Rollback

Delete ArgoCD application
PVCs retained by default (manual cleanup if needed)
Vault secrets retained (no automatic cleanup)

References

NATS Documentation: https://docs.nats.io/
NATS Helm Chart: https://github.com/nats-io/k8s/tree/main/helm/charts/nats
JetStream: https://docs.nats.io/nats-concepts/jetstream
NKey Authentication: https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth