Skip to content
fzymgc-house Cluster Docs

NATS JetStream Design

Date: 2026-01-04 Status: Approved Issue: #497

Summary

Section titled “Summary”

Deploy NATS with JetStream as a general messaging backbone for the cluster, with IoT/Home Assistant integration as a future consumer.

Goals

Section titled “Goals”
  1. Messaging foundation — Central pub/sub and persistence layer for cluster services
  2. High availability — 3-node cluster with JetStream replication
  3. Security — NKey authentication, TLS encryption, Vault-managed credentials
  4. Observability — Prometheus metrics, Grafana dashboard, Loki logs

Architecture

Section titled “Architecture”
┌─────────────────────────────────────────────────────────────┐
│                    NATS 3-Node Cluster                      │
│  ┌─────────┐    ┌─────────┐    ┌─────────┐                 │
│  │ nats-0  │◄──►│ nats-1  │◄──►│ nats-2  │  (Raft consensus)│
│  │         │    │         │    │         │                  │
│  │JetStream│    │JetStream│    │JetStream│                  │
│  └────┬────┘    └────┬────┘    └────┬────┘                  │
│       │              │              │                       │
│  ┌────┴────┐    ┌────┴────┐    ┌────┴────┐                 │
│  │Longhorn │    │Longhorn │    │Longhorn │  (encrypted)     │
│  │  PVC    │    │  PVC    │    │  PVC    │                  │
│  └─────────┘    └─────────┘    └─────────┘                  │
└─────────────────────────────────────────────────────────────┘
          │
          ▼
┌─────────────────────┐     ┌─────────────────────┐
│  nats.nats.svc      │     │  Vault              │
│  (ClusterIP)        │     │  - NKey seeds       │
│  Port 4222 (client) │     │  - Account JWTs     │
│  Port 6222 (cluster)│     │  - Server config    │
│  Port 7777 (metrics)│     │                     │
└─────────────────────┘     └─────────────────────┘

Components

Section titled “Components”
Component Choice
Namespace nats
Helm chart nats/nats v2.12.3 (latest stable)
Replicas 3 (HA cluster)
Storage 10GB longhorn-encrypted per node
Auth NKey with NATS Accounts
TLS cert-manager, cluster-ca-issuer (internal CA)
Secrets Vault + ExternalSecret

Authentication & Security

Section titled “Authentication & Security”

NATS Account Structure

Section titled “NATS Account Structure”
OPERATOR: fzymgc-house
└── ACCOUNT: SYS (system account - monitoring)
└── ACCOUNT: SERVICES (internal cluster services)
└── ACCOUNT: IOT (future: Home Assistant, MQTT bridge)

Vault Secret Structure

Section titled “Vault Secret Structure”

Single secret with properties (matches repository conventions):

secret/fzymgc-house/cluster/nats
├── operator_seed           # Operator NKey seed (private)
├── operator_public         # Operator public key
├── sys_account_seed        # SYS account NKey seed
├── sys_account_jwt         # SYS account signed JWT
├── services_account_seed   # SERVICES account NKey seed
├── services_account_jwt    # SERVICES account signed JWT
├── iot_account_seed        # IOT account NKey seed
├── iot_account_jwt         # IOT account signed JWT
└── resolver_config         # Account resolver config

Vault Policy

Section titled “Vault Policy”
# Policy: nats-server
path "secret/data/fzymgc-house/cluster/nats" {
  capabilities = ["read"]
}
path "secret/metadata/fzymgc-house/cluster/nats" {
  capabilities = ["read"]
}

Initial NKey Setup

Section titled “Initial NKey Setup”

Generate credentials using the nsc CLI (one-time setup):

Terminal window
# Install nsc
brew install nats-io/nats-tools/nsc  # or go install github.com/nats-io/nsc/v2@latest


# Create operator and accounts
nsc init --name fzymgc-house
nsc add account SYS
nsc add account SERVICES
nsc add account IOT


# Export credentials for Vault
nsc describe operator -J > /tmp/operator.jwt
nsc describe account SYS -J > /tmp/sys.jwt
# ... repeat for other accounts


# Store in Vault (manual or via tf/vault module)
vault kv put secret/fzymgc-house/cluster/nats \
  operator_seed="$(nsc keys --operator --private)" \
  operator_public="$(nsc keys --operator)" \
  sys_account_seed="$(nsc keys --account SYS --private)" \
  sys_account_jwt="$(cat /tmp/sys.jwt)" \
  # ... additional properties

Note: Operator seed is root of trust. Storing in Vault is acceptable for automation but consider security implications for production environments.

Network Policies

Section titled “Network Policies”
  • Default deny ingress to nats namespace
  • Allow from namespaces with label nats-client: "true"
  • Allow cluster-internal traffic between NATS pods

Initial client namespaces: None (no consumers yet per YAGNI)

network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: nats-ingress
  namespace: nats
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: nats
  policyTypes:
    - Ingress
  ingress:
    # Allow NATS cluster traffic (pod-to-pod)
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: nats
      ports:
        - port: 6222
    # Allow clients from labeled namespaces
    - from:
        - namespaceSelector:
            matchLabels:
              nats-client: "true"
      ports:
        - port: 4222
    # Allow Prometheus scraping
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - port: 7777

Kubernetes Resources

Section titled “Kubernetes Resources”

ArgoCD Application Structure

Section titled “ArgoCD Application Structure”

Application manifest: argocd/cluster-app/templates/nats.yaml (References the app-config below, follows app-of-apps pattern)

App configuration:

argocd/app-configs/nats/
├── kustomization.yaml
├── namespace.yaml
├── helm-release.yaml         # Helm chart reference
├── values.yaml               # Helm values
├── external-secret.yaml      # NKeys from Vault
├── certificate.yaml          # TLS cert via cert-manager
├── network-policy.yaml       # Ingress rules
└── prometheus-rules.yaml     # Alert rules

Helm Values

Section titled “Helm Values”
cluster:
  enabled: true
  replicas: 3


jetstream:
  enabled: true
  domain: fzymgc
  memoryStore:
    maxSize: 256Mi
  fileStore:
    pvc:
      size: 10Gi
      storageClassName: longhorn-encrypted


natsBox:
  enabled: true  # Debugging/admin pod


config:
  cluster:
    tls:
      enabled: true

Resource Requirements

Section titled “Resource Requirements”
Resource Per Pod Total (3 pods)
CPU request 100m 300m
CPU limit 500m 1500m
Memory request 256Mi 768Mi
Memory limit 512Mi 1536Mi
Storage 10Gi 30Gi

Observability

Section titled “Observability”

Metrics

Section titled “Metrics”
  • NATS exposes Prometheus metrics on port 7777
  • ServiceMonitor scrapes /metrics endpoint
  • Key metrics: connections, messages in/out, JetStream stream/consumer stats

Grafana Dashboard

Section titled “Grafana Dashboard”
  • Use official NATS dashboard (ID: 14605) as base
  • Deploy as ConfigMap to argocd/app-configs/grafana/dashboards/nats.yaml
  • Dashboard will be auto-imported via Grafana sidecar

Alert Rules

Section titled “Alert Rules”

Deploy as PrometheusRule CRD in argocd/app-configs/nats/prometheus-rules.yaml:

Critical:

  • JetStream storage > 80% capacity
  • Cluster lost quorum (< 2 nodes healthy)
  • No leader elected for > 30s

Warning:

  • Message delivery failures
  • Slow consumers (pending > threshold)
  • Connection errors spike

Logging

Section titled “Logging”
  • NATS logs collected by Alloy (existing infrastructure)
  • Label: app=nats

Not Included (YAGNI)

Section titled “Not Included (YAGNI)”
  • MQTT bridge — Add when Home Assistant deploys (#496)
  • External ingress — Internal-only for now
  • Leafnodes — Single cluster sufficient
  • WebSocket gateway — No browser clients yet

Execution Order

Section titled “Execution Order”
  1. Verify storage class: kubectl get sc longhorn-encrypted
  2. Generate NKey credentials using nsc CLI (see Initial NKey Setup)
  3. Create Vault secrets (operator, accounts, server config)
  4. Create Vault policy and Kubernetes auth role
  5. Update docs/reference/secrets.md with NATS secret path
  6. Deploy ArgoCD application
  7. Verify cluster health and JetStream (see Verification table)
  8. Deploy Grafana dashboard ConfigMap
  9. Deploy PrometheusRule for alerts
  10. Update docs/reference/services.md with NATS entry

Verification

Section titled “Verification”
Test Command/Action Expected
Storage class kubectl get sc longhorn-encrypted StorageClass exists
Cluster healthy nats-box: nats server check All nodes connected
Cluster quorum nats-box: nats server report jetstream 3/3 nodes, leader elected
JetStream enabled nats stream ls Empty list, no errors
TLS working nats server info --tls TLS connection succeeds
Auth working Connect without creds Connection refused
Account list nats-box: nats account info Shows configured account
Metrics exposed curl :7777/metrics Prometheus metrics
ServiceMonitor Check Prometheus targets NATS target is Up
Persistence Create stream, delete pod, verify Stream retained after pod restart

Rollback

Section titled “Rollback”
  1. Delete ArgoCD application
  2. PVCs retained by default (manual cleanup if needed)
  3. Vault secrets retained (no automatic cleanup)

References

Section titled “References”