Merlin Operations

Overview

Merlin is a personal AI assistant (powered by OpenClaw) deployed as a StatefulSet in the merlin namespace. It uses Tailscale for secure access and Vault for secrets management.

Access

• Gateway: merlin.<tailnet>.ts.net:18789
• Bridge: merlin.<tailnet>.ts.net:18790

Common Operations

View Logs

kubectl logs -n merlin -l app=merlin -f

Restart Pod

kubectl delete pod -n merlin -l app=merlin

Check ExternalSecret Status

kubectl describe externalsecret -n merlin merlin-secrets

Rotate Secrets

1. Update secrets in Vault (use patch to update specific keys without overwriting others):

vault kv patch secret/fzymgc-house/cluster/merlin \
  gateway-token="<new-token>"

1. Wait 5 minutes for ExternalSecrets to sync, or force refresh:

kubectl annotate externalsecret -n merlin merlin-secrets force-sync=$(date +%s) --overwrite

1. Restart the pod:

kubectl rollout restart statefulset/merlin -n merlin

Backup and Restore

Backups are handled automatically by Velero. To restore:

velero restore create --from-backup <backup-name> --include-namespaces merlin

Troubleshooting

Pod not starting

1. Check events: kubectl describe pod -n merlin -l app=merlin
2. Check PVC binding: kubectl get pvc -n merlin
3. Check ExternalSecret: kubectl describe externalsecret -n merlin

Tailscale not connecting

1. Verify device in admin console: https://login.tailscale.com/admin/machines
2. Check operator logs: kubectl logs -n tailscale -l app.kubernetes.io/name=operator

Workshop Pod

The workshop pod is a persistent Ubuntu 24.04 arm64 development environment in the merlin namespace. It provides a workspace for code reviews, PR creation, builds, and development tasks.

Access

kubectl exec -it -n merlin deployment/merlin-workshop -- bash

Tools

Tools are installed via Homebrew (persisted on a dedicated PVC) and include anything installed via brew install. The Claude Code CLI is installed via its native installer.

To install additional tools:

kubectl exec -it -n merlin deployment/merlin-workshop -- bash -c 'eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)" && brew install <package>'

Storage

PVC	Size	Mount	Purpose
workshop-workspace	100Gi	/home/workshop	Home directory, project files
workshop-homebrew	10Gi	/home/linuxbrew	Homebrew installation and packages

Init Containers

The workshop pod uses an init container to install Homebrew and Claude Code CLI. Installation is skipped if tools are already present on the PVC, so normal restarts are fast.

Gateway Init Containers

The gateway StatefulSet includes two init containers:

1. clear-locks — removes all .lock files from /home/node/.openclaw recursively to prevent session issues after unclean restarts (all locks are stale at init time since no process is running yet)
2. setup-homebrew — installs Homebrew and core tools (kubectl, gh, vault, terraform, Claude Code CLI) to a dedicated 5Gi PVC. Skips if already present.

Gateway Storage

PVC	Size	Mount	Purpose
config	5Gi	/home/node/.openclaw	Gateway config and session data
workspace	20Gi	/home/node/openclaw	Workspace files
gateway-homebrew	5Gi	/home/linuxbrew	Homebrew installation and tools

Network Policies

The merlin namespace uses Calico NetworkPolicy (projectcalico.org/v3) exclusively — not standard Kubernetes NetworkPolicy. This is required because standard ipBlock rules don't work for ClusterIP services with Calico (it evaluates policy before kube-proxy DNAT). Calico's service-based rules resolve endpoints automatically.

See: Calico service rules documentation

Ingress

Policy	Selector	Allowed
merlin-gateway-ingress	app == 'merlin'	TCP 18789 (gateway), TCP 18790 (bridge)
merlin-workshop-ingress	app == 'merlin' && component == 'workshop'	None (exec-only)

Egress

Policy	Destination	Method
merlin-allow-gateway-internal	merlin-gateway service (merlin ns)	Calico service rule
merlin-allow-k8s-api	kubernetes service (default ns)	Calico service rule
merlin-allow-kube-dns	kube-dns service (kube-system ns)	Calico service rule
merlin-allow-vault	Vault pods, port 8200	Calico selector + namespace selector
merlin-allow-external-https	External port 443 (non-RFC1918)	notNets exclusion
merlin-allow-external-http	External port 80 (non-RFC1918)	notNets exclusion
merlin-allow-external-ssh	External port 22 (non-RFC1918)	notNets exclusion

All other egress is denied, including traffic to other in-cluster services not explicitly listed above.

Why Calico NetworkPolicy?

Standard Kubernetes NetworkPolicy uses ipBlock rules for egress, which don't work with Calico for ClusterIP services — Calico evaluates policy before kube-proxy performs DNAT, so the service VIP never matches. Calico's native NetworkPolicy CRD provides:

• Service rules — reference services by name/namespace, endpoints resolved automatically
• notNets — cleaner RFC1918 exclusion than ipBlock.except
• Selector-based destination — target pods by label across namespaces

Architecture

• StatefulSet: Single replica with stable identity
• Workshop: Deployment with single replica, Recreate strategy
• Storage: longhorn-encrypted (5Gi config, 20Gi workspace, 5Gi+10Gi homebrew)
• Secrets: ExternalSecrets from Vault (5m refresh)
• Network: Tailscale operator, ClusterIP service, Calico network policies