Clawdbot Kubernetes Deployment Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Deploy Clawdbot AI assistant to Kubernetes with Vault secrets, encrypted storage, and Tailscale access.

Architecture: StatefulSet with ExternalSecrets pulling credentials from Vault, PVCs using longhorn-encrypted storage class, and Tailscale operator exposing the gateway service on the tailnet.

Tech Stack: Kubernetes StatefulSet, HashiCorp Vault, ExternalSecrets Operator, Tailscale Operator, Longhorn encrypted storage, ArgoCD GitOps, Terraform for Vault policies.

Epic: selfhosted-cluster-t2a

Design Document: docs/plans/2026-01-26-clawdbot-k8s-deployment-design.md

---

Phase 1: Vault Setup

Task 1: Create Vault Policy

Files:

• Create: tf/vault/policy-clawdbot.tf

Step 1: Write the Terraform policy file

resource "vault_policy" "clawdbot" {
  name   = "clawdbot"
  policy = <<EOT
# Allow clawdbot to read its secrets from Vault
path "secret/data/fzymgc-house/cluster/clawdbot" {
  capabilities = ["read"]
}

# Required for ExternalSecrets to verify secret existence
path "secret/metadata/fzymgc-house/cluster/clawdbot" {
  capabilities = ["read"]
}
EOT
}

Step 2: Validate Terraform syntax

Run: terraform -chdir=tf/vault fmt -check policy-clawdbot.tf Expected: No output (file is formatted correctly)

Step 3: Commit

git add tf/vault/policy-clawdbot.tf
git commit -m "feat(vault): add clawdbot policy for secret access"

---

Task 2: Create Kubernetes Auth Role Binding

Files:

• Create: tf/vault/k8s-clawdbot.tf

Step 1: Write the Terraform auth role file

resource "vault_kubernetes_auth_backend_role" "clawdbot" {
  backend                          = vault_auth_backend.kubernetes.path
  role_name                        = "clawdbot"
  bound_service_account_names      = ["clawdbot"]
  bound_service_account_namespaces = ["clawdbot"]
  audience                         = "https://kubernetes.default.svc.cluster.local"
  token_ttl                        = 3600
  token_policies                   = ["default", vault_policy.clawdbot.name]
}

Step 2: Validate Terraform syntax

Run: terraform -chdir=tf/vault fmt -check k8s-clawdbot.tf Expected: No output (file is formatted correctly)

Step 3: Validate full module

Run: terraform -chdir=tf/vault validate Expected: “Success! The configuration is valid.”

Step 4: Commit

git add tf/vault/k8s-clawdbot.tf
git commit -m "feat(vault): add clawdbot kubernetes auth role binding"

---

Task 3: Populate Vault Secrets (Manual Step)

Files:

• None (Vault CLI operation)

Step 1: Document the secrets that need to be added

The following secrets must be added to Vault before ExternalSecrets can sync:

# Login to Vault
export VAULT_ADDR=https://vault.fzymgc.house
vault login -method=oidc

# Create clawdbot secrets (only 2 required)
vault kv put secret/fzymgc-house/cluster/clawdbot \
  gateway-token="<your-gateway-token>" \
  claude-ai-session-key="<your-claude-ai-key>"

# Verify secrets were created
vault kv get secret/fzymgc-house/cluster/clawdbot

Step 2: Add note to operational runbook

This is a manual step - secrets must be obtained from clawdbot setup and stored in Vault before deployment.

Step 3: Commit documentation note

No commit needed - this is documented in the design doc.

---

Phase 2: Kubernetes Manifests

Task 4: Create Application Directory Structure

Files:

• Create: argocd/app-configs/clawdbot/ (directory)

Step 1: Create directory

Run: mkdir -p argocd/app-configs/clawdbot Expected: Directory created

Step 2: Verify directory exists

Run: ls -la argocd/app-configs/clawdbot Expected: Empty directory listed

---

Task 5: Create Namespace and ServiceAccount

Files:

• Create: argocd/app-configs/clawdbot/namespace.yaml

Step 1: Write the namespace manifest

---
apiVersion: v1
kind: Namespace
metadata:
  name: clawdbot
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: clawdbot
  namespace: clawdbot
automountServiceAccountToken: true

Step 2: Validate YAML syntax

Run: yamllint argocd/app-configs/clawdbot/namespace.yaml Expected: No errors

Step 3: Dry-run apply

Run: kubectl apply --dry-run=client -f argocd/app-configs/clawdbot/namespace.yaml Expected: “namespace/clawdbot created (dry run)” and “serviceaccount/clawdbot created (dry run)”

Step 4: Commit

git add argocd/app-configs/clawdbot/namespace.yaml
git commit -m "feat(clawdbot): add namespace and serviceaccount"

---

Task 6: Create ExternalSecret

Files:

• Create: argocd/app-configs/clawdbot/external-secret.yaml

Step 1: Write the ExternalSecret manifest

---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: clawdbot-secrets
  namespace: clawdbot
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: clawdbot-secrets
    creationPolicy: Owner
  data:
    - secretKey: CLAWDBOT_GATEWAY_TOKEN
      remoteRef:
        key: fzymgc-house/cluster/clawdbot
        property: gateway-token
    - secretKey: CLAUDE_AI_SESSION_KEY
      remoteRef:
        key: fzymgc-house/cluster/clawdbot
        property: claude-ai-session-key

Step 2: Validate YAML syntax

Run: yamllint argocd/app-configs/clawdbot/external-secret.yaml Expected: No errors

Step 3: Dry-run apply

Run: kubectl apply --dry-run=client -f argocd/app-configs/clawdbot/external-secret.yaml Expected: “externalsecret.external-secrets.io/clawdbot-secrets created (dry run)”

Step 4: Commit

git add argocd/app-configs/clawdbot/external-secret.yaml
git commit -m "feat(clawdbot): add externalsecret for vault secrets"

---

Task 7: Create StatefulSet

Files:

• Create: argocd/app-configs/clawdbot/statefulset.yaml

Step 1: Write the StatefulSet manifest

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: clawdbot
  namespace: clawdbot
  labels:
    app: clawdbot
spec:
  serviceName: clawdbot
  replicas: 1
  selector:
    matchLabels:
      app: clawdbot
  template:
    metadata:
      labels:
        app: clawdbot
    spec:
      serviceAccountName: clawdbot
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        runAsNonRoot: true
      containers:
        - name: clawdbot
          image: ghcr.io/clawdbot/clawdbot:2026.1.24-1
          imagePullPolicy: IfNotPresent
          command:
            - node
            - dist/index.js
            - gateway
            - --bind
            - "0.0.0.0"
            - --port
            - "18789"
          ports:
            - name: gateway
              containerPort: 18789
              protocol: TCP
            - name: bridge
              containerPort: 18790
              protocol: TCP
          env:
            - name: HOME
              value: /home/node
            - name: TERM
              value: xterm-256color
          envFrom:
            - secretRef:
                name: clawdbot-secrets
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: 2000m
              memory: 2Gi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
          livenessProbe:
            tcpSocket:
              port: 18789
            initialDelaySeconds: 30
            periodSeconds: 30
          readinessProbe:
            tcpSocket:
              port: 18789
            initialDelaySeconds: 10
            periodSeconds: 10
          volumeMounts:
            - name: config
              mountPath: /home/node/.clawdbot
            - name: workspace
              mountPath: /home/node/clawd
  volumeClaimTemplates:
    - metadata:
        name: config
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: longhorn-encrypted
        resources:
          requests:
            storage: 5Gi
    - metadata:
        name: workspace
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: longhorn-encrypted
        resources:
          requests:
            storage: 20Gi

Step 2: Validate YAML syntax

Run: yamllint argocd/app-configs/clawdbot/statefulset.yaml Expected: No errors (may have warnings about line length which are acceptable)

Step 3: Dry-run apply

Run: kubectl apply --dry-run=client -f argocd/app-configs/clawdbot/statefulset.yaml Expected: “statefulset.apps/clawdbot created (dry run)”

Step 4: Commit

git add argocd/app-configs/clawdbot/statefulset.yaml
git commit -m "feat(clawdbot): add statefulset with encrypted storage"

---

Task 8: Create Service with Tailscale Annotations

Files:

• Create: argocd/app-configs/clawdbot/service.yaml

Step 1: Write the Service manifest

---
apiVersion: v1
kind: Service
metadata:
  name: clawdbot-gateway
  namespace: clawdbot
  annotations:
    tailscale.com/expose: "true"
    tailscale.com/hostname: "clawdbot"
    tailscale.com/tags: "tag:k8s-operator"
spec:
  type: ClusterIP
  selector:
    app: clawdbot
  ports:
    - name: gateway
      port: 18789
      targetPort: 18789
      protocol: TCP
    - name: bridge
      port: 18790
      targetPort: 18790
      protocol: TCP

Step 2: Validate YAML syntax

Run: yamllint argocd/app-configs/clawdbot/service.yaml Expected: No errors

Step 3: Dry-run apply

Run: kubectl apply --dry-run=client -f argocd/app-configs/clawdbot/service.yaml Expected: “service/clawdbot-gateway created (dry run)”

Step 4: Commit

git add argocd/app-configs/clawdbot/service.yaml
git commit -m "feat(clawdbot): add service with tailscale annotations"

---

Task 9: Create Kustomization

Files:

• Create: argocd/app-configs/clawdbot/kustomization.yaml

Step 1: Write the Kustomization manifest

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: clawdbot

resources:
  - namespace.yaml
  - external-secret.yaml
  - statefulset.yaml
  - service.yaml

Step 2: Validate Kustomization

Run: kubectl kustomize argocd/app-configs/clawdbot Expected: Combined YAML output of all resources

Step 3: Commit

git add argocd/app-configs/clawdbot/kustomization.yaml
git commit -m "feat(clawdbot): add kustomization for app config"

---

Phase 3: ArgoCD Integration

Task 10: Create ArgoCD Application

Files:

• Create: argocd/cluster-app/templates/clawdbot.yaml

Step 1: Write the ArgoCD Application manifest

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: clawdbot
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: {{ .Values.spec.source.repoURL }}
    targetRevision: {{ .Values.spec.source.targetRevision }}
    path: argocd/app-configs/clawdbot
  destination:
    server: {{ .Values.spec.destination.server }}
    namespace: clawdbot
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

Step 2: Validate Helm template

Run: helm template argocd/cluster-app --show-only templates/clawdbot.yaml Expected: Rendered Application manifest

Step 3: Commit

git add argocd/cluster-app/templates/clawdbot.yaml
git commit -m "feat(argocd): add clawdbot application definition"

---

Phase 4: Documentation

Task 11: Create Operational Runbook

Files:

• Create: docs/operations/clawdbot.md

Step 1: Write the operational runbook

# Clawdbot Operations

## Overview

Clawdbot is a personal AI assistant deployed as a StatefulSet in the `clawdbot` namespace. It uses Tailscale for secure access and Vault for secrets management.

## Access

- **Gateway**: `clawdbot.<tailnet>.ts.net:18789`
- **Bridge**: `clawdbot.<tailnet>.ts.net:18790`

## Common Operations

### View Logs

```bash
kubectl logs -n clawdbot -l app=clawdbot -f

Restart Pod

kubectl delete pod -n clawdbot -l app=clawdbot

Check ExternalSecret Status

kubectl describe externalsecret -n clawdbot clawdbot-secrets

Rotate Secrets

1. Update secrets in Vault:

   vault kv put secret/fzymgc-house/cluster/clawdbot \
     gateway-token="<new-token>" \
     claude-ai-session-key="<new-key>" \
     claude-web-session-key="<new-key>" \
     claude-web-cookie="<new-cookie>"

2. Wait 15 minutes for ExternalSecrets to sync, or force refresh:

   kubectl annotate externalsecret -n clawdbot clawdbot-secrets force-sync=$(date +%s) --overwrite

3. Restart the pod:

   kubectl rollout restart statefulset/clawdbot -n clawdbot

Backup and Restore

Backups are handled automatically by Velero. To restore:

velero restore create --from-backup <backup-name> --include-namespaces clawdbot

Troubleshooting

Pod not starting

1. Check events: kubectl describe pod -n clawdbot -l app=clawdbot
2. Check PVC binding: kubectl get pvc -n clawdbot
3. Check ExternalSecret: kubectl describe externalsecret -n clawdbot

Tailscale not connecting

1. Verify device in admin console: https://login.tailscale.com/admin/machines
2. Check operator logs: kubectl logs -n tailscale -l app.kubernetes.io/name=operator

Architecture

• StatefulSet: Single replica with stable identity
• Storage: longhorn-encrypted (5Gi config, 20Gi workspace)
• Secrets: ExternalSecrets from Vault (15m refresh)
• Network: Tailscale operator, ClusterIP service

**Step 2: Validate markdown links and formatting**

Run: `uvx --with mdformat mdformat --check docs/operations/clawdbot.md || true`
Expected: File validates or minor formatting suggestions

**Step 3: Commit**

```bash
git add docs/operations/clawdbot.md
git commit -m "docs: add clawdbot operational runbook"

---

Phase 5: Create PR and Deploy

Task 12: Push and Create Pull Request

Files:

• None (git operations)

Step 1: Push branch

Run: git push -u origin feat/clawdbot-deployment Expected: Branch pushed to remote

Step 2: Create Pull Request

Run:

gh pr create \
  --title "feat: deploy clawdbot to kubernetes" \
  --body "$(cat <<'EOF'
## Summary

Deploys Clawdbot AI assistant to the Kubernetes cluster following the approved design document.

### Changes

**Terraform (Vault)**
- `tf/vault/policy-clawdbot.tf` - Vault policy for secret access
- `tf/vault/k8s-clawdbot.tf` - Kubernetes auth role binding

**Kubernetes Manifests**
- `argocd/app-configs/clawdbot/` - Complete application configuration
  - Namespace and ServiceAccount
  - ExternalSecret for Vault integration
  - StatefulSet with longhorn-encrypted storage
  - Service with Tailscale annotations

**ArgoCD**
- `argocd/cluster-app/templates/clawdbot.yaml` - Application definition

**Documentation**
- `docs/operations/clawdbot.md` - Operational runbook

## Test Plan

- [ ] Terraform validation passes
- [ ] Kustomize builds successfully
- [ ] Helm template renders correctly
- [ ] Vault secrets populated (manual prerequisite)
- [ ] ArgoCD sync succeeds
- [ ] Tailscale device appears in admin console
- [ ] Gateway port accessible via Tailscale
- [ ] CLI client connects successfully

## Related

- Design: docs/plans/2026-01-26-clawdbot-k8s-deployment-design.md
- Epic: selfhosted-cluster-t2a

🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

Expected: PR created with URL returned

---

Verification Checklist

After deployment, verify:

• ArgoCD shows clawdbot application as Healthy and Synced
• Pod is Running: kubectl get pods -n clawdbot
• PVCs are Bound: kubectl get pvc -n clawdbot
• ExternalSecret is SecretSynced: kubectl get externalsecret -n clawdbot
• Tailscale device active: Check admin console
• Connectivity: nc -zv clawdbot.<tailnet>.ts.net 18789
• CLI client connects and persists state

---

Rollback Procedure

If issues occur:

1. ArgoCD Rollback:

   argocd app rollback clawdbot

2. Manual Cleanup (if needed):

   kubectl delete sts clawdbot -n clawdbot --cascade=orphan  # Preserve PVCs
   kubectl delete ns clawdbot  # Full cleanup including PVCs

3. Revert PR if Terraform changes cause issues