Tailscale Deployment Design

Issue: #514 Date: 2025-12-31 Status: Approved

Summary

Deploy Tailscale to the cluster as a subnet router and exit node, enabling secure remote access to the home network (192.168.20.0/22) from any device on the existing tailnet.

Goals

1. Remote admin access — SSH to nodes, kubectl from anywhere on the tailnet
2. Service exposure — Access internal services directly via Tailscale (bypassing Cloudflare)
3. Exit node — Route device traffic through home network when traveling
4. Reliable DNS — Ensure consistent route to Firewalla DNS (192.168.20.1) for MagicDNS

Approach

Tailscale Operator with Connector CRD — The operator manages node lifecycle, and the Connector resource defines the subnet router with HA enabled.

Why This Approach

• Official Kubernetes-native solution
• Built-in HA with automatic failover (no manual replica management)
• OAuth-based auth eliminates key expiration issues
• Clean CRD-based config fits GitOps workflow

Prerequisites

Tailscale Admin Console

1. Create OAuth client at Settings → OAuth clients with scopes:
   • devices:write
   • routes:write
2. Note the Client ID and Client Secret

Tailscale ACL (Optional but Recommended)

Add autoApprovers to avoid manual route approval:

"autoApprovers": {
  "routes": {
    "192.168.20.0/22": ["tag:k8s-operator"]
  },
  "exitNode": ["tag:k8s-operator"]
}

Vault Secret

Store OAuth credentials at secret/tailscale/operator:

clientId: "k..."
clientSecret: "tskey-client-..."

Vault Policy

Add read access for External Secrets Operator:

path "secret/data/tailscale/*" {
  capabilities = ["read"]
}

Architecture

┌─────────────────────────────────────────────────────────────┐
│                     Tailscale Control Plane                 │
└─────────────────────────────────────────────────────────────┘
                              │
                              │ OAuth + WireGuard
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                    tailscale namespace                      │
│  ┌──────────────────┐    ┌──────────────────────────────┐  │
│  │ tailscale-       │    │ Connector: home-network      │  │
│  │ operator         │───▶│  - advertises 192.168.20.0/22│  │
│  │                  │    │  - exit node enabled         │  │
│  └──────────────────┘    │  - HA: 2 replicas            │  │
│                          └──────────────────────────────────┘  │
│                                      │                      │
│                          ┌───────────┴───────────┐         │
│                          ▼                       ▼         │
│                 ┌─────────────────┐    ┌─────────────────┐ │
│                 │k8s-subnet-      │    │k8s-subnet-      │ │
│                 │router-0         │    │router-1         │ │
│                 │(primary)        │    │(standby)        │ │
│                 └─────────────────┘    └─────────────────┘ │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                   Home Network 192.168.20.0/22              │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐            │
│  │ Firewalla  │  │ k8s nodes  │  │ Other      │            │
│  │ DNS .20.1  │  │ .20.101+   │  │ devices    │            │
│  └────────────┘  └────────────┘  └────────────┘            │
└─────────────────────────────────────────────────────────────┘

Implementation

Directory Structure

argocd/app-configs/tailscale/
├── namespace.yaml
├── external-secret.yaml
├── connector.yaml
└── values.yaml

Namespace

apiVersion: v1
kind: Namespace
metadata:
  name: tailscale
  labels:
    app.kubernetes.io/name: tailscale

ExternalSecret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: tailscale-oauth
  namespace: tailscale
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: tailscale-oauth
    creationPolicy: Owner
  data:
    - secretKey: oauth-client-id
      remoteRef:
        key: secret/tailscale/operator
        property: clientId
    - secretKey: oauth-client-secret
      remoteRef:
        key: secret/tailscale/operator
        property: clientSecret

Helm Values (values.yaml)

operatorConfig:
  oauthSecretName: tailscale-oauth
  defaultTags:
    - "tag:k8s-operator"

resources:
  limits:
    memory: 256Mi
  requests:
    cpu: 50m
    memory: 128Mi

Connector

apiVersion: tailscale.com/v1alpha1
kind: Connector
metadata:
  name: home-network
  namespace: tailscale
spec:
  hostname: k8s-subnet-router
  subnetRouter:
    advertiseRoutes:
      - "192.168.20.0/22"
    highAvailability: true
  exitNode: true
  tags:
    - "tag:k8s-operator"

ArgoCD Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: tailscale
  namespace: argocd
spec:
  project: default
  sources:
    - repoURL: https://pkgs.tailscale.com/helmcharts
      chart: tailscale-operator
      targetRevision: "1.82.0"
      helm:
        valueFiles:
          - $values/argocd/app-configs/tailscale/values.yaml
    - repoURL: https://github.com/fzymgc-house/selfhosted-cluster
      targetRevision: HEAD
      ref: values
    - repoURL: https://github.com/fzymgc-house/selfhosted-cluster
      targetRevision: HEAD
      path: argocd/app-configs/tailscale
  destination:
    server: https://kubernetes.default.svc
    namespace: tailscale
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Verification

Cluster Checks

# Operator running
kubectl get pods -n tailscale -l app.kubernetes.io/name=tailscale-operator

# Connector status
kubectl get connector -n tailscale

# Proxy pods (subnet router replicas)
kubectl get pods -n tailscale -l tailscale.com/parent-resource=home-network

Tailscale Admin Console

• Verify k8s-subnet-router-0 and k8s-subnet-router-1 appear in Machines
• Confirm routes are approved (automatic if using autoApprovers)
• Confirm exit node capability is enabled

Client Testing

# Check route availability
tailscale status

# Test connectivity to cluster node
ping 192.168.20.101

# Test DNS via Firewalla
dig @192.168.20.1 some-local-hostname

# Test exit node
tailscale up --exit-node=k8s-subnet-router-0
curl ifconfig.me  # Should show home IP

Security Considerations

• OAuth credentials stored in Vault, never in Git
• Nodes tagged k8s-operator for ACL targeting
• Routes restricted to home network CIDR only
• Exit node optional per-client (user chooses when to enable)

Future Enhancements

• Add Tailscale Funnel for specific services (alternative to Cloudflare Tunnel)
• Explore app-specific ingress via Tailscale Ingress class
• Consider MagicDNS split-DNS for cluster service discovery