System Overview
The fzymgc-house cluster is a k3s cluster on two Turing Pi 2 boards. This page states how the pieces fit together and where each one is defined. The services catalog lists every deployed service, and the network reference lists every address.
High-level architecture
Section titled “High-level architecture”graph TB
subgraph Internet["Internet"]
Users["Users"]
CF["Cloudflare"]
end
subgraph Hardware["Hardware"]
TPI_A["Turing Pi alpha"]
TPI_B["Turing Pi beta"]
end
subgraph Kubernetes["Kubernetes (k3s)"]
CP["Control plane<br/>tpi-alpha-1, 2, 3"]
Workers["Workers<br/>tpi-alpha-4, tpi-beta-1 to 4"]
end
subgraph Platform["Platform services"]
Vault["Vault<br/>secrets"]
Keycloak["Keycloak<br/>SSO and OIDC"]
Traefik["Traefik<br/>ingress"]
ArgoCD["ArgoCD<br/>GitOps"]
end
subgraph Apps["Applications"]
LiteLLM["LiteLLM and the MCP servers"]
ClickStack["ClickStack<br/>logs, metrics, traces"]
Other["Other applications"]
end
Users --> CF --> Traefik
Traefik --> Keycloak
Traefik --> Apps
TPI_A --> CP
TPI_B --> Workers
CP --> Platform
Workers --> Apps
Vault --> Platform
Vault --> Apps
ArgoCD --> Apps
Three layers
Section titled “Three layers”| Layer | Location | Tool | What it holds |
|---|---|---|---|
| 1. Cluster deployment | ansible/ |
Ansible | Node configuration, k3s, the CNI, the resolver pair, heimdall, the NAS |
| 2. Infrastructure configuration | tf/ |
Terraform on HCP Terraform | Vault, Keycloak, Cloudflare, DNS records, Tailscale, UniFi, LiteLLM keys, Uptime Kuma |
| 3. Application deployment | argocd/ |
ArgoCD | Every Kubernetes application, as Helm charts, Kustomize overlays or plain manifests |
graph LR
subgraph L1["Layer 1: Ansible"]
A1["Node configuration"]
A2["k3s"]
A3["Cilium"]
end
subgraph L2["Layer 2: Terraform"]
T1["Vault"]
T2["Keycloak"]
T3["Cloudflare and DNS"]
end
subgraph L3["Layer 3: ArgoCD"]
K1["Helm charts"]
K2["Kustomize"]
K3["Plain manifests"]
end
A1 --> A2 --> A3 --> T1 --> T2 --> T3 --> K1 & K2 & K3
Layer 1: Ansible
Section titled “Layer 1: Ansible”The roles under ansible/roles/ configure the physical nodes and install k3s. The roles that
matter for the cluster itself:
| Role | Purpose |
|---|---|
tp2-bootstrap-node |
Node operating system, network legs, chrony, mail relay |
k3s-server |
Control plane nodes |
k3s-agent |
Worker nodes |
cilium |
The CNI, installed through Helm |
longhorn-disks |
Extra disks for Longhorn |
ansible/CLAUDE.md lists every role, including the resolver, heimdall and NAS roles.
Layer 2: Terraform
Section titled “Layer 2: Terraform”Each directory under tf/ is one HCP Terraform workspace. A merge to main that touches a
workspace’s directory queues a run there. The HCP Terraform runbook
lists every workspace and what it manages. The cluster-bootstrap workspace runs locally and
installs the charts ArgoCD depends on: ArgoCD itself, cert-manager, External Secrets and
Longhorn.
Layer 3: ArgoCD
Section titled “Layer 3: ArgoCD”argocd/cluster-app/templates/ holds one Application per service. Each Application points at a
directory under argocd/app-configs/, which holds a Helm chart pin with values, a Kustomize
overlay, or plain manifests. ArgoCD syncs the cluster to main.
Hardware
Section titled “Hardware”Two Turing Pi 2 boards carry eight Turing RK1 compute modules. The alpha board holds the three control plane nodes and one worker. The beta board holds four workers.
graph TB
subgraph Alpha["Turing Pi alpha"]
A1["tpi-alpha-1<br/>control plane"]
A2["tpi-alpha-2<br/>control plane"]
A3["tpi-alpha-3<br/>control plane"]
A4["tpi-alpha-4<br/>worker"]
end
subgraph Beta["Turing Pi beta"]
B1["tpi-beta-1<br/>worker"]
B2["tpi-beta-2<br/>worker"]
B3["tpi-beta-3<br/>worker"]
B4["tpi-beta-4<br/>worker"]
end
VIP["API VIP over BGP<br/>192.168.250.254"]
A1 & A2 & A3 --> VIP
| Component | Value |
|---|---|
| Compute module | Turing RK1 (Rockchip RK3588, 8 cores, arm64) |
| Memory | 32 GB on each alpha node, 16 GB on each beta node |
| Storage | One NVMe drive per node |
| Network | One 1 GbE port per node, end0, trunked with the Lab VLAN |
| Operating system | Armbian |
Read the live values with kubectl --context fzymgc-house get nodes -o wide.
High availability
Section titled “High availability”| Component | Strategy |
|---|---|
| API server | The three servers announce 192.168.250.254 and fddb:f665:73f7:250::254 over BGP through Cilium, and the gateway routes each connection to one of them |
| etcd | Embedded in k3s on the three control plane nodes |
| Storage | Longhorn replicates volumes across nodes |
| LoadBalancer VIPs | Each node with a ready endpoint announces the VIP over BGP, and the gateway spreads flows across those nodes |
GitOps flow
Section titled “GitOps flow”sequenceDiagram
participant Dev as Developer
participant GH as GitHub
participant HCP as HCP Terraform
participant Argo as ArgoCD
participant K8s as Kubernetes
Dev->>GH: Push a feature branch and open a PR
GH->>HCP: Speculative plan for each touched workspace
HCP-->>GH: Plan result on the PR
Dev->>GH: Merge the PR
GH->>HCP: Run for each touched workspace
HCP->>K8s: Apply
Argo->>GH: Poll main
Argo->>K8s: Sync applications
| Change under | What runs |
|---|---|
tf/<workspace>/ |
An HCP Terraform run, applied by the workspace’s rules |
argocd/ |
An ArgoCD sync |
ansible/ |
A playbook run by the operator |
Secrets
Section titled “Secrets”sequenceDiagram
participant Vault as Vault
participant ESO as External Secrets Operator
participant K8s as Kubernetes Secret
participant Pod as Pod
ESO->>Vault: Authenticate with Kubernetes auth
Vault-->>ESO: Secret data
ESO->>K8s: Create or update the Secret
K8s-->>Pod: Mount as env or volume
No secret is committed to Git. Vault holds two subtrees under secret/fzymgc-house/:
| Subtree | Readers |
|---|---|
cluster/ |
Cluster workloads, through ExternalSecrets |
infrastructure/ |
Terraform, Ansible and scripts |
The secrets reference lists every path and its reader. A new path needs a
Vault policy update in tf/vault.
Authentication
Section titled “Authentication”Keycloak at id.fzymgc.house is the identity provider. Four methods exist:
| Method | Used by |
|---|---|
| OIDC | Web applications with native OIDC, kubectl through kubelogin, Vault, ArgoCD |
Forward auth through oauth2-proxy |
Dashboards without native SSO: Temporal, Hubble UI, the Traefik dashboard, Longhorn |
| Client certificates from the Vault PKI | The external OTel collectors and other machine clients |
| Cloudflare Access | Public hostnames under fzymgc.net that the tunnel serves |
sequenceDiagram
participant User
participant Traefik
participant Proxy as oauth2-proxy
participant Keycloak
participant App as Application
User->>Traefik: Request a protected dashboard
Traefik->>Proxy: Forward auth check
Proxy-->>User: Redirect to Keycloak
User->>Keycloak: Log in
Keycloak-->>Proxy: OIDC token
Proxy-->>Traefik: Allow
Traefik->>App: Forward the request
Keycloak is OIDC only. The forward-auth path goes through a shared oauth2-proxy, which
an ADR records.
Kubernetes RBAC binds the Keycloak groups k8s-admins, k8s-developers and k8s-viewers to
the ClusterRoles cluster-admin, edit and view (argocd/app-configs/k8s-oidc-rbac/).
Network
Section titled “Network”graph TB
subgraph External["External"]
Internet["Internet"]
CF["Cloudflare tunnel"]
end
subgraph Routed["Routed VIPs, announced over BGP"]
VIPs["LoadBalancer VIPs<br/>192.168.250.0/24, fddb:f665:73f7:250::/64"]
Traefik["Traefik"]
end
subgraph Cluster["Cluster network"]
Cilium["Cilium"]
Pods["Pods<br/>10.245.0.0/16, fd00:10:245::/48"]
Svc["Services<br/>10.43.0.0/16, fd00:10:43::/112"]
end
Internet --> CF --> Traefik
VIPs --> Traefik
Traefik --> Cilium --> Pods
Pods <--> Svc
| Network | CIDR | Purpose |
|---|---|---|
| Node addresses | 192.168.20.0/24, inside Main 192.168.20.0/22 |
The nodes’ primary interface |
| Node Lab legs | 192.168.200.141-144, .151-154 and their ULAs |
The tagged leg each node carries for BGP and for the replies of the VIPs |
| Pods | 10.245.0.0/16, fd00:10:245::/48 |
Cilium cluster-pool IPAM |
| Services | 10.43.0.0/16, fd00:10:43::/112 |
Kubernetes Services, dual-stack |
| Routed VIPs | 192.168.250.0/24, fddb:f665:73f7:250::/64 |
LoadBalancer VIPs from Cilium LB-IPAM, announced over BGP |
| API server VIP | 192.168.250.254, fddb:f665:73f7:250::254 |
Cilium BGP, pool api-vip |
A client on Main reaches a VIP through the UCG-Fiber gateway, so the path is routed and not on-link. The Lab service VLAN runbook describes the node legs.
Security
Section titled “Security”| Domain | Solution |
|---|---|
| Identity | Keycloak, OIDC |
| Secrets | Vault, delivered by External Secrets |
| Network policy | Cilium, as CiliumNetworkPolicy objects per namespace |
| Ingress | Traefik terminates TLS with certificates from cert-manager |
| Public exposure | Cloudflare tunnel and Cloudflare Access, no inbound port |
| Audit | ClickStack holds logs, metrics and traces from every service and host |
Design principles
Section titled “Design principles”GitOps first. Git holds the desired state, ArgoCD and HCP Terraform apply it, and Git history is the audit trail. A change that is not in Git is drift.
Secrets in Vault. Workloads read secrets through External Secrets with Kubernetes auth, and Vault logs every read. Terraform, Ansible and scripts read through their own auth methods.
SSO through Keycloak. One identity for every service. A service without native OIDC sits behind
oauth2-proxy.
Observability by default. The OTel collectors on the nodes, the resolvers and the NAS send to ClickStack. HyperDX serves the dashboards and evaluates the alerts, and Pushover delivers them.
Key design decisions
Section titled “Key design decisions”k3s over upstream Kubernetes. One binary with embedded etcd fits eight arm64 boards and keeps upgrades to a single System Upgrade Controller plan.
Vault over sealed secrets. Vault gives one store for the cluster, Terraform, Ansible and the external hosts, with policies, audit and rotation in one place.
Keycloak as the identity provider. An ADR records the choice.
ArgoCD over Flux. The application dashboard and the app-of-apps pattern fit one operator running many small services.
Related documentation
Section titled “Related documentation”| Topic | Link |
|---|---|
| Network details | Network reference |
| Secrets | Secrets reference |
| Services | Services catalog |
| Technology stack | Technologies reference |
| Decisions | Architecture decision records |