Skip to content

System Overview

The fzymgc-house cluster is a k3s cluster on two Turing Pi 2 boards. This page states how the pieces fit together and where each one is defined. The services catalog lists every deployed service, and the network reference lists every address.

graph TB
    subgraph Internet["Internet"]
        Users["Users"]
        CF["Cloudflare"]
    end

    subgraph Hardware["Hardware"]
        TPI_A["Turing Pi alpha"]
        TPI_B["Turing Pi beta"]
    end

    subgraph Kubernetes["Kubernetes (k3s)"]
        CP["Control plane<br/>tpi-alpha-1, 2, 3"]
        Workers["Workers<br/>tpi-alpha-4, tpi-beta-1 to 4"]
    end

    subgraph Platform["Platform services"]
        Vault["Vault<br/>secrets"]
        Keycloak["Keycloak<br/>SSO and OIDC"]
        Traefik["Traefik<br/>ingress"]
        ArgoCD["ArgoCD<br/>GitOps"]
    end

    subgraph Apps["Applications"]
        LiteLLM["LiteLLM and the MCP servers"]
        ClickStack["ClickStack<br/>logs, metrics, traces"]
        Other["Other applications"]
    end

    Users --> CF --> Traefik
    Traefik --> Keycloak
    Traefik --> Apps
    TPI_A --> CP
    TPI_B --> Workers
    CP --> Platform
    Workers --> Apps
    Vault --> Platform
    Vault --> Apps
    ArgoCD --> Apps
Layer Location Tool What it holds
1. Cluster deployment ansible/ Ansible Node configuration, k3s, the CNI, the resolver pair, heimdall, the NAS
2. Infrastructure configuration tf/ Terraform on HCP Terraform Vault, Keycloak, Cloudflare, DNS records, Tailscale, UniFi, LiteLLM keys, Uptime Kuma
3. Application deployment argocd/ ArgoCD Every Kubernetes application, as Helm charts, Kustomize overlays or plain manifests
graph LR
    subgraph L1["Layer 1: Ansible"]
        A1["Node configuration"]
        A2["k3s"]
        A3["Cilium"]
    end

    subgraph L2["Layer 2: Terraform"]
        T1["Vault"]
        T2["Keycloak"]
        T3["Cloudflare and DNS"]
    end

    subgraph L3["Layer 3: ArgoCD"]
        K1["Helm charts"]
        K2["Kustomize"]
        K3["Plain manifests"]
    end

    A1 --> A2 --> A3 --> T1 --> T2 --> T3 --> K1 & K2 & K3

The roles under ansible/roles/ configure the physical nodes and install k3s. The roles that matter for the cluster itself:

Role Purpose
tp2-bootstrap-node Node operating system, network legs, chrony, mail relay
k3s-server Control plane nodes
k3s-agent Worker nodes
cilium The CNI, installed through Helm
longhorn-disks Extra disks for Longhorn

ansible/CLAUDE.md lists every role, including the resolver, heimdall and NAS roles.

Each directory under tf/ is one HCP Terraform workspace. A merge to main that touches a workspace’s directory queues a run there. The HCP Terraform runbook lists every workspace and what it manages. The cluster-bootstrap workspace runs locally and installs the charts ArgoCD depends on: ArgoCD itself, cert-manager, External Secrets and Longhorn.

argocd/cluster-app/templates/ holds one Application per service. Each Application points at a directory under argocd/app-configs/, which holds a Helm chart pin with values, a Kustomize overlay, or plain manifests. ArgoCD syncs the cluster to main.

Two Turing Pi 2 boards carry eight Turing RK1 compute modules. The alpha board holds the three control plane nodes and one worker. The beta board holds four workers.

graph TB
    subgraph Alpha["Turing Pi alpha"]
        A1["tpi-alpha-1<br/>control plane"]
        A2["tpi-alpha-2<br/>control plane"]
        A3["tpi-alpha-3<br/>control plane"]
        A4["tpi-alpha-4<br/>worker"]
    end

    subgraph Beta["Turing Pi beta"]
        B1["tpi-beta-1<br/>worker"]
        B2["tpi-beta-2<br/>worker"]
        B3["tpi-beta-3<br/>worker"]
        B4["tpi-beta-4<br/>worker"]
    end

    VIP["API VIP over BGP<br/>192.168.250.254"]
    A1 & A2 & A3 --> VIP
Component Value
Compute module Turing RK1 (Rockchip RK3588, 8 cores, arm64)
Memory 32 GB on each alpha node, 16 GB on each beta node
Storage One NVMe drive per node
Network One 1 GbE port per node, end0, trunked with the Lab VLAN
Operating system Armbian

Read the live values with kubectl --context fzymgc-house get nodes -o wide.

Component Strategy
API server The three servers announce 192.168.250.254 and fddb:f665:73f7:250::254 over BGP through Cilium, and the gateway routes each connection to one of them
etcd Embedded in k3s on the three control plane nodes
Storage Longhorn replicates volumes across nodes
LoadBalancer VIPs Each node with a ready endpoint announces the VIP over BGP, and the gateway spreads flows across those nodes
sequenceDiagram
    participant Dev as Developer
    participant GH as GitHub
    participant HCP as HCP Terraform
    participant Argo as ArgoCD
    participant K8s as Kubernetes

    Dev->>GH: Push a feature branch and open a PR
    GH->>HCP: Speculative plan for each touched workspace
    HCP-->>GH: Plan result on the PR
    Dev->>GH: Merge the PR
    GH->>HCP: Run for each touched workspace
    HCP->>K8s: Apply
    Argo->>GH: Poll main
    Argo->>K8s: Sync applications
Change under What runs
tf/<workspace>/ An HCP Terraform run, applied by the workspace’s rules
argocd/ An ArgoCD sync
ansible/ A playbook run by the operator
sequenceDiagram
    participant Vault as Vault
    participant ESO as External Secrets Operator
    participant K8s as Kubernetes Secret
    participant Pod as Pod

    ESO->>Vault: Authenticate with Kubernetes auth
    Vault-->>ESO: Secret data
    ESO->>K8s: Create or update the Secret
    K8s-->>Pod: Mount as env or volume

No secret is committed to Git. Vault holds two subtrees under secret/fzymgc-house/:

Subtree Readers
cluster/ Cluster workloads, through ExternalSecrets
infrastructure/ Terraform, Ansible and scripts

The secrets reference lists every path and its reader. A new path needs a Vault policy update in tf/vault.

Keycloak at id.fzymgc.house is the identity provider. Four methods exist:

Method Used by
OIDC Web applications with native OIDC, kubectl through kubelogin, Vault, ArgoCD
Forward auth through oauth2-proxy Dashboards without native SSO: Temporal, Hubble UI, the Traefik dashboard, Longhorn
Client certificates from the Vault PKI The external OTel collectors and other machine clients
Cloudflare Access Public hostnames under fzymgc.net that the tunnel serves
sequenceDiagram
    participant User
    participant Traefik
    participant Proxy as oauth2-proxy
    participant Keycloak
    participant App as Application

    User->>Traefik: Request a protected dashboard
    Traefik->>Proxy: Forward auth check
    Proxy-->>User: Redirect to Keycloak
    User->>Keycloak: Log in
    Keycloak-->>Proxy: OIDC token
    Proxy-->>Traefik: Allow
    Traefik->>App: Forward the request

Keycloak is OIDC only. The forward-auth path goes through a shared oauth2-proxy, which an ADR records. Kubernetes RBAC binds the Keycloak groups k8s-admins, k8s-developers and k8s-viewers to the ClusterRoles cluster-admin, edit and view (argocd/app-configs/k8s-oidc-rbac/).

graph TB
    subgraph External["External"]
        Internet["Internet"]
        CF["Cloudflare tunnel"]
    end

    subgraph Routed["Routed VIPs, announced over BGP"]
        VIPs["LoadBalancer VIPs<br/>192.168.250.0/24, fddb:f665:73f7:250::/64"]
        Traefik["Traefik"]
    end

    subgraph Cluster["Cluster network"]
        Cilium["Cilium"]
        Pods["Pods<br/>10.245.0.0/16, fd00:10:245::/48"]
        Svc["Services<br/>10.43.0.0/16, fd00:10:43::/112"]
    end

    Internet --> CF --> Traefik
    VIPs --> Traefik
    Traefik --> Cilium --> Pods
    Pods <--> Svc
Network CIDR Purpose
Node addresses 192.168.20.0/24, inside Main 192.168.20.0/22 The nodes’ primary interface
Node Lab legs 192.168.200.141-144, .151-154 and their ULAs The tagged leg each node carries for BGP and for the replies of the VIPs
Pods 10.245.0.0/16, fd00:10:245::/48 Cilium cluster-pool IPAM
Services 10.43.0.0/16, fd00:10:43::/112 Kubernetes Services, dual-stack
Routed VIPs 192.168.250.0/24, fddb:f665:73f7:250::/64 LoadBalancer VIPs from Cilium LB-IPAM, announced over BGP
API server VIP 192.168.250.254, fddb:f665:73f7:250::254 Cilium BGP, pool api-vip

A client on Main reaches a VIP through the UCG-Fiber gateway, so the path is routed and not on-link. The Lab service VLAN runbook describes the node legs.

Domain Solution
Identity Keycloak, OIDC
Secrets Vault, delivered by External Secrets
Network policy Cilium, as CiliumNetworkPolicy objects per namespace
Ingress Traefik terminates TLS with certificates from cert-manager
Public exposure Cloudflare tunnel and Cloudflare Access, no inbound port
Audit ClickStack holds logs, metrics and traces from every service and host

GitOps first. Git holds the desired state, ArgoCD and HCP Terraform apply it, and Git history is the audit trail. A change that is not in Git is drift.

Secrets in Vault. Workloads read secrets through External Secrets with Kubernetes auth, and Vault logs every read. Terraform, Ansible and scripts read through their own auth methods.

SSO through Keycloak. One identity for every service. A service without native OIDC sits behind oauth2-proxy.

Observability by default. The OTel collectors on the nodes, the resolvers and the NAS send to ClickStack. HyperDX serves the dashboards and evaluates the alerts, and Pushover delivers them.

k3s over upstream Kubernetes. One binary with embedded etcd fits eight arm64 boards and keeps upgrades to a single System Upgrade Controller plan.

Vault over sealed secrets. Vault gives one store for the cluster, Terraform, Ansible and the external hosts, with policies, audit and rotation in one place.

Keycloak as the identity provider. An ADR records the choice.

ArgoCD over Flux. The application dashboard and the app-of-apps pattern fit one operator running many small services.

Topic Link
Network details Network reference
Secrets Secrets reference
Services Services catalog
Technology stack Technologies reference
Decisions Architecture decision records