System Overview

Architectural overview of the fzymgc-house self-hosted Kubernetes cluster.

High-Level Architecture

graph TB
    subgraph Internet["Internet"]
        Users["Users"]
        CF["Cloudflare"]
    end

    subgraph Hardware["Hardware Layer"]
        TPI_A["TuringPi Alpha"]
        TPI_B["TuringPi Beta"]
    end

    subgraph Kubernetes["Kubernetes Layer (k3s)"]
        CP["Control Plane<br/>tpi-alpha-1,2,3"]
        Workers["Workers<br/>tpi-alpha-4, tpi-beta-1-4"]
    end

    subgraph Platform["Platform Services"]
        Vault["Vault<br/>Secrets"]
        Keycloak["Keycloak<br/>SSO/OIDC"]
        Traefik["Traefik<br/>Ingress"]
        ArgoCD["ArgoCD<br/>GitOps"]
    end

    subgraph Apps["Applications"]
        Grafana["Grafana"]
        VM["VictoriaMetrics"]
        Other["Other Apps"]
    end

    Users --> CF --> Traefik
    Traefik --> Keycloak
    Traefik --> Apps
    TPI_A --> CP
    TPI_B --> Workers
    CP --> Platform
    Workers --> Apps
    Vault --> Platform
    Vault --> Apps
    ArgoCD --> Apps

Three-Layer Architecture

Layer	Location	Tool	Purpose
1. Cluster Deployment	`ansible/`	Ansible	Node configuration, k3s installation
2. Infrastructure Config	`tf/`	Terraform	Vault, Keycloak, Grafana setup
3. Application Deployment	`argocd/`	ArgoCD	GitOps-driven app manifests

graph LR
    subgraph L1["Layer 1: Ansible"]
        A1["Node Config"]
        A2["k3s Install"]
        A3["CNI Setup"]
    end

    subgraph L2["Layer 2: Terraform"]
        T1["Vault Config"]
        T2["Keycloak Setup"]
        T3["Grafana Config"]
    end

    subgraph L3["Layer 3: ArgoCD"]
        K1["Helm Charts"]
        K2["Kustomize"]
        K3["Raw Manifests"]
    end

    A1 --> A2 --> A3 --> T1 --> T2 --> T3 --> K1 & K2 & K3

Layer 1: Ansible

Handles physical node configuration and k3s installation.

Role	Purpose
`k3s-server`	Control plane deployment
`k3s-agent`	Worker node deployment
`kube-vip`	API server HA with VIP
`cilium`	CNI networking via Helm (eBPF + Hubble)
`longhorn-disks`	Storage preparation

Layer 2: Terraform

Configures infrastructure services via HCP Terraform.

Module	Purpose
`cluster-bootstrap`	Initial secrets and RBAC
`vault`	Secrets management policies
`keycloak`	Identity provider configuration
`grafana`	Dashboards and datasources
`cloudflare`	DNS records and tunnels
`core-services`	Shared service configuration
`hcp-terraform`	HCP Terraform workspace management

Layer 3: ArgoCD

GitOps-driven application deployment.

Pattern	Use Case
Helm charts	Complex applications (Grafana, Vault)
Kustomize	Environment overlays
Raw manifests	Simple resources

Hardware Architecture

TuringPi 2 Cluster

Two TuringPi 2 boards with RK1 compute modules:

graph TB
    subgraph Alpha["TuringPi Alpha"]
        A1["tpi-alpha-1<br/>Control Plane"]
        A2["tpi-alpha-2<br/>Control Plane"]
        A3["tpi-alpha-3<br/>Control Plane"]
        A4["tpi-alpha-4<br/>Worker"]
    end

    subgraph Beta["TuringPi Beta"]
        B1["tpi-beta-1<br/>Worker"]
        B2["tpi-beta-2<br/>Worker"]
        B3["tpi-beta-3<br/>Worker"]
        B4["tpi-beta-4<br/>Worker"]
    end

    VIP["kube-vip<br/>192.168.20.140"]
    A1 & A2 & A3 --> VIP

Board	Nodes	Role	Count
Alpha	1-3	Control Plane	3
Alpha	4	Worker	1
Beta	1-4	Workers	4

Hardware Specifications

Component	Specification
Compute	Turing RK1 (RK3588, 8-core ARM)
Memory	32GB per node
Storage	NVMe SSD per node
Network	1GbE per node
OS	Armbian 25.08

High Availability

Component	HA Strategy
API Server	kube-vip VIP (`192.168.20.140`)
etcd	Embedded in k3s (3-node quorum)
Storage	Longhorn distributed replication
Ingress	MetalLB load balancing

GitOps Deployment Flow

sequenceDiagram
    participant Dev as Developer
    participant GH as GitHub
    participant HCP as HCP Terraform
    participant Argo as ArgoCD
    participant K8s as Kubernetes

    Dev->>GH: Push to feature branch
    Dev->>GH: Create PR
    GH->>HCP: Trigger speculative plan
    HCP-->>GH: Post plan results
    Dev->>GH: Merge PR
    GH->>HCP: Trigger apply (tf/ changes)
    HCP->>K8s: Apply Terraform changes
    GH->>Argo: Webhook notification
    Argo->>GH: Pull latest manifests
    Argo->>K8s: Sync applications
    K8s-->>Argo: Report sync status

Deployment Triggers

Change Type	Automation
`tf/` changes	HCP Terraform apply
`argocd/` changes	ArgoCD sync
`ansible/` changes	Manual playbook run

Secret Management Flow

sequenceDiagram
    participant Vault as HashiCorp Vault
    participant ESO as External Secrets Operator
    participant K8s as Kubernetes Secret
    participant Pod as Application Pod

    Vault->>Vault: Store secret at path
    ESO->>Vault: Authenticate via K8s auth
    Vault-->>ESO: Return secret data
    ESO->>K8s: Create/update Secret
    K8s-->>Pod: Mount as env/volume

    Note over ESO,K8s: Automatic refresh on TTL

Secret Paths

Category	Vault Path	Example
Cluster services	`secret/fzymgc-house/cluster/*`	`cluster/keycloak`
Infrastructure	`secret/fzymgc-house/infrastructure/*`	`infrastructure/bmc`
Applications	`secret/fzymgc-house/apps/*`	`apps/grafana`

Authentication Flow

sequenceDiagram
    participant User
    participant App as Application
    participant Traefik
    participant Keycloak
    participant Vault

    User->>App: Access protected resource
    App->>Traefik: Request forwarded
    Traefik->>Keycloak: Forward auth check
    Keycloak-->>User: Redirect to login
    User->>Keycloak: Authenticate
    Keycloak->>Keycloak: Validate credentials
    Keycloak-->>User: Issue OIDC token
    User->>App: Request with token
    App->>Traefik: Forward with auth
    Traefik->>Keycloak: Validate token
    Keycloak-->>Traefik: Token valid
    Traefik-->>App: Allow request
    App-->>User: Return resource

Note: For the dashboards this cluster actually forward-auths (Temporal, Hubble UI, agentgateway admin UI, Traefik dashboard, Longhorn), forward-auth is mediated by a shared oauth2-proxy component in front of Keycloak, not Keycloak directly — Keycloak is OIDC-only. See ADR hl-1fia for details.

Authentication Methods

Method	Use Case
OIDC	Web applications (Grafana, ArgoCD)
Forward Auth	Services without native SSO
LDAP	Legacy application support
Certificate	Machine-to-machine auth

Network Architecture

graph TB
    subgraph External["External"]
        CF["Cloudflare Tunnel"]
        Internet["Internet"]
    end

    subgraph Cluster["Cluster Network"]
        MetalLB["MetalLB<br/>145-149, 155-159"]
        Traefik["Traefik Ingress"]
        Cilium["Cilium CNI"]
    end

    subgraph Pods["Pod Network"]
        PodCIDR["10.42.0.0/16"]
    end

    subgraph Services["Service Network"]
        SvcCIDR["10.43.0.0/16"]
    end

    Internet --> CF --> Traefik
    MetalLB --> Traefik
    Traefik --> Cilium --> Pods
    Pods <--> SvcCIDR

Network	CIDR	Purpose
Cluster nodes	`192.168.20.0/24`	Physical node network
Pod network	`10.42.0.0/16`	Kubernetes pods
Service network	`10.43.0.0/16`	Kubernetes services
MetalLB pools	`192.168.20.145-149`, `192.168.20.155-159`	Load balancer IPs
API VIP	`192.168.20.140`	kube-vip HA

See Network Reference for detailed configuration.

Security Architecture

graph TB
    subgraph Identity["Identity Layer"]
        Keycloak["Keycloak SSO"]
        OIDC["OIDC Provider"]
    end

    subgraph Secrets["Secrets Layer"]
        Vault["HashiCorp Vault"]
        ESO["External Secrets"]
    end

    subgraph Network["Network Layer"]
        Cilium["Cilium Policies"]
        Traefik["TLS Termination"]
    end

    subgraph Audit["Audit Layer"]
        Loki["Loki Logs"]
        VM["VictoriaMetrics"]
    end

    Keycloak --> OIDC --> Services
    Vault --> ESO --> Services
    Cilium --> Services
    Traefik --> Services
    Services --> Loki & VM

Domain	Solution	Purpose
Identity	Keycloak	SSO via OIDC
Secrets	Vault + ESO	Dynamic secret injection
Network	Cilium	Pod network policies (CiliumNetworkPolicy)
Ingress	Traefik	TLS termination, routing
Audit	Loki + VM	Centralized logging/metrics

Design Principles

GitOps-First

All cluster state is defined in Git. ArgoCD ensures the cluster matches the repository.

Single source of truth: Git repository
Declarative: Desired state, not imperative commands
Automated sync: Changes apply automatically
Audit trail: Git history tracks all changes

Secrets in Vault

No secrets stored in Git. All sensitive data lives in Vault.

Dynamic secrets: Generated on-demand with TTL
Kubernetes auth: Pods authenticate via service accounts
Automatic rotation: ESO refreshes secrets automatically
Audit logging: All access logged in Vault

SSO via Keycloak

Single sign-on for all services with consistent identity.

OIDC standard: Industry-standard authentication
Forward auth: Protect services without native SSO
Centralized policies: One place for access control
MFA support: Additional security when needed

Observability by Default

Every service is monitored and logged from deployment.

Metrics/Logs/Traces: OTel collectors push to ClickHouse (otel_metrics_*, otel_logs, otel_traces)
Dashboards: HyperDX (ClickStack) provides visibility
Alerts: HyperDX TILE alerts notify via Pushover

Key Design Decisions

Why k3s over k8s?

Factor	k3s	Full k8s
Resource usage	~512MB RAM	~2GB RAM
Installation	Single binary	Many components
ARM support	Native	Requires tuning
Updates	Simple	Complex
Trade-off	Less customizable	More flexible

Decision: Resource efficiency on ARM hardware outweighs customization needs.

Why Vault over Sealed Secrets?

Factor	Vault	Sealed Secrets
Dynamic secrets	Yes	No
Rotation	Automatic	Manual re-seal
Audit	Built-in	None
UI	Yes	No
Complexity	Higher	Lower

Decision: Dynamic secrets and audit logging justify additional complexity.

Identity Provider: Keycloak

Keycloak is the cluster’s current identity provider, handling OIDC SSO for web applications, Kubernetes API access, and (via a shared oauth2-proxy middleware) forward-auth for internal dashboards. The cluster originally ran Authentik as its IdP and migrated to Keycloak; see ADR hl-12i8 for the migration rationale.

Why ArgoCD over Flux?

Factor	ArgoCD	Flux
UI	Full dashboard	None (Weave add-on)
Multi-tenancy	Built-in	Basic
App-of-Apps	Native	Requires setup
Notifications	Extensive	Basic
Learning curve	Lower	Higher

Decision: UI and multi-tenancy support improve operations.

Topic	Link
Network details	Network Reference
Secrets management	Secrets Reference
Services catalog	Services Reference
Technology stack	Technologies Reference