Services Catalog

Inventory of all services deployed in the fzymgc-house cluster.

Quick Reference

Service	URL	Namespace	Category
Vault	vault.fzymgc.house	vault	Platform
Keycloak	id.fzymgc.house	keycloak	Platform
Grafana	grafana.fzymgc.house	grafana	Platform
ArgoCD	argocd.fzymgc.house	argocd	Platform
Temporal	temporal.fzymgc.house	temporal	Application
Mealie	mealie.fzymgc.house	mealie	Application
Octopus	octopus.fzymgc.house	octopus	Application
Longhorn	longhorn.fzymgc.house	longhorn-system	Infrastructure
Tailscale	Internal	tailscale	Infrastructure
Traefik	Internal	traefik	Infrastructure
Router Hosts Operator	Internal	router-hosts-operator	Infrastructure
NATS	Internal	nats	Infrastructure
Mosquitto	mqtt.fzymgc.house:8883	mosquitto	Infrastructure
Cloudflared	N/A (outbound)	cloudflared	Infrastructure
VictoriaMetrics	Internal	prometheus	Observability
Loki	Internal	loki	Observability
Hubble UI	hubble.fzymgc.house	hubble	Observability
Uptime Kuma	status.fzymgc.house	uptime-kuma	Observability
HyperDX	hyperdx.fzymgc.house	clickstack	Observability
ClickStack Operators	Internal	clickstack-operators	Observability
Vector	Internal	vector	Observability
OTel Scraper	Internal	otel-scraper	Observability
Dolt SQL Server	doltdb.fzymgc.house	dolt	Application
CNPG	postgres.fzymgc.house	postgres	Infrastructure
Gateway API	Internal	kube-system	Infrastructure
Grafana Operator	Internal	grafana-operator	Infrastructure
Grafana MCP	grafana-mcp.fzymgc.house	grafana-mcp	Application
Karakeep	karakeep.fzymgc.house	karakeep	Application
Agentgateway	llm-gw.fzymgc.house / mcp-gw.fzymgc.house	agentgateway	Application
Agentgateway Controller	Internal	agentgateway-system	Application
Agent Memory (Engram)	mcp-gw.fzymgc.house/mcp/engram	agent-memory	Application
Headroom (apps)	headroom-apps.headroom-apps.svc.cluster.local:8787 (internal)	headroom-apps	Application
Headroom (agents)	https://headroom.fzymgc.house (Tailscale-internal)	headroom-agents	Application
Firewalla MCP	mcp-gw.fzymgc.house/mcp/firewalla-ro	firewalla-mcp	Application
RustDesk	rd.fzymgc.house (edge node)	rustdesk	Application
K8s OIDC RBAC	Internal	N/A	Platform
Kubernetes Replicator	Internal	kube-system	Infrastructure
Reloader	Internal	kube-system	Infrastructure
System Upgrade Controller	Internal	system-upgrade	Infrastructure
Velero	Internal	velero	Infrastructure
HCP Terraform Operator	Internal	hcp-terraform	GitOps
Actions Runner Controller	Internal	arc-systems	GitOps
GitHub Actions Runners	Internal	arc-runners	GitOps

Platform Services

Vault

Property	Value
URL	vault.fzymgc.house
Alt URLs	vault-0.fzymgc.house, vault-1.fzymgc.house, vault-2.fzymgc.house
Namespace	vault
Ingress Type	TCP Passthrough (TLS termination at Vault)
Auth Method	OIDC (Keycloak)
Vault Path	secret/fzymgc-house/cluster/vault/*
Status	Operational

Keycloak

Property	Value
URL	id.fzymgc.house
Namespace	keycloak
Ingress Type	Traefik IngressRoute (/admin + /realms/master additionally gated to LAN/Tailscale by the admin-allowlist Middleware)
Auth Method	Native (IdP)
Vault Path	secret/fzymgc-house/cluster/keycloak
Status	Operational

Grafana

Property	Value
URL	grafana.fzymgc.house
Namespace	grafana
Ingress Type	Helm Managed
Auth Method	OIDC (Keycloak)
Vault Path	secret/fzymgc-house/cluster/grafana
Status	Operational

ArgoCD

Property	Value
URL	argocd.fzymgc.house
Namespace	argocd
Ingress Type	Helm Managed
Auth Method	OIDC (Keycloak, via the Dex connector)
Vault Path	secret/fzymgc-house/cluster/argocd, secret/fzymgc-house/cluster/argocd/oidc
Status	Operational

K8s OIDC RBAC

Property	Value
URL	Internal only
Namespace	Cluster-scoped
Purpose	Kubernetes RBAC bindings for Keycloak OIDC groups
Auth Method	OIDC (Keycloak)
Status	Operational

RBAC Bindings:

Keycloak Group	ClusterRole	Description
k8s-admins	cluster-admin	Full cluster access
k8s-developers	edit	Edit resources in namespaces
k8s-viewers	view	Read-only access

How it works:

1. Keycloak issues OIDC tokens with group claims (k8s-admins, etc.)
2. ClusterRoleBindings map those groups to Kubernetes RBAC roles
3. k3s validates tokens using --kube-apiserver-arg=oidc-* flags

Application Services

Temporal

Property	Value
URL	temporal.fzymgc.house
Namespace	temporal
Ingress Type	Traefik IngressRoute
Auth Method	Forward-Auth (shared oauth2-proxy, Keycloak-backed — group temporal-users, ADR hl-1fia)
Vault Path	secret/fzymgc-house/cluster/temporal/*
Database	CNPG main cluster: temporal (default store), temporal_visibility (visibility store)
Workers Repo	fzymgc-house/temporal-workers
Status	Active

Components:

• temporal-server - Core Temporal services (frontend, history, matching, worker)
• temporal-web - Web UI for workflow visibility
• temporal-admintools - CLI tools for namespace management
• temporal-worker-controller - Manages worker deployments via CRDs

Mealie

Property	Value
URL	mealie.fzymgc.house
Alt URL	mealie.k8s.fzymgc.house
Namespace	mealie
Ingress Type	Traefik IngressRoute
Auth Method	OIDC (Keycloak, native login — not forward-auth)
Vault Path	secret/fzymgc-house/cluster/mealie
Status	Operational

Octopus

Property	Value
URL (admin UI)	octopus.fzymgc.house (internal, router-hosts)
Webhook (public)	octopus-wh.fzymgc.net/api/github/webhook
Namespace	octopus
Ingress Type	Traefik IngressRoute (UI) + Cloudflare tunnel webhook_services (webhook)
Auth Method	Better Auth (GitHub OAuth + ADMIN_EMAILS); webhook by GitHub HMAC signature
Image	ghcr.io/seanb4t/octopus (fork) + ghcr.io/seanb4t/octopus-migrate
Vault Path	secret/fzymgc-house/cluster/octopus
Runbook	docs/operations/octopus.md
Status	Staged (replicas:0 until go-live, hl-0we.13)

Purpose: Self-hosted AI PR review. LLM + embeddings routed through agentgateway (OpenRouter reviews under ZDR + in-cluster bge-m3 embeddings). Dedicated Qdrant + dedicated CNPG octopus database. Epic hl-0we.

Grafana MCP

Property	Value
URL	grafana-mcp.fzymgc.house
Namespace	grafana-mcp
Ingress Type	Traefik IngressRoute
Auth Method	OIDC (Keycloak)
Vault Path	secret/fzymgc-house/cluster/grafana
Chart	grafana/mcp-grafana v0.2.2
Status	Operational

Purpose:

MCP (Model Context Protocol) server for Claude Code integration. Provides AI assistants with structured access to Grafana data sources:

• Query metrics from Prometheus/VictoriaMetrics
• Query logs from Loki
• List and access dashboards
• Execute Grafana API operations

Usage:

Claude Code connects via the fzymgc-house:grafana skill which uses this MCP server to:

• Investigate infrastructure issues via Grafana data
• Check application metrics and logs
• Create or update dashboards

Dolt SQL Server

Property	Value
URL	doltdb.fzymgc.house
Namespace	dolt
Port	3306 (MySQL wire protocol)
Ingress Type	MetalLB LoadBalancer
Auth Method	MySQL native (password)
Vault Path	secret/fzymgc-house/cluster/dolt
Status	Operational

Purpose:

Version-controlled SQL database (MySQL-compatible) for Beads, Gastown, and AI agents. Dolt provides Git-like versioning for database content, enabling branch/merge workflows on structured data.

Karakeep

Property	Value
URL	karakeep.fzymgc.house
Namespace	karakeep
Ingress Type	Traefik IngressRoute
Auth Method	OIDC (Keycloak)
Vault Path	secret/fzymgc-house/cluster/karakeep
Status	In Progress

Self-hosted bookmark manager with AI-powered auto-tagging, full-text search (Meilisearch), and page crawling/archiving (browserless/chrome).

Agentgateway

Property	Value
LLM URL	llm-gw.fzymgc.house (curated, alias-gated OpenAI-compatible: /v1/chat/completions, /v1/embeddings, /v1/models)
OpenRouter passthrough	openrouter-gw.fzymgc.house/api/v1 (transparent: any OpenRouter model slug + endpoint, any params; opt-in ZDR via header x-agentgateway-backend: openrouter-generic-zdr)
MCP URL	mcp-gw.fzymgc.house/mcp/<server>
Admin UI	agentgateway.fzymgc.house (Traefik + shared oauth2-proxy forward-auth, Keycloak-backed)
Namespace	agentgateway
Ingress Type	Gateway API (gatewayClassName: agentgateway) for LLM/MCP; Traefik IngressRoute for the admin UI
Auth Method	LLM + SaaS MCP: per-client virtual key (apiKeyAuthentication); engram MCP: OAuth/JWT (Keycloak mcp-public, real RFC 7591 DCR)
Vault Path	secret/fzymgc-house/cluster/agentgateway
Charts	agentgateway-crds + agentgateway 1.3.1 (images cr.agentgateway.dev/{controller,agentgateway}:v1.3.1)
Status	Operational

Purpose: k8s-native AI gateway (XDS control-plane via the agentgateway controller + agentgateway.dev/v1alpha1 CRDs). Single front door for the LLM proxy (OpenRouter chat/models + Ollama bge-m3 embeddings, with ZDR + drop_params) and a 6-server MCP gateway. Replaced the standalone (local config.yaml) deployment (ADR hl-jd4, superseded by hl-m3c).

Generic OpenRouter passthrough (openrouter-gw.fzymgc.house): a dedicated transparent lane — use it exactly like https://openrouter.ai/api/v1 (set the client base URL to https://openrouter-gw.fzymgc.house/api/v1). It forwards any OpenRouter OpenAI-compatible endpoint with any model slug and any params, with no per-model alias registration, while still capturing OTel usage/token telemetry (Detect route type). For Zero-Data-Retention, add request header x-agentgateway-backend: openrouter-generic-zdr (chat + embeddings; ZDR is enforced server-side via provider.zdr=true). Auth is a dedicated accept-list (agentgateway-openrouter-vkeys) holding only vk_openrouter_passthrough — separate from the curated agentgateway-vkeys, so the service-account keys (karakeep, mealie, octopus, …) cannot reach this unrestricted lane. Provision vk_openrouter_passthrough in Vault (fzymgc-house/cluster/agentgateway) before use; the lane fails closed until then. The curated llm-gw lane and its or-* aliases are unaffected. ADRs hl-v4vo (dedicated host), hl-l875 (Detect over Passthrough), hl-d5lu (header-selected ZDR).

fovea ZDR lanes (/fovea-scout, /fovea-deepdive): fovea (AI code review) is an agentgateway client that routes its two LLM roles through dedicated header-free ZDR path lanes on llm-gw.fzymgc.house: scout → /fovea-scout/v1/chat/completions (DeepSeek V4-Flash via the openrouter-zdr backend) and deepdive → /fovea-deepdive/v1/chat/completions (GLM-5.2 via the dedicated openrouter-zdr-glm backend). fovea’s correctness and security scout aspects run on the deepdive (GLM-5.2) role; types and tests run on the cheaper scout (Flash) role — a two-tier split bounded by fovea’s closed scout|deepdive role set. Auth is the vk_fovea virtual key on the curated agentgateway-vkeys accept-list. Path-prefix routing (rather than the x-agentgateway-backend header used by llm-chat-zdr) is required because fovea’s per-role base_url cannot set a custom backend-selection header; the backend’s pathPrefix: /api/v1 rewrites the upstream path. ZDR is enforced backend-global (provider.zdr=true, require_parameters=true). A third role, guard (fovea 0.4.4 injection-guard layer), reuses the scout lane (/fovea-scout/v1, DeepSeek V4-Flash) for its LLM-as-judge classifier — it calls with RoleGuard + RequireZDR (fovea injects provider.zdr=true per request), so no new backend or route is needed. The input gate runs in monitor mode (detect-only; emits the fovea.guard.checks metric, never blocks a review — fail-open per fovea ADR fovea-iod); escalate to flag/block after the signal is trusted. Epics hl-3w5i (routing), hl-dbrb (0.5.0 + guard).

Deployment model: Two ArgoCD apps — agentgateway-controller (sync-wave −1: CRDs + controller via multi-source Helm) and agentgateway (sync-wave 0: Gateway + HTTPRoutes + AgentgatewayBackend/AgentgatewayPolicy CRs + ESO secrets). The controller provisions the data-plane Deployment/Service (LoadBalancer, MetalLB). Host DNS (llm-gw/mcp-gw/openrouter-gw → Gateway LB) is registered via a HostMapping CR (the router-hosts operator doesn’t watch Gateways); the litellm host was dropped at decommission (hl-0sr Phase 5).

Observability: agentgateway exports telemetry to ClickStack via two paths (its Rust data plane has no OTLP metrics exporter, so the surfaces are asymmetric):

• Metrics (gen_ai_token_usage, gen_ai_request_duration, gen_ai_time_to_first_token for LLM; mcp_requests for MCP) are exposed on the data plane’s stats port :15020/metrics (the admin :15000 returns 404 on /metrics) via the repo-managed agentgateway-metrics Service, and scraped by monitoring-otel-scraper (static job agentgateway → cs-otel-collector → otel_metrics_{sum,histogram}, ServiceName='agentgateway').
• Traces are pushed natively (OTLP/gRPC to cs-otel-collector:4317, configured via AgentgatewayParameters.rawConfig.config.tracing, authed with the otel_ingest_api_key raw token).

The HyperDX dashboard agentgateway-llm-mcp-usage reports LLM-vs-MCP usage distinctly.

MCP servers: engram (OAuth/Keycloak), exa, firecrawl, context7, fal, deepwiki (the SaaS five gated by virtual key, with per-upstream Authorization keys injected from Vault; deepwiki needs no upstream key).

DCR (Dynamic Client Registration): real RFC 7591 DCR is live. agentgateway’s jwtAuthentication.mcp middleware serves the OAuth discovery + DCR endpoints and proxies anonymous /register calls to Keycloak (no clientId mock short-circuit — mcp-oauth-shared.yaml); authorize/token point at Keycloak directly. The static mcp-public PKCE client remains the discovery anchor/fallback for clients that skip DCR. The DCR call is routed cluster-internally to Keycloak (CoreDNS split-horizon rewrite) with pod-level TLS trust for Keycloak’s ICA1 certificate, restoring cluster-internal-only DCR at the network layer — see ADR hl-ll5p (supersedes the earlier Authentik-era mock-DCR limitation described in ADR hl-23m).

Client config (claude-code / codex):

// LLM: base URL https://llm-gw.fzymgc.house , api key = vk_claude_code
// MCP servers (mcpServers):
"engram":    { "type": "http", "url": "https://mcp-gw.fzymgc.house/mcp/engram" },          // OAuth (Keycloak, DCR)
"exa":       { "type": "http", "url": "https://mcp-gw.fzymgc.house/mcp/exa",       "headers": { "Authorization": "Bearer <vk_claude_code>" } },
"firecrawl": { "type": "http", "url": "https://mcp-gw.fzymgc.house/mcp/firecrawl", "headers": { "Authorization": "Bearer <vk_claude_code>" } },
"context7":  { "type": "http", "url": "https://mcp-gw.fzymgc.house/mcp/context7",  "headers": { "Authorization": "Bearer <vk_claude_code>" } },
"fal-ai":    { "type": "http", "url": "https://mcp-gw.fzymgc.house/mcp/fal",       "headers": { "Authorization": "Bearer <vk_claude_code>" } },
"deepwiki":  { "type": "http", "url": "https://mcp-gw.fzymgc.house/mcp/deepwiki",  "headers": { "Authorization": "Bearer <vk_claude_code>" } }

Agentgateway Controller

Property	Value
URL	Internal only
Namespace	agentgateway-system
Purpose	XDS control plane + CRD lifecycle for agentgateway data-plane
Charts	agentgateway-crds + agentgateway controller 1.3.1
ArgoCD App	agentgateway-controller (sync-wave −1)
Status	Operational

Purpose: Installs agentgateway.dev/v1alpha1 CRDs and runs the agentgateway controller that watches those CRDs and configures the data-plane pods in the agentgateway namespace via XDS. Deployed ahead of the gateway app (sync-wave −1).

Agent Memory (Engram)

Property	Value
MCP URL	mcp-gw.fzymgc.house/mcp/engram (via agentgateway, OAuth)
Console URL	engram.fzymgc.house (root redirects to /ui since 0.7.0; LAN/Tailscale-only; Traefik IngressRoute, vault-issuer TLS)
Namespace	agent-memory
Chart	ghcr.io/seanb4t/charts/engram 0.7.0
Auth Method	MCP: OAuth/JWT (Keycloak mcp-public, DCR, validated by agentgateway). Console: OIDC auth-code (Keycloak engram-ui confidential client, distinct MEM_UI_ISSUER)
Vault Path	secret/fzymgc-house/cluster/agent-memory (embedder/OTLP), secret/fzymgc-house/cluster/engram-ui (console OIDC client secret + cookie key)
Storage	Qdrant vector DB (qdrant-data PVC, longhorn-encrypted) + R2 snapshot backup
Status	Operational

Purpose: Persistent vector-memory store for AI agent harness. Wraps a Qdrant instance with an MCP server (seanb4t/engram). Embeddings routed via agentgateway (ollama/bge-m3). The MCP endpoint is exposed through agentgateway — clients authenticate with Keycloak mcp-public OAuth tokens (DCR). The operator console (“Connect” web UI) is exposed LAN-only at engram.fzymgc.house via a Traefik IngressRoute, authenticated through a separate Keycloak engram-ui confidential client. engram 0.6.1’s MEM_UI_ISSUER decouples the console’s OIDC issuer from the MCP bearer issuer so both auth lanes coexist (per-app Keycloak issuers differ). Since 0.7.0 the MCP transport mounts at an explicit MEM_MCP_PATH (/mcp, matching the agentgateway backend) instead of the root catch-all, freeing the host root to redirect to the console.

Firewalla MCP

Property	Value
MCP URL	mcp-gw.fzymgc.house/mcp/firewalla-ro (via agentgateway, virtual key vk_firewalla)
Namespace	firewalla-mcp
Image	ghcr.io/fzymgc-house/firewalla-mcp (pinned by digest)
Auth Method	Client: per-client virtual key (apiKeyAuthentication); gateway→server static bearer; server→MSP personal access token
Vault Path	secret/fzymgc-house/cluster/firewalla-mcp (msp_id/token/box_id); bearer + VK at secret/fzymgc-house/cluster/agentgateway
Source Repo	github.com/fzymgc-house/firewalla-mcp (Go, modelcontextprotocol/go-sdk)
Status	Read-only v1 (boxes/devices/alarms/flows/rules/target-lists); scoped writes are a fast-follow

Purpose: Read-only MCP server wrapping the Firewalla MSP API v2 (docs.firewalla.net), so agents can triage Firewalla state (the class of incident behind hl-gy4x) over MCP instead of manual SSH spelunking. Built in fzymgc-house/firewalla-mcp (engram pattern: own repo, distroless image, CI-published), deployed in-cluster behind agentgateway. v1 exposes only read tools on /mcp/firewalla-ro; scoped writes (rule + target-list management) will land on a separate /mcp/firewalla-rw route (ADR hl-rzrg). Tracked under epic hl-vbvu.1.

Headroom compression service

Two single-replica instances providing LLM context compression. Each lives in its own namespace with an independent PVC and telemetry service.name.

headroom-apps

Property	Value
URL	http://headroom-apps.headroom-apps.svc.cluster.local:8787 (ClusterIP, internal only)
Namespace	headroom-apps
Surface	OpenAI /v1/chat/completions
Upstream	https://llm-gw.fzymgc.house/v1
Run mode	--mode token (maximize JSON compression)
Image	ghcr.io/chopratejas/headroom:0.27.0-nonroot
Vault Path	secret/fzymgc-house/cluster/clickstack (otel_ingest_api_key, reused)
Runbook	docs/operations/headroom.md
Status	Operational

headroom-agents

Property	Value
URL	https://headroom.fzymgc.house (Tailscale-internal; not on public DNS)
Namespace	headroom-agents
Surface	Anthropic /v1/messages
Upstream	https://api.anthropic.com (direct)
Run mode	--mode cache (preserve Anthropic prefix-cache hits)
Image	ghcr.io/chopratejas/headroom:0.27.0-code-nonroot
Vault Path	secret/fzymgc-house/cluster/clickstack (otel_ingest_api_key, reused)
Runbook	docs/operations/headroom.md
Status	Operational

RustDesk

Property	Value
URL	rd.fzymgc.house
Namespace	rustdesk
Status	Remnant (namespace exists; workloads run on edge node)

Note: RustDesk Pro (hbbs + hbbr) runs on the DigitalOcean edge node (edge-node.fzymgc.house) via Docker Compose — not in the k3s cluster. The rustdesk namespace exists in the cluster but contains no active workloads; it is a remnant from an earlier deployment attempt. See docs/operations/rustdesk-clients.md for client configuration.

Infrastructure Services

Traefik

Property	Value
URL	Internal only
Namespace	traefik
Ingress Type	N/A (is the ingress controller)
Auth Method	None
Ports	80 (HTTP), 443 (HTTPS)
Status	Operational

Longhorn

Property	Value
URL	longhorn.fzymgc.house
Namespace	longhorn-system
Ingress Type	Traefik IngressRoute
Auth Method	Forward-Auth (shared oauth2-proxy, Keycloak-backed — group longhorn-users, ADR hl-1fia)
Status	Operational

MetalLB

Property	Value
URL	Internal only
Namespace	metallb
Ingress Type	N/A (provides LoadBalancer IPs)
Auth Method	None
IP Pools	192.168.20.145-149, 192.168.20.155-159
Status	Operational

cert-manager

Property	Value
URL	Internal only
Namespace	cert-manager
Ingress Type	N/A
Auth Method	None
Issuers	Let’s Encrypt (production), Self-signed (internal)
Status	Operational

External Secrets Operator

Property	Value
URL	Internal only
Namespace	external-secrets
Ingress Type	N/A
Auth Method	Vault Kubernetes Auth
ClusterSecretStore	vault
Status	Operational

Cloudflared

Property	Value
URL	N/A (outbound tunnel)
Namespace	cloudflared
Purpose	External ingress via Cloudflare Tunnel
Status	Operational

Tailscale

Property	Value
URL	Internal only
Namespace	tailscale
Ingress Type	N/A (outbound mesh network)
Auth Method	OAuth (Tailscale API)
Purpose	Subnet router and exit node for tailnet access
Vault Path	secret/fzymgc-house/cluster/tailscale/oauth
Status	Operational

Router Hosts Operator

Property	Value
URL	Internal only
Namespace	router-hosts-operator
Chart Source	ghcr.io/fzymgc-house/charts/router-hosts-operator
Chart Version	0.8.14
Image	ghcr.io/fzymgc-house/router-hosts-operator:0.8.14
Auth Method	mTLS (Vault PKI)
Purpose	DNS host entry management via HostEntry CRD
Vault PKI Role	router-hosts-client
Server Version	0.9.7 (Ansible: Firewalla router image tag)
Status	Operational

NATS

Property	Value
URL	Internal only
Namespace	nats
Ingress Type	N/A (internal messaging)
Auth Method	NKey (Ed25519 signatures)
Vault Path	secret/fzymgc-house/cluster/nats
Storage	10Gi per node (longhorn-encrypted)
Cluster Size	3 replicas
Status	Active

Features:

• JetStream persistence with Raft consensus
• 3-node cluster for high availability
• TLS for client and cluster connections
• Account-based multi-tenancy (SYS, SERVICES, IOT)
• Prometheus metrics on port 7777

Accounts:

Account	Purpose	Consumers
SYS	System monitoring and admin	nats-box, monitoring
SERVICES	Cluster service communication	Temporal, future services
IOT	IoT device messaging	Home Assistant (future)

Ports:

Port	Protocol	Purpose
4222	NATS	Client connections
1883	MQTT	MQTT listener
6222	NATS	Cluster routes
7777	HTTP	Prometheus metrics
8222	HTTP	Monitoring endpoint

See NATS Operations for key management and administration.

Mosquitto

Property	Value
URL	mqtt.fzymgc.house:8883
Namespace	mosquitto
Ingress Type	LoadBalancer (MetalLB)
Auth Method	Username/password (mosquitto_passwd)
Password File	/mosquitto/config/passwd (from mosquitto-auth)
Purpose	MQTT broker for IoT and Home Assistant
TLS	External on port 8883
Bridge TLS	Uses fzymgc-ica1-ca full chain bundle
Status	Operational

CNPG (CloudNativePG)

Property	Value
URL	postgres.fzymgc.house
Namespace	postgres
Ingress Type	Traefik IngressRoute
Auth Method	PostgreSQL native (TLS + password)
Cluster Name	main
Instances	3 replicas
Storage	10Gi per instance (postgres-storage)
PostgreSQL Version	18.1
Status	Operational

Purpose:

CloudNativePG is a Kubernetes operator for PostgreSQL. It manages the main PostgreSQL cluster that provides databases for multiple applications.

Databases:

Database	Owner	Application
keycloak	keycloak	Keycloak IdP
grafana	grafana	Grafana
mealie	mealie	Mealie
octopus	octopus	Octopus PR review
temporal	temporal	Temporal (default store)
temporal_visibility	temporal	Temporal (visibility store)

Features:

• Automatic failover with Raft consensus
• Continuous backup to Longhorn snapshots
• TLS encryption for all connections
• WAL archiving via Barman Cloud plugin
• 15-day retention policy for backups

Connection:

postgres://user:password@main-rw.postgres.svc.cluster.local:5432/database?sslmode=require

Connection pooling (Keycloak):

Keycloak connects through a CNPG Pooler (PgBouncer) instead of main-rw directly, so a primary switchover is absorbed by the pooler — Keycloak keeps its JDBC socket while PgBouncer reconnects server-side to the new primary — rather than collapsing Keycloak’s pool and dropping it out of Traefik rotation (mirrors the same failure mode originally documented against Authentik on 2026-06-16, before the decommission). Epic hl-1jn1; pairs with primaryUpdateMethod: switchover.

Property	Value
Pooler	keycloak-pooler-rw (ns postgres)
Service DNS	keycloak-pooler-rw.postgres.svc.cluster.local:5432
Instances	2 (HA)
Type / poolMode	rw / session
Auth	auto-provisioned (cnpg_pooler_pgbouncer role + user_search)

• session pool mode is used (not transaction) because transaction pooling would break session-scoped state Keycloak relies on (advisory locks during startup Liquibase migrations, prepared statements via the Agroal/JDBC pool).
• TLS: the client→pooler hop is TLS-required. PgBouncer otherwise serves clients plaintext by default, so the pooler sets client_tls_sslmode: require (encrypt only — not verify-ca/verify-full, which would force client-cert mTLS that Keycloak’s password auth does not do). The operator supplies the cluster server cert (postgres-main-server-tls, vault-issuer) for the pooler to present; its SAN covers only main-r/ro/rw, so clients cannot use sslmode=verify-full against the pooler DNS. Closing that gap (add the pooler DNS to the postgres-main-server cert SAN, then move both ends to verify-full) is tracked in hl-20ep.
• CNPG does not auto-pause PgBouncer on switchover. For a planned switchover where zero connection errors are required, manually pause first: set spec.pgbouncer.paused: true on the Pooler (PgBouncer PAUSE), perform the switchover, then set it back to false.
• Deliberately no conn_max_age / conn_health_checks: the session-mode pooler already pools and stabilizes the socket across a switchover, so ephemeral per-request connections recover on their own. App-side persistent connections (conn_max_age>0) double-pool and pin session slots — they crashlooped a previous co-tenant’s server via liveness timeouts (PR #1329, reverted #1330); avoid setting it for any pooled consumer of this cluster.

Gateway API

Property	Value
URL	Internal only
Namespace	kube-system
Version	v1.4.1 (experimental CRDs)
Purpose	Kubernetes Gateway API resources
Status	Operational

Purpose:

Gateway API provides Kubernetes-native traffic routing resources. Currently deployed for experimental features and future traffic management capabilities.

Installed CRDs:

• GatewayClass, Gateway, HTTPRoute
• TCPRoute, UDPRoute, TLSRoute
• ReferenceGrant, BackendTLSPolicy

Grafana Operator

Property	Value
URL	Internal only
Namespace	grafana-operator
Purpose	Kubernetes operator for Grafana resources
Status	Operational

Purpose:

Manages Grafana resources declaratively via Kubernetes CRDs:

• GrafanaFolder - Dashboard organization
• GrafanaDashboard - Dashboard definitions
• GrafanaAlertRuleGroup - Alert rules
• GrafanaDatasource - Data source configuration

Usage:

Applications define their Grafana resources in their app-configs directories, and the operator syncs them to the Grafana instance.

Kubernetes Replicator

Property	Value
URL	Internal only
Namespace	kube-system
Source	mittwald/kubernetes-replicator
Purpose	Replicate Secrets and ConfigMaps across namespaces
Status	Operational

Purpose:

Automatically replicates Secrets and ConfigMaps to multiple namespaces based on annotations.

Usage:

Add annotation to source Secret/ConfigMap:

replicator.v1.mittwald.de/replicate-to: "namespace-a,namespace-b"

Or use regex patterns:

replicator.v1.mittwald.de/replicate-to-matching: "app-.*"

Reloader

Property	Value
URL	Internal only
Namespace	kube-system
Replicas	2
Purpose	Auto-reload Deployments on ConfigMap/Secret changes
Status	Operational

Purpose:

Watches for changes in ConfigMaps and Secrets, then triggers rolling restarts of associated Deployments/StatefulSets/DaemonSets.

Usage:

Add annotation to Deployment:

reloader.stakater.com/auto: "true"

Or specify exact resources:

configmap.reloader.stakater.com/reload: "my-configmap"
secret.reloader.stakater.com/reload: "my-secret"

System Upgrade Controller

Property	Value
URL	Internal only
Namespace	system-upgrade
Version	v0.18.0
Purpose	Automated k3s cluster upgrades
Status	Operational

Purpose:

Manages rolling upgrades of k3s server and agent nodes using Plan CRDs.

Upgrade Plans:

Plan	Target	Channel
k3s-server	Control plane nodes	stable
k3s-agent	Worker nodes	stable

Process:

1. Plans check k3s release channel for new versions
2. Server nodes upgrade first (one at a time)
3. Agent nodes upgrade after servers complete
4. Nodes are cordoned/drained during upgrade

Velero

Property	Value
URL	Internal only
Namespace	velero
Purpose	Kubernetes backup and disaster recovery
Vault Path	secret/fzymgc-house/cluster/velero
Status	Operational

Purpose:

Backs up Kubernetes resources and persistent volumes for disaster recovery.

Backup Strategy:

Uses exclude-only approach - all namespaces backed up by default except infrastructure/stateless ones.

Schedules:

Schedule	Frequency	TTL	Description
daily-backup	Daily 2 AM	30 days	Core resources
weekly-full-backup	Sunday 3 AM	90 days	Extended resources including NetworkPolicies

Excluded Namespaces:

• Kubernetes core: kube-system, kube-node-lease, kube-public, default
• Networking: traefik, metallb (Cilium runs in kube-system which is already excluded)
• Operators: cert-manager, external-secrets, cnpg-system, grafana-operator
• Ephemeral: arc-systems, arc-runners, system-upgrade
• Telemetry: loki, cloudflared, vector, grafana-mcp

Observability Services

VictoriaMetrics

Property	Value
URL	Internal only
Namespace	prometheus
Ingress Type	None
Auth Method	None
Purpose	Metrics storage (Prometheus-compatible)
Status	Operational

Loki

Property	Value
URL	Internal only
Namespace	loki
Ingress Type	None
Auth Method	None
Purpose	Log aggregation
Status	Operational

Uptime Kuma

Property	Value
URL	status.fzymgc.house
Namespace	uptime-kuma
Ingress Type	Traefik IngressRoute (status.fzymgc.house redirects to status.fzymgc.net)
Auth Method	Cloudflare Access (status.fzymgc.net) — no group restriction, broadly accessible to cluster operators
Storage	1Gi Longhorn PVC (SQLite)
Status	Operational

Purpose: External service uptime monitoring with status page capabilities.

Initial Monitors:

• https://id.fzymgc.house - Keycloak
• https://vault.fzymgc.house - Vault UI
• https://grafana.fzymgc.house - Grafana

Planned Enhancement: Terraform-managed monitors via breml/uptimekuma provider (see selfhosted-cluster-ucb).

Hubble UI

Property	Value
URL	hubble.fzymgc.house
Namespace	hubble (ingress); upstream service in kube-system
Ingress Type	Traefik IngressRoute (cross-namespace service ref to hubble-ui.kube-system.svc:80)
Auth Method	ForwardAuth (shared oauth2-proxy, Keycloak-backed), group hubble-users (ADR hl-1fia)
Storage	None (read-only flow visualization from hubble-relay)
Status	Operational

Purpose: Cilium network flow observability dashboard. Visualizes live pod-to-pod traffic, filters by namespace/verdict/L7 attributes, and renders a service-map graph. Used during CiliumNetworkPolicy rollouts to verify intended flows are permitted and unintended flows are dropped.

Operator runbook: docs/operations/hubble-ui.md.

HyperDX

Property	Value
URL	hyperdx.fzymgc.house
Namespace	clickstack
Ingress Type	Traefik IngressRoute
Auth Method	OIDC (Keycloak)
Vault Path	secret/fzymgc-house/cluster/clickstack
OTel Endpoint	192.168.20.149:4317 (MetalLB, TLS + bearer token)
Status	Evaluation (parallel-run with Loki)

Purpose: ClickStack UI providing logs, metrics, and traces in a unified observability interface. Evaluation period compares against Loki + Grafana stack. See docs/operations/clickstack-comparison.md for the evaluation scorecard.

ClickStack Operators

Property	Value
URL	Internal only
Namespace	clickstack-operators
Chart	clickhouse.github.io/ClickStack-helm-charts/clickstack-operators 1.0.0
Purpose	MongoDB and ClickHouse operators for the ClickStack observability stack
Status	Operational

Purpose: Bundles the MongoDB Community Operator and the ClickHouse operator (Altinity). Deployed in clickstack-operators, configured to reconcile CRs in the clickstack namespace. Required before the clickstack ArgoCD app.

Vector

Property	Value
URL	Internal only
Namespace	vector
Chart	helm.vector.dev/vector 0.52.0 (app 0.55.0-distroless-libc)
Controller	DaemonSet (role: Agent)
Purpose	Ship pod logs directly to ClickHouse (otel_logs)
Status	Operational

Purpose: Cluster-wide log shipper. Runs as a DaemonSet (tolerates control-plane taint) reading pod logs from the Kubernetes API and writing directly to ClickHouse (otelcollector user), bypassing the OTel Collector. Part of the ClickStack observability pipeline. Config via argocd/app-configs/monitoring-vector/.

OTel Scraper

Property	Value
URL	Internal only
Namespace	otel-scraper
Chart	open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-collector 0.154.0
Controllers	Deployment (static Prometheus scrape) + DaemonSet (host metrics per node)
Vault Path	secret/fzymgc-house/cluster/clickstack (otel_ingest_api_key)
Purpose	Scrape cluster metrics and node metrics, forward to ClickStack via OTLP
Status	Operational

Purpose: Two OpenTelemetry collector deployments — otel-scrape (Deployment: scrapes ~20 in-cluster Prometheus targets + k8scluster receiver) and otel-node (DaemonSet: hostmetrics + kubeletstats per node). Both export OTLP/gRPC to cs-otel-collector.clickstack.svc.cluster.local:4317. See docs/operations/clickstack.md for token rotation.

GitOps Services

HCP Terraform Operator

Property	Value
URL	Internal only
Namespace	hcp-terraform
Chart	helm.releases.hashicorp.com/hcp-terraform-operator 2.11.0
Purpose	Terraform Cloud workspace management via Workspace CRDs
Status	Operational

Purpose: Watches Workspace CRDs in the cluster and drives HCP Terraform to plan/apply the corresponding workspaces. Credentials sourced from Vault via ESO. See docs/operations/hcp-terraform.md.

Actions Runner Controller

Property	Value
URL	Internal only
Controller Namespace	arc-systems
Runners Namespace	arc-runners
Chart	ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller 0.14.1
Purpose	GitHub Actions self-hosted runners (autoscaling scale sets)
Status	Operational

Purpose: Actions Runner Controller (ARC) deploys autoscaling GitHub Actions runner scale sets. The controller (arc-systems) manages runner pods in arc-runners. Scale-set registration token from arc-runners ExternalSecret.

External Services

Service	Purpose	Management
Cloudflare	DNS, Tunnels, WAF	Terraform (tf/cloudflare)
HCP Terraform	Infrastructure automation	Web UI
GitHub	Source control, Actions	Web UI
Let’s Encrypt	TLS certificates	cert-manager

Auth Method Reference

Method	Description	Configuration
OIDC	Direct OpenID Connect authentication	Keycloak client integration
Forward-Auth	Traefik middleware proxies auth to the shared oauth2-proxy (Keycloak-backed)	forwardAuth middleware
Certificate	mTLS client certificate	Vault PKI integration
None	No authentication required	Internal services only

Ingress Type Reference

Type	Description	TLS Handling
Traefik IngressRoute	Native Traefik CRD	Traefik terminates TLS
TCP Passthrough	Raw TCP proxy	Backend terminates TLS
Helm Managed	Ingress defined in Helm values	Varies by chart
Cloudflare Tunnel	External via cloudflared	Cloudflare terminates
kube-vip VIP	Direct LoadBalancer IP	Service handles TLS

Adding a New Service

1. Create Kubernetes manifests in argocd/app-configs/<service>/
2. Configure ingress (IngressRoute or Ingress resource)
3. Set up authentication:
   • OIDC: Create a Keycloak client (tf/keycloak/)
   • Forward-Auth: Add a route to the shared oauth2-proxy middleware (with a ?allowed_groups=<app>-users gate) — see ADR hl-1fia
4. Add secrets to Vault if needed
5. Create ExternalSecret for Kubernetes secret sync
6. Update this catalog