Services Catalog

Inventory of all services deployed in the fzymgc-house cluster.

Quick Reference

Service	URL	Namespace	Category
Vault	vault.fzymgc.house	vault	Platform
Keycloak	id.fzymgc.house	keycloak	Platform
Grafana	grafana.fzymgc.house	grafana	Platform
ArgoCD	argocd.fzymgc.house	argocd	Platform
Temporal	temporal.fzymgc.house	temporal	Application
Mealie	mealie.fzymgc.house	mealie	Application
Octopus	octopus.fzymgc.house	octopus	Application
Longhorn	longhorn.fzymgc.house	longhorn-system	Infrastructure
Tailscale	Internal	tailscale	Infrastructure
Traefik	Internal	traefik	Infrastructure
NATS	Internal	nats	Infrastructure
Mosquitto	mqtt.fzymgc.house:8883	mosquitto	Infrastructure
Cloudflared	N/A (outbound)	cloudflared	Infrastructure
VictoriaMetrics	Internal	prometheus	Observability
Loki	Internal	loki	Observability
Hubble UI	hubble.fzymgc.house	hubble	Observability
Uptime Kuma	status.fzymgc.house	uptime-kuma	Observability
HyperDX	hyperdx.fzymgc.house	clickstack	Observability
ClickStack Operators	Internal	clickstack-operators	Observability
Vector	Internal	vector	Observability
OTel Scraper	Internal	otel-scraper	Observability
Dolt SQL Server	doltdb.fzymgc.house	dolt	Application
CNPG	postgres.fzymgc.house	postgres	Infrastructure
Gateway API	Internal	kube-system	Infrastructure
Grafana Operator	Internal	grafana-operator	Infrastructure
Grafana MCP	grafana-mcp.fzymgc.house	grafana-mcp	Application
Karakeep	karakeep.fzymgc.house	karakeep	Application
LiteLLM	llm.fzymgc.house (LLM /v1, MCP /<server>/mcp, admin /ui)	litellm	Application
Agent Memory (Engram)	llm.fzymgc.house/engram/mcp	agent-memory	Application
Headroom (apps)	headroom-apps.headroom-apps.svc.cluster.local:8787 (internal)	headroom-apps	Application
Headroom (agents)	https://headroom.fzymgc.house (Tailscale-internal)	headroom-agents	Application
Firewalla MCP	llm.fzymgc.house/firewalla_ro/mcp	firewalla-mcp	Application
hl-assets Uploader	drop.fzymgc.net (write); i.hl-assets.dev (read, edge-only — no origin)	hl-assets	Application
RustDesk	rd.fzymgc.house (edge node)	— (edge node only)	Application
K8s OIDC RBAC	Internal	N/A	Platform
Kubernetes Replicator	Internal	kube-system	Infrastructure
Reloader	Internal	kube-system	Infrastructure
System Upgrade Controller	Internal	system-upgrade	Infrastructure
Velero	Internal	velero	Infrastructure
HCP Terraform Operator	Internal	hcp-terraform	GitOps
Actions Runner Controller	Internal	arc-systems	GitOps
GitHub Actions Runners	Internal	arc-runners	GitOps

Platform Services

Vault

Property	Value
URL	vault.fzymgc.house
Alt URLs	vault-0.fzymgc.house, vault-1.fzymgc.house, vault-2.fzymgc.house
Namespace	vault
Ingress Type	TCP Passthrough (TLS termination at Vault)
Auth Method	OIDC (Keycloak)
Vault Path	secret/fzymgc-house/cluster/vault/*
Status	Operational

Keycloak

Property	Value
URL	id.fzymgc.house
Namespace	keycloak
Ingress Type	Traefik IngressRoute (/admin + /realms/master additionally gated to LAN/Tailscale by the admin-allowlist Middleware)
Auth Method	Native (IdP)
Vault Path	secret/fzymgc-house/cluster/keycloak
Status	Operational

Grafana

Property	Value
URL	grafana.fzymgc.house
Namespace	grafana
Ingress Type	Helm Managed
Auth Method	OIDC (Keycloak)
Vault Path	secret/fzymgc-house/cluster/grafana
Status	Operational

ArgoCD

Property	Value
URL	argocd.fzymgc.house
Namespace	argocd
Ingress Type	Helm Managed
Auth Method	OIDC (Keycloak, via the Dex connector)
Vault Path	secret/fzymgc-house/cluster/argocd, secret/fzymgc-house/cluster/argocd/oidc
Status	Operational

K8s OIDC RBAC

Property	Value
URL	Internal only
Namespace	Cluster-scoped
Purpose	Kubernetes RBAC bindings for Keycloak OIDC groups
Auth Method	OIDC (Keycloak)
Status	Operational

RBAC Bindings:

Keycloak Group	ClusterRole	Description
k8s-admins	cluster-admin	Full cluster access
k8s-developers	edit	Edit resources in namespaces
k8s-viewers	view	Read-only access

How it works:

1. Keycloak issues OIDC tokens with group claims (k8s-admins, etc.)
2. ClusterRoleBindings map those groups to Kubernetes RBAC roles
3. k3s validates tokens using --kube-apiserver-arg=oidc-* flags

Application Services

Temporal

Property	Value
URL	temporal.fzymgc.house
Namespace	temporal
Ingress Type	Traefik IngressRoute
Auth Method	Forward-Auth (shared oauth2-proxy, Keycloak-backed — group temporal-users, ADR hl-1fia)
Vault Path	secret/fzymgc-house/cluster/temporal/*
Database	CNPG main cluster: temporal (default store), temporal_visibility (visibility store)
Workers Repo	fzymgc-house/temporal-workers
Status	Active

Components:

• temporal-server - Core Temporal services (frontend, history, matching, worker)
• temporal-web - Web UI for workflow visibility
• temporal-admintools - CLI tools for namespace management
• temporal-worker-controller - Manages worker deployments via CRDs

Mealie

Property	Value
URL	mealie.fzymgc.house
Alt URL	mealie.k8s.fzymgc.house
Namespace	mealie
Ingress Type	Traefik IngressRoute
Auth Method	OIDC (Keycloak, native login — not forward-auth)
Vault Path	secret/fzymgc-house/cluster/mealie
Status	Operational

Octopus

Property	Value
URL (admin UI)	octopus.fzymgc.house (internal)
Webhook (public)	octopus-wh.fzymgc.net/api/github/webhook
Namespace	octopus
Ingress Type	Traefik IngressRoute (UI) + Cloudflare tunnel webhook_services (webhook)
Auth Method	Better Auth (GitHub OAuth + ADMIN_EMAILS); webhook by GitHub HMAC signature
Image	ghcr.io/seanb4t/octopus (fork) + ghcr.io/seanb4t/octopus-migrate
Vault Path	secret/fzymgc-house/cluster/octopus
Runbook	docs/operations/octopus.md
Status	Staged (replicas:0 until go-live, hl-0we.13)

Purpose: Self-hosted AI PR review. LLM + embeddings routed through LiteLLM at llm.fzymgc.house (by way of headroom-apps): OpenRouter reviews under ZDR on the openrouter-zdr/* lane, embeddings on the octopus-embed lane. Dedicated Qdrant + dedicated CNPG octopus database. Epic hl-0we.

Grafana MCP

Property	Value
URL	grafana-mcp.fzymgc.house
Namespace	grafana-mcp
Ingress Type	Traefik IngressRoute
Auth Method	OIDC (Keycloak)
Vault Path	secret/fzymgc-house/cluster/grafana
Chart	grafana/mcp-grafana v0.2.2
Status	Operational

Purpose:

MCP (Model Context Protocol) server for Claude Code integration. Provides AI assistants with structured access to Grafana data sources:

• Query metrics from Prometheus/VictoriaMetrics
• Query logs from Loki
• List and access dashboards
• Execute Grafana API operations

Usage:

Claude Code connects via the fzymgc-house:grafana skill which uses this MCP server to:

• Investigate infrastructure issues via Grafana data
• Check application metrics and logs
• Create or update dashboards

Dolt SQL Server

Property	Value
URL	doltdb.fzymgc.house
Namespace	dolt
Port	3306 (MySQL wire protocol)
Ingress Type	MetalLB LoadBalancer
Auth Method	MySQL native (password)
Vault Path	secret/fzymgc-house/cluster/dolt
Status	Operational

Purpose:

Version-controlled SQL database (MySQL-compatible) for Beads, Gastown, and AI agents. Dolt provides Git-like versioning for database content, enabling branch/merge workflows on structured data.

Karakeep

Property	Value
URL	karakeep.fzymgc.house
Namespace	karakeep
Ingress Type	Traefik IngressRoute
Auth Method	OIDC (Keycloak)
Vault Path	secret/fzymgc-house/cluster/karakeep
Status	In Progress

Self-hosted bookmark manager with AI-powered auto-tagging, full-text search (Meilisearch), and page crawling/archiving (browserless/chrome).

LiteLLM

Property	Value
LLM URL	https://llm.fzymgc.house/v1 (OpenAI-compatible: /v1/chat/completions, /v1/embeddings, /v1/models; per-consumer virtual key in Authorization: Bearer)
MCP URL	https://llm.fzymgc.house/<server>/mcp — ten routes, admission by virtual key in x-litellm-api-key: Bearer <key> (see MCP Gateway Clients)
Admin UI	https://llm.fzymgc.house/ui (LiteLLM’s native generic OIDC → Keycloak SSO; client in tf/keycloak/litellm.tf)
Namespace	litellm
Ingress Type	Traefik IngressRoute — the host root is published with a recorded denylist of excluded prefixes (/login, /register, /authorize, /token, the root .well-known documents) and a priority-100 allow for the per-server /.well-known/oauth-protected-resource/<server>/mcp documents (D-67)
Auth Method	LLM + key-scoped MCP: per-consumer virtual key (/key/generate; budgets, rate limits, object_permission.mcp_servers). Identity MCP (engram, kubernetes): the client’s own Keycloak token relayed to the upstream (oauth_passthrough), obtained by the client through RFC 9728 discovery → RFC 7591 DCR → PKCE
Vault Path	secret/fzymgc-house/cluster/litellm (master key, OpenRouter keys, the seven shared upstream/MCP credentials); each consumer’s own virtual key on that consumer’s path (litellm_api_key)
State	CNPG cluster main, database litellm (keys, spend, budgets); auth cache in a dedicated Valkey cluster in litellm-valkey
Chart	litellm-helm 1.96.2, image digest-pinned (argocd/app-configs/litellm-chart/values.yaml; secrets and ingress in argocd/app-configs/litellm/)
Runbooks	docs/operations/litellm.md (operator), docs/operations/mcp-gateway-clients.md (clients)
Status	Operational — the only LLM and MCP data plane

Purpose: The unified LLM and MCP data plane (ADR 2026-08-15; the previous gateway was decommissioned by the second ADR dated 2026-09-08 in the ADR index). One OpenAI-compatible proxy fronts OpenRouter, Google Gemini and the MCP upstreams; every consumer authenticates with its own virtual key, which carries a per-key USD budget, a rate limit and — for MCP — the set of routes it may reach. Spend is attributed per key alias.

LLM lanes (proxy_config.model_list in values.yaml; the lane name is what a client sends as model): consumer lanes fovea-scout, fovea-deepdive, fovea-embed, octopus-embed, engram-embed, engram-summarize, gemini-embed, gpt-4o-mini; the passthrough wildcards openrouter/* and openrouter-zdr/* (any OpenRouter slug, the latter with Zero-Data-Retention enforced by the lane); and the operator-only zdr-probe-embed. A key’s models list is its allow-list and is checked before aliases resolve, so every string a consumer sends must appear on its key — see LiteLLM → Minting and scoping virtual keys. The openrouter-passthrough key is the generic passthrough lane’s credential, scoped to the two wildcards.

MCP routes (proxy_config.mcp_servers; ten entries, three dispositions): context7, exa, firecrawl, fal (SaaS, LiteLLM injects the upstream key) and firewalla_ro, clickhouse_ro, clickhouse_rw (in-cluster, LiteLLM injects the upstream bearer) overwrite the client’s Authorization; engram and kubernetes relay the client’s own Keycloak token (oauth_passthrough: true) and hold no upstream credential; deepwiki holds no upstream credential and no relay flag. Tool names are prefixed with the server name (context7-resolve-library-id). clickhouse_rw is the only route that can mutate state and is reachable only by a key scoped to it.

Identity routes and discovery: an unauthenticated call to engram or kubernetes answers 401 with WWW-Authenticate: Bearer resource_metadata=…/.well-known/oauth-protected-resource/<server>/mcp. That document is the upstream’s own RFC 9728 metadata, relayed by LiteLLM with only resource rewritten (D-66, D-68); it names Keycloak as the authorization server, the client registers itself dynamically against Keycloak (loopback Trusted-Hosts policy the sole control, D-69) and completes PKCE. kubernetes-mcp validates the relayed token locally against the realm’s JWKS with audience https://llm.fzymgc.house/kubernetes/mcp (hl-94n1), and the k3s API server authorizes it as oidc:<email> (hl-bc94). The engram route’s cutover waits on an engram release that serves its own document (seanb4t/engram#526).

Observability: LiteLLM emits GenAI metrics, logs and spans over OTLP to cs-otel-collector (ClickStack): gen_ai.client.operation.duration, gen_ai.client.response.duration, gen_ai.client.token.usage and gen_ai.usage.cost, labelled by metadata.user_api_key_alias. HyperDX dashboards LiteLLM — LLM & key usage (six tiles) and the engram-embed brownout tile/alert (engram-embed-p95-litellm) read them; the OBS-02 key-leak detector (saved search + alert over logs and spans) reads the same streams. Key material is scrubbed at the collector (sk-[REDACTED]) on the Traefik log, the uvicorn log and the span attributes.

Client config (claude-code / codex):

// LLM: base URL https://llm.fzymgc.house/v1 , api key = your virtual key (claude_code_api_key on cluster/litellm)
// MCP servers (mcpServers) — eight key-scoped routes share one header shape:
"context7": { "type": "http", "url": "https://llm.fzymgc.house/context7/mcp", "headers": { "x-litellm-api-key": "Bearer <your-virtual-key>" } },
"exa":      { "type": "http", "url": "https://llm.fzymgc.house/exa/mcp",      "headers": { "x-litellm-api-key": "Bearer <your-virtual-key>" } },
// ... firecrawl, fal, deepwiki, firewalla_ro, clickhouse_ro (clickhouse_rw only on a key scoped to it)
// identity routes: the same x-litellm-api-key header and NO static Authorization — Claude Code obtains the
// Keycloak token itself; add them with `claude mcp add … --callback-port <port>` (see the client runbook)
"kubernetes": { "type": "http", "url": "https://llm.fzymgc.house/kubernetes/mcp", "headers": { "x-litellm-api-key": "Bearer <your-virtual-key>" } }

Agent Memory (Engram)

Property	Value
MCP URL	llm.fzymgc.house/engram/mcp (via LiteLLM, the client’s Keycloak token relayed; cutover pending the engram release that serves its own RFC 9728 document — seanb4t/engram#526)
Console URL	engram.fzymgc.house (root redirects to /ui since 0.7.0; LAN/Tailscale-only; Traefik IngressRoute, vault-issuer TLS)
Namespace	agent-memory
Chart	ghcr.io/seanb4t/charts/engram 0.11.1
Auth Method	MCP: OAuth/JWT (Keycloak, audience mcp-public, client registered by DCR; validated by engram itself — LiteLLM relays, admission by virtual key in x-litellm-api-key). Console: OIDC auth-code (Keycloak engram-ui confidential client, distinct ENGRAM_UI_ISSUER)
Vault Path	secret/fzymgc-house/cluster/agent-memory (embedder/OTLP), secret/fzymgc-house/cluster/engram-ui (console OIDC client secret + cookie key)
Storage	Qdrant vector DB (qdrant-data PVC, longhorn-encrypted) + R2 snapshot backup
Status	Operational

Purpose: Persistent vector-memory store for AI agent harness. Wraps a Qdrant instance with an MCP server (seanb4t/engram). Embeddings are routed via LiteLLM’s engram-embed lane to gemini-embedding-2 (3072-dim, Google’s paid embedder), into Qdrant collection memory_v4 (the Phase 21 cutover off the earlier ollama/bge-m3 and qwen3-embedding-8b lanes). The MCP endpoint is exposed through LiteLLM — clients authenticate with Keycloak OAuth tokens (audience mcp-public, DCR) which LiteLLM relays to engram untouched. The operator console (“Connect” web UI) is exposed LAN-only at engram.fzymgc.house via a Traefik IngressRoute, authenticated through a separate Keycloak engram-ui confidential client. engram 0.6.1’s ENGRAM_UI_ISSUER decouples the console’s OIDC issuer from the MCP bearer issuer so both auth lanes coexist (per-app Keycloak issuers differ). Since 0.7.0 the MCP transport mounts at an explicit ENGRAM_MCP_PATH (/mcp, matching the LiteLLM upstream URL) instead of the root catch-all, freeing the host root to redirect to the console. Chart 0.11.1 additionally makes available — none of it enabled here — an opt-in service-auth chain for machine callers, write idempotency keys, structured citations, and memory supersession.

Firewalla MCP

Property	Value
MCP URL	llm.fzymgc.house/firewalla_ro/mcp (via LiteLLM; a virtual key scoped to firewalla_ro)
Namespace	firewalla-mcp
Image	ghcr.io/fzymgc-house/firewalla-mcp (pinned by digest)
Auth Method	Client: LiteLLM virtual key scoped to firewalla_ro (x-litellm-api-key); gateway→server static bearer injected by LiteLLM; server→MSP personal access token
Vault Path	secret/fzymgc-house/cluster/firewalla-mcp (msp_id/token/box_id); the bearer the server validates is firewalla_mcp_auth_token on secret/fzymgc-house/cluster/litellm
Source Repo	github.com/fzymgc-house/firewalla-mcp (Go, modelcontextprotocol/go-sdk)
Status	Read-only v1 (boxes/devices/alarms/flows/rules/target-lists); scoped writes are a fast-follow

Purpose: Read-only MCP server wrapping the Firewalla MSP API v2 (docs.firewalla.net), so agents can triage Firewalla state (the class of incident behind hl-gy4x) over MCP instead of manual SSH spelunking. Built in fzymgc-house/firewalla-mcp (engram pattern: own repo, distroless image, CI-published), deployed in-cluster behind LiteLLM. v1 exposes only read tools on the firewalla_ro route; scoped writes (rule + target-list management) will land on a separate firewalla_rw route (ADR hl-rzrg). Tracked under epic hl-vbvu.1.

hl-assets Uploader

Property	Value
URL	Public: https://drop.fzymgc.net via Cloudflare Access over the cloudflared-main tunnel; the tunnel dials the ClusterIP http://hl-assets-uploader.hl-assets.svc.cluster.local:8080 directly (no Traefik hop, D-69), and the CiliumNetworkPolicy now carries an ingress key admitting TCP/8080 from the cloudflared namespace only (D-71). Operations: hl-assets Uploader
Namespace	hl-assets
Workloads	Deployment/hl-assets-uploader, Service/hl-assets-uploader (ClusterIP:8080). replicas: 1, no autoscaler of any kind
Image	ghcr.io/fzymgc-house/hl-assets-uploader:0.2.0@sha256:0bd2dc3e…d34f86c (private GHCR package, dual-arch). Pinned by semver plus the OCI index digest (D-49/D-50) — never a child manifest digest, which would be unpullable on this all-arm64 estate. Re-pinned on each phase merge that rebuilds the image; read 2026-08-16 from argocd/app-configs/hl-assets/deployment.yaml
Source	apps/hl-assets-uploader/ (Go module, cgo against libheif + libwebp)
Auth Method	Cloudflare Access headers are read without being verified (D-61); unauthenticated uploads are accepted and recorded as such. Origin-side Cf-Access-Jwt-Assertion verification was deliberately declined under D-70, relying on Cloudflare’s Tunnel exemption plus the compensating Cilium ingress rule (D-71); the record shape is final, only the trust model changed. A3 measured the forged email stripped on the service-token path — see the Access edge ADR. Four service-token client classes now exist: share.fish, the curl one-liner, the iOS Shortcut, and the uptime monitor (its own pair, on its own Vault path)
Vault Paths	secret/fzymgc-house/cluster/hl-assets/r2 (six fields → Secret hl-assets-r2), secret/fzymgc-house/cluster/hl-assets/salt (→ Secret hl-assets-salt), secret/fzymgc-house/cluster/hl-assets/access-token (the three client-class pairs), secret/fzymgc-house/cluster/hl-assets/monitor-token (the uptime monitor’s own pair), secret/fzymgc-house/cluster/clickstack (otel_ingest_api_key → Secret hl-assets-otlp-headers), secret/fzymgc-house/cluster/ghcr/pull-secret (→ Secret ghcr-pull-secret)
Egress	CiliumNetworkPolicy/hl-assets-egress — DNS, TCP/443 to world, and the ClickStack collector on 4318. Port-scoped, not host-scoped (see the credential-isolation ADR)
Storage	None. Request-scoped spill on a disk-backed emptyDir at /spill, unlinked at creation; the pod’s root filesystem is read-only
Operator surface	Three takedown verbs on the same binary, run via kubectl exec: unshare (reversible), purge (irreversible, requires a confirmation token) and republish (the restore path). Procedure, decision table and edge-purge step: hl-assets Uploader → Taking a link down
Monitoring	Two Uptime Kuma monitors — a keyword canary on i.hl-assets.dev/_health/canary and an authenticated /readyz probe — both paging the cluster-edge Pushover app. Four R2 usage notification policies watch operation volume. See Monitoring
Runbook	apps/hl-assets-uploader/PORTABILITY.md (relocation checklist)
Status	Deployed and ready (one replica, measured 2026-08-15). Phase 6’s takedown verbs and its Terraform are merged-pending, not live — see below

Purpose: The write path for hl-assets.dev. A client POSTs a file; the pod strips metadata, transcodes HEIC to WebP, derives a content-addressed object key, writes to R2 and returns the public URL in one request. No client ever holds an R2 credential.

The read path is not this service. Shared links are served entirely from Cloudflare’s edge off an R2 custom domain — there is no origin in a public GET, so this pod being down, restarting or removed does not affect a single existing link. See the split read/write ADR and the edge-policy ADR.

Endpoints:

Path	Purpose
POST /api/v1/upload	The only functional route. Multipart body
GET /healthz	Liveness. Process is alive
GET /readyz	Readiness. Performs no network call by design — it answers “can this pod accept an upload”, not “is R2 reachable”. A dependency outage must not restart a healthy process
GET /statusz	Object-store reachability, in-flight reservation total, encoder versions. Deliberately not wired to a kubelet probe — it calls R2, and a probe that calls R2 turns an upstream blip into a restart loop

Upload contract. Two success shapes, and the distinction is the deduplication signal:

• 201 Created — fresh upload. Body is a JSON object carrying the public URL; an X-Token header carries the capability that permits a later takedown. It is issued once, in a header only, with no re-issue path — a client that drops it loses delete capability for that object permanently.
• 200 OK — deduplication hit. Byte-identical content already exists; the same URL is returned, no X-Token is issued, and the uploader’s filename is not recorded.
• Both carry an X-HL-Assets-Dedup header, on both content types, because the plain-text body has nowhere to carry the signal.
• Accept: text/plain returns the bare URL plus one newline and nothing else — no JSON parsing needed, for curl and shell clients.
• Every refusal is structured JSON with a code from a closed enum — never HTML, never a redirect. Video and camera-RAW are refused with 415, whose body names the detected type.
• A non-POST request to the upload path returns 404 with Allow: POST, not 405. That is deliberate: adding a “wrong method” code is additive and belongs to a change that owns the error enum.

How an operator checks it: scripts/share.fish drives the live upload path end to end through Cloudflare Access and prints the public URL; the Go suite inside the image build holds the contract above. The verifier (tools/hl-assets-verify/) covers the operational items only.

Deployment state, measured 2026-08-15 — the three items previously listed here as “not yet true” have all since landed, and are recorded as measured rather than left to be assumed:

• The service is deployed and ready. kubectl -n hl-assets get deploy hl-assets-uploader reports one ready replica, running the exact digest pinned in the Deployment.
• The image digest pin is in force. The hl-assets.fzymgc.house/image-digest annotation is gone and the image line carries @sha256:…; a test asserts the biconditional, so the annotation and the digest cannot both be present or both absent.
• Upload events reach ClickStack. A batch was accepted by the real cs-otel-collector and the event is queryable in ClickHouse (Phase 4). Query default.otel_logs for hl_assets.upload. If a stream ever goes empty, check the collector for an authentication rejection before suspecting the emitter — it degrades silently by design.

What is NOT yet in the running pod: the three takedown verbs shipped in Phase 6 are newer than the deployed tag. They reach the cluster only once the image is rebuilt, merged and synced by ArgoCD — so a kubectl exec … uploader unshare against the pod running today falls through to the server path: the deployed binary does not recognise the verb at all, so the argument slice reaches run(), which loads config from the pod’s environment and tries to open a second listener on the address the running server already holds. Expect a bind failure that never mentions takedowns — not a usage error, and not a takedown.

Everything Terraform-side in Phase 6 (the fourth service token, the four usage notification policies, the R2 lifecycle rule, both Uptime Kuma monitors) is likewise declared but unapplied: those workspaces are VCS-connected and apply only on merge to main.

The salt is not rotatable. See docs/reference/secrets.md; rotating it does not error, it silently stops deduplication.

Related: uploader-service ADR (the write path’s decisions), credential-isolation ADR (what the R2 token can reach, and the accepted residual).

Headroom compression service

Two single-replica instances providing LLM context compression. Each lives in its own namespace with an independent PVC and telemetry service.name.

headroom-apps

Property	Value
URL	http://headroom-apps.headroom-apps.svc.cluster.local:8787 (ClusterIP, internal only)
Namespace	headroom-apps
Surface	OpenAI /v1/chat/completions
Upstream	https://llm.fzymgc.house/v1 (LiteLLM)
Run mode	--mode token (maximize JSON compression)
Image	ghcr.io/chopratejas/headroom:0.27.0-nonroot
Vault Path	secret/fzymgc-house/cluster/clickstack (otel_ingest_api_key, reused)
Runbook	docs/operations/headroom.md
Status	Operational

headroom-agents

Property	Value
URL	https://headroom.fzymgc.house (Tailscale-internal; not on public DNS)
Namespace	headroom-agents
Surface	Anthropic /v1/messages
Upstream	https://api.anthropic.com (direct)
Run mode	--mode cache (preserve Anthropic prefix-cache hits)
Image	ghcr.io/chopratejas/headroom:0.27.0-code-nonroot
Vault Path	secret/fzymgc-house/cluster/clickstack (otel_ingest_api_key, reused)
Runbook	docs/operations/headroom.md
Status	Operational

RustDesk

Property	Value
URL	rd.fzymgc.house
Namespace	— (edge node only; the in-cluster namespace was deleted 2026-08-16)
Status	Runs on the edge node via Docker Compose

Note: RustDesk Pro (hbbs + hbbr) runs on the DigitalOcean edge node (edge-node.fzymgc.house) via Docker Compose — not in the k3s cluster. See docs/operations/rustdesk-clients.md for client configuration.

The in-cluster rustdesk namespace was deleted on 2026-08-16. It was a remnant of an earlier deployment attempt and its IngressRoute published rd-admin.fzymgc.house, the last live name the incoming PowerDNS tier did not carry — see docs/operations/dns.md. Its PVC used the longhorn-retain storage class, so the underlying volume survives as a Released PV, and the namespace is inside Velero’s backup scope.

This entry previously read “the rustdesk namespace exists in the cluster but contains no active workloads.” That was wrong at the time it was acted on: the namespace held a Deployment at 1/1 with a pod that had 96 days of uptime. A namespace being a remnant and being empty are different claims, and the second is the one people act on without looking. Corrected here rather than quietly overwritten, because the wrong version is the kind that gets trusted.

Infrastructure Services

Traefik

Property	Value
URL	Internal only
Namespace	traefik
Ingress Type	N/A (is the ingress controller)
Auth Method	None
Ports	80 (HTTP), 443 (HTTPS)
Status	Operational

Longhorn

Property	Value
URL	longhorn.fzymgc.house
Namespace	longhorn-system
Ingress Type	Traefik IngressRoute
Auth Method	Forward-Auth (shared oauth2-proxy, Keycloak-backed — group longhorn-users, ADR hl-1fia)
Status	Operational

MetalLB

Property	Value
URL	Internal only
Namespace	metallb
Ingress Type	N/A (provides LoadBalancer IPs)
Auth Method	None
IP Pools	192.168.20.145-149, 192.168.20.155-159
Status	Operational

cert-manager

Property	Value
URL	Internal only
Namespace	cert-manager
Ingress Type	N/A
Auth Method	None
Issuers	Let’s Encrypt (production), Self-signed (internal)
Status	Operational

External Secrets Operator

Property	Value
URL	Internal only
Namespace	external-secrets
Ingress Type	N/A
Auth Method	Vault Kubernetes Auth
ClusterSecretStore	vault
Status	Operational

Cloudflared

Property	Value
URL	N/A (outbound tunnel)
Namespace	cloudflared
Purpose	External ingress via Cloudflare Tunnel
Status	Operational

Tailscale

Property	Value
URL	Internal only
Namespace	tailscale
Ingress Type	N/A (outbound mesh network)
Auth Method	OAuth (Tailscale API)
Purpose	Subnet router and exit node for tailnet access
Vault Path	secret/fzymgc-house/cluster/tailscale/oauth
Status	Operational

NATS

Property	Value
URL	Internal only
Namespace	nats
Ingress Type	N/A (internal messaging)
Auth Method	NKey (Ed25519 signatures)
Vault Path	secret/fzymgc-house/cluster/nats
Storage	10Gi per node (longhorn-encrypted)
Cluster Size	3 replicas
Status	Active

Features:

• JetStream persistence with Raft consensus
• 3-node cluster for high availability
• TLS for client and cluster connections
• Account-based multi-tenancy (SYS, SERVICES, IOT)
• Prometheus metrics on port 7777

Accounts:

Account	Purpose	Consumers
SYS	System monitoring and admin	nats-box, monitoring
SERVICES	Cluster service communication	Temporal, future services
IOT	IoT device messaging	Home Assistant (future)

Ports:

Port	Protocol	Purpose
4222	NATS	Client connections
1883	MQTT	MQTT listener
6222	NATS	Cluster routes
7777	HTTP	Prometheus metrics
8222	HTTP	Monitoring endpoint

See NATS Operations for key management and administration.

Mosquitto

Property	Value
URL	mqtt.fzymgc.house:8883
Namespace	mosquitto
Ingress Type	LoadBalancer (MetalLB)
Auth Method	Username/password (mosquitto_passwd)
Password File	/mosquitto/config/passwd (from mosquitto-auth)
Purpose	MQTT broker for IoT and Home Assistant
TLS	External on port 8883
Bridge TLS	Uses fzymgc-ica1-ca full chain bundle
Status	Operational

CNPG (CloudNativePG)

Property	Value
URL	postgres.fzymgc.house
Namespace	postgres
Ingress Type	Traefik IngressRoute
Auth Method	PostgreSQL native (TLS + password)
Cluster Name	main
Instances	3 replicas
Storage	10Gi per instance (postgres-storage)
PostgreSQL Version	18.1
Status	Operational

Purpose:

CloudNativePG is a Kubernetes operator for PostgreSQL. It manages the main PostgreSQL cluster that provides databases for multiple applications.

Databases:

Database	Owner	Application
keycloak	keycloak	Keycloak IdP
grafana	grafana	Grafana
mealie	mealie	Mealie
octopus	octopus	Octopus PR review
temporal	temporal	Temporal (default store)
temporal_visibility	temporal	Temporal (visibility store)

Features:

• Automatic failover with Raft consensus
• Continuous backup to Longhorn snapshots
• TLS encryption for all connections
• WAL archiving via Barman Cloud plugin
• 15-day retention policy for backups

Connection:

postgres://user:password@main-rw.postgres.svc.cluster.local:5432/database?sslmode=require

Connection pooling (Keycloak):

Keycloak connects through a CNPG Pooler (PgBouncer) instead of main-rw directly, so a primary switchover is absorbed by the pooler — Keycloak keeps its JDBC socket while PgBouncer reconnects server-side to the new primary — rather than collapsing Keycloak’s pool and dropping it out of Traefik rotation (mirrors the same failure mode originally documented against Authentik on 2026-06-16, before the decommission). Epic hl-1jn1; pairs with primaryUpdateMethod: switchover.

Property	Value
Pooler	keycloak-pooler-rw (ns postgres)
Service DNS	keycloak-pooler-rw.postgres.svc.cluster.local:5432
Instances	2 (HA)
Type / poolMode	rw / session
Auth	auto-provisioned (cnpg_pooler_pgbouncer role + user_search)

• session pool mode is used (not transaction) because transaction pooling would break session-scoped state Keycloak relies on (advisory locks during startup Liquibase migrations, prepared statements via the Agroal/JDBC pool).
• TLS: the client→pooler hop is TLS-required. PgBouncer otherwise serves clients plaintext by default, so the pooler sets client_tls_sslmode: require (encrypt only — not verify-ca/verify-full, which would force client-cert mTLS that Keycloak’s password auth does not do). The operator supplies the cluster server cert (postgres-main-server-tls, vault-issuer) for the pooler to present; its SAN covers only main-r/ro/rw, so clients cannot use sslmode=verify-full against the pooler DNS. Closing that gap (add the pooler DNS to the postgres-main-server cert SAN, then move both ends to verify-full) is tracked in hl-20ep.
• CNPG does not auto-pause PgBouncer on switchover. For a planned switchover where zero connection errors are required, manually pause first: set spec.pgbouncer.paused: true on the Pooler (PgBouncer PAUSE), perform the switchover, then set it back to false.
• Deliberately no conn_max_age / conn_health_checks: the session-mode pooler already pools and stabilizes the socket across a switchover, so ephemeral per-request connections recover on their own. App-side persistent connections (conn_max_age>0) double-pool and pin session slots — they crashlooped a previous co-tenant’s server via liveness timeouts (PR #1329, reverted #1330); avoid setting it for any pooled consumer of this cluster.

Gateway API

Property	Value
URL	Internal only
Namespace	kube-system
Version	v1.4.1 (experimental CRDs)
Purpose	Kubernetes Gateway API resources
Status	Operational

Purpose:

Gateway API provides Kubernetes-native traffic routing resources. The CRDs are vendored by the Cilium Ansible role and stay installed for Cilium’s own use; no application on the cluster drives LLM or MCP traffic through them (the LiteLLM data plane is fronted by a Traefik IngressRoute).

Installed CRDs:

• GatewayClass, Gateway, HTTPRoute
• TCPRoute, UDPRoute, TLSRoute
• ReferenceGrant, BackendTLSPolicy

Grafana Operator

Property	Value
URL	Internal only
Namespace	grafana-operator
Purpose	Kubernetes operator for Grafana resources
Status	Operational

Purpose:

Manages Grafana resources declaratively via Kubernetes CRDs:

• GrafanaFolder - Dashboard organization
• GrafanaDashboard - Dashboard definitions
• GrafanaAlertRuleGroup - Alert rules
• GrafanaDatasource - Data source configuration

Usage:

Applications define their Grafana resources in their app-configs directories, and the operator syncs them to the Grafana instance.

Kubernetes Replicator

Property	Value
URL	Internal only
Namespace	kube-system
Source	mittwald/kubernetes-replicator
Purpose	Replicate Secrets and ConfigMaps across namespaces
Status	Operational

Purpose:

Automatically replicates Secrets and ConfigMaps to multiple namespaces based on annotations.

Usage:

Add annotation to source Secret/ConfigMap:

replicator.v1.mittwald.de/replicate-to: "namespace-a,namespace-b"

Or use regex patterns:

replicator.v1.mittwald.de/replicate-to-matching: "app-.*"

Reloader

Property	Value
URL	Internal only
Namespace	kube-system
Replicas	2
Purpose	Auto-reload Deployments on ConfigMap/Secret changes
Status	Operational

Purpose:

Watches for changes in ConfigMaps and Secrets, then triggers rolling restarts of associated Deployments/StatefulSets/DaemonSets.

Usage:

Add annotation to Deployment:

reloader.stakater.com/auto: "true"

Or specify exact resources:

configmap.reloader.stakater.com/reload: "my-configmap"
secret.reloader.stakater.com/reload: "my-secret"

System Upgrade Controller

Property	Value
URL	Internal only
Namespace	system-upgrade
Version	v0.18.0
Purpose	Automated k3s cluster upgrades
Status	Operational

Purpose:

Manages rolling upgrades of k3s server and agent nodes using Plan CRDs.

Upgrade Plans:

Plan	Target	Channel
k3s-server	Control plane nodes	stable
k3s-agent	Worker nodes	stable

Process:

1. Plans check k3s release channel for new versions
2. Server nodes upgrade first (one at a time)
3. Agent nodes upgrade after servers complete
4. Nodes are cordoned/drained during upgrade

Velero

Property	Value
URL	Internal only
Namespace	velero
Purpose	Kubernetes backup and disaster recovery
Vault Path	secret/fzymgc-house/cluster/velero
Status	Operational

Purpose:

Backs up Kubernetes resources and persistent volumes for disaster recovery.

Backup Strategy:

Uses exclude-only approach - all namespaces backed up by default except infrastructure/stateless ones.

Schedules:

Schedule	Frequency	TTL	Description
daily-backup	Daily 2 AM	30 days	Core resources
weekly-full-backup	Sunday 3 AM	90 days	Extended resources including NetworkPolicies

Excluded Namespaces:

• Kubernetes core: kube-system, kube-node-lease, kube-public, default
• Networking: traefik, metallb (Cilium runs in kube-system which is already excluded)
• Operators: cert-manager, external-secrets, cnpg-system, grafana-operator
• Ephemeral: arc-systems, arc-runners, system-upgrade
• Telemetry: loki, cloudflared, vector, grafana-mcp

Observability Services

VictoriaMetrics

Property	Value
URL	Internal only
Namespace	prometheus
Ingress Type	None
Auth Method	None
Purpose	Metrics storage (Prometheus-compatible)
Status	Operational

Loki

Property	Value
URL	Internal only
Namespace	loki
Ingress Type	None
Auth Method	None
Purpose	Log aggregation
Status	Operational

Uptime Kuma

Property	Value
URL	status.fzymgc.house
Namespace	uptime-kuma
Ingress Type	Traefik IngressRoute (status.fzymgc.house redirects to status.fzymgc.net)
Auth Method	Cloudflare Access (status.fzymgc.net) — no group restriction, broadly accessible to cluster operators
Storage	1Gi Longhorn PVC (SQLite)
Status	Operational

Purpose: External service uptime monitoring with status page capabilities.

Initial Monitors:

• https://id.fzymgc.house - Keycloak
• https://vault.fzymgc.house - Vault UI
• https://grafana.fzymgc.house - Grafana

Planned Enhancement: Terraform-managed monitors via breml/uptimekuma provider (see selfhosted-cluster-ucb).

Hubble UI

Property	Value
URL	hubble.fzymgc.house
Namespace	hubble (ingress); upstream service in kube-system
Ingress Type	Traefik IngressRoute (cross-namespace service ref to hubble-ui.kube-system.svc:80)
Auth Method	ForwardAuth (shared oauth2-proxy, Keycloak-backed), group hubble-users (ADR hl-1fia)
Storage	None (read-only flow visualization from hubble-relay)
Status	Operational

Purpose: Cilium network flow observability dashboard. Visualizes live pod-to-pod traffic, filters by namespace/verdict/L7 attributes, and renders a service-map graph. Used during CiliumNetworkPolicy rollouts to verify intended flows are permitted and unintended flows are dropped.

Operator runbook: docs/operations/hubble-ui.md.

HyperDX

Property	Value
URL	hyperdx.fzymgc.house
Namespace	clickstack
Ingress Type	Traefik IngressRoute
Auth Method	OIDC (Keycloak)
Vault Path	secret/fzymgc-house/cluster/clickstack
OTel Endpoint	192.168.20.149:4317 (MetalLB, TLS + bearer token)
Status	Evaluation (parallel-run with Loki)

Purpose: ClickStack UI providing logs, metrics, and traces in a unified observability interface. Evaluation period compares against Loki + Grafana stack. See docs/operations/clickstack-comparison.md for the evaluation scorecard.

ClickStack Operators

Property	Value
URL	Internal only
Namespace	clickstack-operators
Chart	clickhouse.github.io/ClickStack-helm-charts/clickstack-operators 1.0.0
Purpose	MongoDB and ClickHouse operators for the ClickStack observability stack
Status	Operational

Purpose: Bundles the MongoDB Community Operator and the ClickHouse operator (Altinity). Deployed in clickstack-operators, configured to reconcile CRs in the clickstack namespace. Required before the clickstack ArgoCD app.

Vector

Property	Value
URL	Internal only
Namespace	vector
Chart	helm.vector.dev/vector 0.58.0 (app 0.58.0-distroless-libc)
Controller	DaemonSet (role: Agent)
Purpose	Ship pod logs directly to ClickHouse (otel_logs)
Status	Operational

Purpose: Cluster-wide log shipper. Runs as a DaemonSet (tolerates control-plane taint) reading pod logs from the Kubernetes API and writing directly to ClickHouse (otelcollector user), bypassing the OTel Collector. Part of the ClickStack observability pipeline. Config via argocd/app-configs/monitoring-vector/.

Secrets: Vector 0.57+ does not interpolate environment variables in its config, so a secret reaches the config as a file: mount the Secret key under /etc/vector-secrets (extraVolumes in monitoring-vector.yaml) and reference it as SECRET[clickhouse.<key>]. Check a change with a real vector run, not vector validate: validate does not resolve the secrets backend and reports a false auth failure.

OTel Scraper

Property	Value
URL	Internal only
Namespace	otel-scraper
Chart	open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-collector 0.154.0
Controllers	Deployment (static Prometheus scrape) + DaemonSet (host metrics per node)
Vault Path	secret/fzymgc-house/cluster/clickstack (otel_ingest_api_key)
Purpose	Scrape cluster metrics and node metrics, forward to ClickStack via OTLP
Status	Operational

Purpose: Two OpenTelemetry collector deployments — otel-scrape (Deployment: scrapes ~20 in-cluster Prometheus targets + k8scluster receiver) and otel-node (DaemonSet: hostmetrics + kubeletstats per node). Both export OTLP/gRPC to cs-otel-collector.clickstack.svc.cluster.local:4317. See docs/operations/clickstack.md for token rotation.

GitOps Services

HCP Terraform Operator

Property	Value
URL	Internal only
Namespace	hcp-terraform
Chart	helm.releases.hashicorp.com/hcp-terraform-operator 2.11.0
Purpose	Terraform Cloud workspace management via Workspace CRDs
Status	Operational

Purpose: Watches Workspace CRDs in the cluster and drives HCP Terraform to plan/apply the corresponding workspaces. Credentials sourced from Vault via ESO. See docs/operations/hcp-terraform.md.

Actions Runner Controller

Property	Value
URL	Internal only
Controller Namespace	arc-systems
Runners Namespace	arc-runners
Chart	ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller 0.14.1
Purpose	GitHub Actions self-hosted runners (autoscaling scale sets)
Status	Operational

Purpose: Actions Runner Controller (ARC) deploys autoscaling GitHub Actions runner scale sets. The controller (arc-systems) manages runner pods in arc-runners. Scale-set registration token from arc-runners ExternalSecret.

External Services

Service	Purpose	Management
Cloudflare	DNS, Tunnels, WAF	Terraform (tf/cloudflare)
HCP Terraform	Infrastructure automation	Web UI
GitHub	Source control, Actions	Web UI
Let’s Encrypt	TLS certificates	cert-manager

Auth Method Reference

Method	Description	Configuration
OIDC	Direct OpenID Connect authentication	Keycloak client integration
Forward-Auth	Traefik middleware proxies auth to the shared oauth2-proxy (Keycloak-backed)	forwardAuth middleware
Certificate	mTLS client certificate	Vault PKI integration
None	No authentication required	Internal services only

Ingress Type Reference

Type	Description	TLS Handling
Traefik IngressRoute	Native Traefik CRD	Traefik terminates TLS
TCP Passthrough	Raw TCP proxy	Backend terminates TLS
Helm Managed	Ingress defined in Helm values	Varies by chart
Cloudflare Tunnel	External via cloudflared	Cloudflare terminates
kube-vip VIP	Direct LoadBalancer IP	Service handles TLS

Adding a New Service

1. Create Kubernetes manifests in argocd/app-configs/<service>/
2. Configure ingress (IngressRoute or Ingress resource)
3. Set up authentication:
   • OIDC: Create a Keycloak client (tf/keycloak/)
   • Forward-Auth: Add a route to the shared oauth2-proxy middleware (with a ?allowed_groups=<app>-users gate) — see ADR hl-1fia
4. Add secrets to Vault if needed
5. Create ExternalSecret for Kubernetes secret sync
6. Update this catalog