Inventory of the services that run in the fzymgc-house cluster. Every entry names the manifest that
defines it. An ArgoCD Application in argocd/cluster-app/templates/ delivers each in-cluster
service, unless the entry names another source. secret/ prefixes every Vault path below.
Keycloak operator CRDs and controller 26.6.4 (argocd/app-configs/keycloak-operator/)
Instances
2
Ingress Type
Traefik IngressRoute. The admin-allowlist Middleware limits /admin and /realms/master to LAN and Tailscale sources. The Cloudflare tunnel delivers the public path straight to keycloak-service
Auth Method
Native (the identity provider)
Database
CNPG main, database keycloak, through the keycloak-pooler-rw Pooler
Keycloak issues OIDC tokens with group claims. ClusterRoleBindings map those groups to the
ClusterRoles above. k3s validates the tokens with its --kube-apiserver-arg=oidc-* flags.
The shared forward-auth provider. middlewares.yaml defines one Traefik forwardAuth Middleware
per consumer, each with an allowed_groups gate: temporal-users, longhorn-users,
hubble-users and traefik-dashboard-users.
Deployment/temporal-node-gateway in the namespace temporal: Envoy with an OPA sidecar, two replicas, behind Service/temporal-node-gateway-vip on 192.168.250.146:7233 and [fddb:f665:73f7:250::146]:7233. The resolver Pis dial the IPv4 literal. Envoy ends the mTLS of the resolver Pis and admits only a client certificate with a URI SAN under spiffe://fzymgc.house/dns-node-worker/. OPA binds each node to the Temporal namespace dns-control and to its own task queue. Manifests in argocd/app-configs/temporal-server/node-gateway/
Temporal namespaces
workflows for the cluster workers, dns-control for dns-control and the node workers (retention 7 days)
Components:
temporal-server runs the frontend, history, matching and worker services.
temporal-web is the UI for workflow visibility.
temporal-admintools holds the CLI for namespace management.
temporal-worker-controller manages worker deployments through CRDs.
temporal-webhook-receiver turns inbound webhooks into workflow signals.
Self-hosted AI pull request review. LLM calls and embeddings go to the LiteLLM Service in the
cluster. Reviews use the octopus-review lane, utility calls use the octopus-utility lane and
embeddings use the octopus-embed lane. The two chat lanes allow only providers with zero data
retention (ZDR).
doltdb.fzymgc.house (routed VIP 192.168.250.147 and fddb:f665:73f7:250::147, Service dolt-vip)
Remote API
doltdb-remote.fzymgc.house (Traefik IngressRoute to port 50051)
Namespace
dolt
Image
dolthub/dolt-sql-server:2.3.3
Ports
3306 (MySQL wire protocol), 9418
Ingress Type
Routed LoadBalancer dolt-vip, dual-stack. The ClusterIP Service dolt serves the in-cluster scraper
Auth Method
MySQL native (password)
Vault Path
fzymgc-house/cluster/dolt
Manifests
argocd/app-configs/dolt/
Version-controlled SQL database with a MySQL-compatible interface, for Beads, Gastown and AI
agents. It gives branch and merge workflows on structured data.
https://llm.fzymgc.house/<server>/mcp, twelve routes, admission by virtual key in x-litellm-api-key: Bearer <key> (see MCP Gateway Clients)
Admin UI
https://llm.fzymgc.house/ui (generic OIDC to Keycloak SSO, client in tf/keycloak/litellm.tf)
Namespace
litellm
Ingress Type
Traefik IngressRoute. The host root is published with a denylist of excluded prefixes (/login, /register, /authorize, /token, the root .well-known documents) and a priority-100 allow for the per-server /.well-known/oauth-protected-resource/<server>/mcp documents
Auth Method
LLM and key-scoped MCP: a per-consumer virtual key (/key/generate; budgets, rate limits, object_permission.mcp_servers, or a litellm_unified_access_group for a key that Terraform owns). Identity MCP (engram, kubernetes, dns): the client’s own Keycloak token, relayed to the upstream (oauth_passthrough), obtained by the client through RFC 9728 discovery, RFC 7591 dynamic registration and PKCE
Vault Path
fzymgc-house/cluster/litellm (master key, OpenRouter keys, shared upstream and MCP credentials), fzymgc-house/cluster/litellm-keys/master, fzymgc-house/cluster/litellm/oidc, fzymgc-house/cluster/litellm-valkey, fzymgc-house/cluster/postgres/users/main-litellm; each consumer’s own virtual key on fzymgc-house/cluster/litellm-keys/<consumer>
State
CNPG main, database litellm (keys, spend, budgets). Auth cache in the litellm-valkey ValkeyCluster in the litellm namespace
Chart
litellm-helm1.105.0-rc.3, image pinned by digest (argocd/app-configs/litellm-chart/; secrets and ingress in argocd/app-configs/litellm/)
The unified LLM and MCP data plane (ADR
2026-08-15). One
OpenAI-compatible proxy fronts OpenRouter, Google Gemini and the MCP upstreams. Every consumer
authenticates with its own virtual key. The key carries a USD budget, a rate limit and, for MCP,
the set of routes it can reach. Spend is attributed per key alias.
LLM lanes (proxy_config.model_list in values.yaml; the lane name is what a client sends as
model): the consumer lanes octopus-embed, octopus-review, octopus-utility,
engram-embed, engram-summarize, gemini-embed and gpt-4o-mini; the passthrough wildcards openrouter/* and
openrouter-zdr/* (any OpenRouter slug, the second with zero data retention enforced by the
OpenRouter key of the lane); and the operator-only zdr-probe-embed. A key’s models list is its allow-list, and
LiteLLM checks it before aliases resolve, so every string a consumer sends must appear on its key.
See Minting and scoping virtual keys.
MCP routes (proxy_config.mcp_servers, twelve entries in three shapes): context7, exa,
firecrawl and fal (SaaS, LiteLLM injects the upstream key) and firewalla_ro,
clickhouse_ro and clickhouse_rw (in-cluster, LiteLLM injects the upstream bearer) overwrite
the client’s Authorization. engram, kubernetes and dns relay the client’s own Keycloak
token (oauth_passthrough: true) and hold no upstream credential. deepwiki (SaaS) and
unifi_ro (in-cluster) hold no upstream credential and no relay flag. Tool names carry the
server name as a prefix (context7-resolve-library-id). clickhouse_rw is the only route that
can change state, and only a key scoped to it can reach it.
Identity routes and discovery: an unauthenticated call to engram or kubernetes answers 401
with WWW-Authenticate: Bearer resource_metadata=…/.well-known/oauth-protected-resource/<server>/mcp.
That document is the upstream’s own RFC 9728 metadata, relayed by LiteLLM with only resource
rewritten. It names Keycloak as the authorization server. The client registers itself against
Keycloak (the loopback Trusted Hosts policy is the sole control) and completes PKCE. kubernetes-mcp
validates the relayed token against the realm’s JWKS with audience
https://llm.fzymgc.house/kubernetes/mcp, and the k3s API server authorizes it as
oidc:<email>.
Observability: LiteLLM emits GenAI metrics, logs and spans over OTLP to cs-otel-collector
(ClickStack): gen_ai.client.operation.duration, gen_ai.client.response.duration,
gen_ai.client.token.usage and gen_ai.usage.cost, labelled by metadata.user_api_key_alias.
The cost metric is a cumulative histogram that HyperDX cannot total, so the spend tiles and the
spend alert sum litellm.cost.total on the LiteLLM spans instead. The collector scrubs key material
(sk-[REDACTED]) from the Traefik log, the uvicorn log and the span attributes.
Client configuration (claude-code and codex):
// LLM: base URL https://llm.fzymgc.house/v1 , api key = your virtual key (api_key on cluster/litellm-keys/claude-code)
llm.fzymgc.house/engram/mcp (through LiteLLM, the client’s Keycloak token relayed)
Console URL
engram.fzymgc.house (LAN and Tailscale only; Traefik IngressRoute, vault-issuer TLS; the root redirects to /ui)
Namespace
agent-memory
Chart
ghcr.io/seanb4t/charts/engram, pinned by targetRevision in argocd/cluster-app/templates/agent-memory.yaml. Renovate moves it
Image
ghcr.io/seanb4t/engram (private GHCR package), pinned by version tag in argocd/app-configs/agent-memory/values.yaml. Argo CD Image Updater moves the pin, and the pull request waits for a review. See engram
Auth Method
MCP: OAuth JWT from Keycloak, issuer https://id.fzymgc.house/realms/fzymgc, audience mcp-public, validated by engram. Console: OIDC auth code through the Keycloak engram-ui confidential client, with its own ENGRAM_UI_ISSUER
Qdrant (qdrant-data PVC, longhorn-encrypted), collection memory_v4. A CronJob uploads Qdrant snapshots to B2 (fzymgc-k8s-backups, memory/qdrant-snapshots), and a dedicated Velero schedule backs the namespace up daily at 04:00
Persistent vector-memory store for the AI agent harness. It wraps Qdrant with an MCP server
(seanb4t/engram). Embeddings go through the LiteLLM engram-embed lane to gemini-embedding-2.
Summaries go through the engram-summarize lane. Typed decisions (memory.decisions, provider
Jev through the LiteLLM /openrouter pass-through) and the jev search ranker are on. They attach
advisory verdicts to engram spine-review consolidate pairs and rerank every search_memory and
search_discovery call. Record text leaves the cluster for TypeSafe without zero data retention.
The 2026-09-24 ADR accepts that.
Audit capture (memory.search.rerankAudit) is on for the Jev grading window, and off is the steady
state. Every reranked search writes one log line with the query text and the candidate ids, ranks
and scores to ClickStack. Record content never appears in the line, and no new data leaves the
cluster. This query shows the newest audit lines:
SELECTTimestamp, LogAttributes['surface'] AS surface, LogAttributes['outcome'] AS outcome,
LogAttributes['query'] AS query
FROMdefault.otel_logs
WHERE ServiceName ='engram'AND Body ='search rerank audit'
llm.fzymgc.house/kubernetes/mcp (through LiteLLM, the client’s Keycloak token relayed)
Namespace
kubernetes-mcp
Image
ghcr.io/containers/kubernetes-mcp-server:v0.0.66 (pinned by digest)
Auth Method
require_oauth = true, audience https://llm.fzymgc.house/kubernetes/mcp, validated against the realm’s JWKS. The k3s API server authorizes the caller as oidc:<email>
Mode
read_only = true
Manifests
argocd/app-configs/kubernetes-mcp/
The Deployment carries a checksum annotation of its ConfigMap, so a configuration change rolls
the pod. A test in tools/litellm-verify/ recomputes the value.
llm.fzymgc.house/firewalla_ro/mcp (through LiteLLM, a virtual key scoped to firewalla_ro)
Namespace
firewalla-mcp
Image
ghcr.io/fzymgc-house/firewalla-mcp, pinned by version tag in the kustomization.yaml of the app. Argo CD Image Updater moves the pin
Auth Method
Client: LiteLLM virtual key scoped to firewalla_ro (x-litellm-api-key). Gateway to server: a static bearer that LiteLLM injects. Server to MSP: a personal access token
Vault Path
fzymgc-house/cluster/firewalla-mcp (msp_id, token, box_id); the bearer the server validates is firewalla_mcp_auth_token on fzymgc-house/cluster/litellm
llm.fzymgc.house/unifi_ro/mcp (through LiteLLM, a virtual key scoped to unifi_ro)
Namespace
unifi-mcp
Image
ghcr.io/sirkirby/unifi-network-mcp:0.38.1 (pinned by digest, Renovate moves the pin)
Auth Method
Client: LiteLLM virtual key scoped to unifi_ro (x-litellm-api-key). Gateway to server: none. Only the litellm namespace reaches the pod. Server to controller: the UniFi local account automation-readonly
Vault Path
fzymgc-house/cluster/unifi-mcp (username, password), a copy of fzymgc-house/infrastructure/unifi/readonly
Lazy mode: unifi_tool_index and unifi_execute reach every read tool. Every create, update and delete tool is gated off by policy
Manifests
argocd/app-configs/unifi-mcp/
Read-only MCP server over the UniFi Network API of the UCG-Fiber at unifi.fzymgc.house, so that
an agent can triage devices, clients, firewall policies and events without the controller UI. The
design is in
UniFi Network MCP behind LiteLLM.
ghcr.io/seanb4t/life-agent-hermes (pinned by version tag), ghcr.io/seanb4t/life-agent-sandbox and ghcr.io/seanb4t/life-agent-smokescreen (pinned by the digest of the tag main). All three are signed with keyless cosign. Argo CD Image Updater moves the pins
https://drop.fzymgc.net behind Cloudflare Access over the cloudflared-main tunnel. The tunnel dials the ClusterIP http://hl-assets-uploader.hl-assets.svc.cluster.local:8080 with no Traefik hop
Namespace
hl-assets
Workloads
Deployment/hl-assets-uploader (replicas: 1, no autoscaler), Service/hl-assets-uploader (ClusterIP 8080)
Image
ghcr.io/fzymgc-house/hl-assets-uploader (private GHCR package, dual-arch), pinned by version tag in the kustomization.yaml of the app. Argo CD Image Updater moves the pin
Source
apps/hl-assets-uploader/ (Go module, cgo against libheif and libwebp)
Auth Method
The pod reads the Cloudflare Access headers without verifying them, and records an unauthenticated upload as such. The CiliumNetworkPolicy admits TCP/8080 from the cloudflared namespace only. Service-token client classes: share.fish, the curl one-liner, the iOS Shortcut, and the uptime monitor (its own pair)
Vault Paths
fzymgc-house/cluster/hl-assets/r2 (Secret hl-assets-r2), fzymgc-house/cluster/hl-assets/salt (Secret hl-assets-salt), fzymgc-house/cluster/hl-assets/access-token (the client-class pairs), fzymgc-house/cluster/hl-assets/monitor-token (the monitor’s pair), fzymgc-house/cluster/clickstack (Secret hl-assets-otlp-headers), fzymgc-house/cluster/ghcr/pull-secret
Egress
CiliumNetworkPolicy/hl-assets-egress: DNS, TCP/443 to world, and the ClickStack collector on 4318. Port-scoped, not host-scoped
Storage
None. Request-scoped spill on a disk-backed emptyDir at /spill, unlinked at creation. The root filesystem is read-only
Operator surface
Three takedown verbs on the same binary, run through kubectl exec: unshare (reversible), purge (irreversible, needs a confirmation token) and republish (the restore path). See Taking a link down
Monitoring
Two Uptime Kuma monitors, a keyword canary on i.hl-assets.dev/_health/canary and an authenticated /readyz probe, both page the cluster-edge Pushover app. Four R2 usage notification policies watch operation volume. See Monitoring
The write path for hl-assets.dev. A client POSTs a file. The pod strips metadata, transcodes
HEIC to WebP, derives a content-addressed object key, writes to R2 and returns the public URL in
one request. No client holds an R2 credential.
The read path is not this service. Cloudflare’s edge serves shared links from an R2 custom domain
with no origin in a public GET, so this pod has no effect on an existing link. See the
split read/write ADR and the
edge-policy ADR.
Endpoints:
Path
Purpose
POST /api/v1/upload
The only functional route. Multipart body
GET /healthz
Liveness. The process is alive
GET /readyz
Readiness. It makes no network call, so a dependency outage does not restart a healthy process
GET /statusz
Object-store reachability, in-flight reservation total, encoder versions. It calls R2, so no kubelet probe reads it
Upload contract:
201 Created is a fresh upload. The body is a JSON object with the public URL. An X-Token
header carries the capability for a later takedown. The pod issues it once, in a header only,
with no re-issue path.
200 OK is a deduplication hit. The same URL comes back, with no X-Token, and the pod does
not record the uploader’s filename.
Both carry an X-HL-Assets-Dedup header on both content types.
Accept: text/plain returns the bare URL plus one newline.
Every refusal is structured JSON with a code from a closed enum. Video and camera RAW get 415,
and the body names the detected type.
A non-POST request to the upload path gets 404 with Allow: POST.
scripts/share.fish drives the live upload path through Cloudflare Access and prints the public
URL. The Go suite in the image build holds the contract above. tools/hl-assets-verify/ covers the
operational items.
The salt is not rotatable. Rotating it raises no error and silently stops deduplication. See
docs/reference/secrets.md.
ghcr.io/fzymgc-house/dns-control (private GHCR package, dual-arch), pinned by version tag in the kustomization.yaml of the app and in the values.yaml of external-dns. Argo CD Image Updater moves both pins
Source
apps/dns-control/ (Go binary with an embedded Svelte UI)
Auth Method
Native OIDC in the binary against the Keycloak client dns-control (tf/keycloak/dns_control.tf). Every /api/* route requires the client role dns-reader, which the group dns-readers grants. The operations actions under /api/ops/ also require dns-operator, which the group dns-operators grants
Vault Paths
fzymgc-house/cluster/dns-control/oidc (oidc_client_secret, written by Terraform), fzymgc-house/infrastructure/dns/pdns-api (api_key), fzymgc-house/cluster/clickstack (dns_control_reader_password, and the OTLP header Secret dns-control-otlp-headers), fzymgc-house/cluster/ghcr/pull-secret, fzymgc-house/cluster/dns-control (nats_creds, mounted at /etc/dns-control/nats/dns-control.creds), fzymgc-house/cluster/postgres/users/main-dns-control (username and password, rendered into the database URL by the ExternalSecret dns-control-db with sslmode=verify-full)
Egress
CiliumNetworkPolicy/dns-control-default: kube-dns on 53, Keycloak on 8443, ClickStack on 8123 and 4318, the pdns API of 192.168.20.6 and 192.168.20.7 on 8081, the CNPG pods of the cluster main on 5432, the NATS servers on 4222 and the Temporal frontend on 7233. No world egress
Storage
None in the namespace. The record index lives in memory, and dns-control rebuilds it at each start. Velero excludes the namespace. The audit rows of the operations actions live in the CNPG database dns_control
dns-node-worker on each resolver Pi, installed by the Ansible role dns-node-worker. The unit dns-node-worker.service runs the worker as dns-worker and polls the Temporal queue dns-node-<node> through the node gateway. The unit dns-node-worker-pki.service runs Vault Agent, which renews the client certificate. The binary is the GitHub release dns-node-worker-v<version> of this repository
The control plane over the resolver pair. It serves Search, Name detail, Query log and Health. It
reads both pdns nodes by IP address and ClickHouse as dns_control_reader, and changes no record.
It runs the cache flush, the list refresh, the drain and the undrain of a node through the node
workers. It does not read the Terraform state of main-cluster-dns-records until issue #2413
adds a scoped token.
Publishes A and AAAA records for the Lab names under fzymgc.house. An IngressRoute gets a
record only when it carries the external-dns.alpha.kubernetes.io/target annotation. See
DNS records and
the PowerDNS authoritative tier.
Public ingress. The tunnel serves id.fzymgc.house, the *.fzymgc.net services and the
<service>-wh.fzymgc.net webhook hosts (argocd, discord, octopus, renovate).
Connection pooling for Keycloak. Keycloak connects through a CNPG Pooler (PgBouncer) instead of
main-rw, so the pooler absorbs a primary switchover and Keycloak keeps its JDBC socket. This
pairs with primaryUpdateMethod: switchover.
auto-provisioned (cnpg_pooler_pgbouncer role plus user_search)
The pool mode is session, not transaction. Transaction pooling breaks the session-scoped
state that Keycloak relies on (advisory locks during Liquibase migrations, prepared statements in
the JDBC pool).
The client-to-pooler hop requires TLS (client_tls_sslmode: require, encrypt only). The pooler
presents the cluster server certificate postgres-main-server-tls, whose SAN covers only
main-r, main-ro and main-rw, so a client cannot use sslmode=verify-full against the
pooler name.
CNPG does not pause PgBouncer on a switchover. For a planned switchover with zero connection
errors, set spec.pgbouncer.paused: true on the Pooler first, switch over, then set it back to
false.
Do not set conn_max_age or conn_health_checks on a pooled consumer. The session-mode pooler
already stabilizes the socket, and app-side persistent connections double-pool and pin session
slots.
The plans hold the only pin for the k3s version of the cluster. A bump to the pin upgrades the server
nodes first, one at a time, then the agent nodes. The controller cordons and drains each node during
its upgrade. Renovate proposes each bump in one attended pull request. The procedure is in
docs/operations/k3s.md.
The observability stack: logs, metrics and traces in ClickHouse, with HyperDX as the UI. The
tables are default.otel_logs, otel_metrics_* and otel_traces, with 30 days of history.
MongoDB and ClickHouse operators for the ClickStack stack
Bundles the MongoDB Community Operator and the Altinity ClickHouse operator. It reconciles the
custom resources in the clickstack namespace, so it must exist before the clickstack
Application.
Vector does not interpolate environment variables in its configuration, so a secret reaches the
configuration as a file: mount the Secret key under /etc/vector-secrets (extraVolumes) and
reference it as SECRET[clickhouse.<key>]. Check a change with a real vector run, not
vector validate. validate does not resolve the secrets backend and reports a false
authentication failure.
otel-gateway.fzymgc.house (OTLP gRPC ingress on 4317 for off-cluster emitters)
Namespace
otel-scraper
Chart
opentelemetry-collector0.174.2
Controllers
otel-scrape (Deployment: Prometheus scrape targets and the k8scluster receiver), otel-node (DaemonSet: hostmetrics and kubeletstats per node), otel-network (Deployment, three replicas: syslog from the UniFi gateway, switches and APs on the VIP 192.168.250.150)
Cilium network flow dashboard. It shows live pod-to-pod traffic, filters by namespace, verdict and
L7 attributes, and draws a service map. Use it during a CiliumNetworkPolicy rollout to make sure
that the intended flows pass and the others drop.
Create the Kubernetes manifests in argocd/app-configs/<service>/ and an Application in
argocd/cluster-app/templates/.
Configure the ingress (IngressRoute or Ingress). For a name that clients resolve on the LAN,
add the external-dns.alpha.kubernetes.io/target annotation (see
DNS records).
Set up authentication:
OIDC: create a Keycloak client in tf/keycloak/.
Forward-auth: add a Middleware to argocd/app-configs/oauth2-proxy/middlewares.yaml with an
allowed_groups=<app>-users gate.
Add secrets to Vault and update the Vault policy (tf/vault/).
Create an ExternalSecret for the Kubernetes Secret.