Operations
Each page in this section states the current state of one part of the estate, the procedures that touch it, and a check for each procedure.
Network and gateway
Section titled “Network and gateway”- Internal DNS: the resolver pair, its floating addresses, the PowerDNS tier, Kea and DDNS, and the UCG-Fiber gateway beside it.
- dns-control: the read-only control plane over the resolver pair, its sign-in, its image pin, and the recovery for each data alert.
- Lab Service VLAN: the tagged Lab leg on every node and the dual-stack LoadBalancer VIPs on it.
- Tailscale Subnet Routers: the three tailnet nodes that advertise the LAN routes, and how to change or check a route.
- Cilium: the CNI, its settings and where each is set, and the network policies.
- Cilium kube-proxy Replacement: the settings that replace kube-proxy, and the commands that replace the iptables workflow.
- UniFi: the switching fabric, what
tf/unifimanages, and the controller settings that can veto work elsewhere. - Terraform Provider Fork (UniFi): our UniFi provider fork, how to consume it, and how to release it to the private registry.
- heimdall: the out-of-band recovery host, its role, its gated self-reboot, and the rescue path.
Platform
Section titled “Platform”- Vault: the secret store, its auth methods, the PKI, and the procedures to unseal, rotate and add a path.
- Cloudflare: the DNS records, zones, tunnels, Access applications, service tokens and R2 buckets that
tf/cloudflareowns. - HCP Terraform: the workspaces, how a pull request plans and a merge applies, and the procedures to read and fix a run.
- cluster-bootstrap: the
tf/cluster-bootstrapmodule that installs cert-manager, External Secrets, Longhorn and ArgoCD. - ArgoCD: GitOps deployments, application sync, and the GitHub webhook.
- etcd Snapshot Restore: how to take a control-plane backup and restore it on the first server.
- Longhorn: the block storage, its classes, the backup target, and the procedures to add a disk and restore a volume.
- GitHub Credential Operations: the GitHub Apps and OAuth clients the estate uses, where each credential lives, and how to rotate one.
- Renovate: the dependency-update worker, the constraints on its configuration, and the procedures an operator runs.
- App Releases: how dns-control and hl-assets-uploader get a version, the changelog fragments, and the release pull request.
- Outbound Email: how applications send email through Cloudflare Email Sending with one token per app.
- Pushover: the mobile notification provider and the applications that send through it.
Observability
Section titled “Observability”- ClickStack: the logs, metrics and traces backend, its tables, its feeders, and its query forms.
- Alerting: the HyperDX alerts, how they are defined in Git, and how they reach Pushover.
- Hubble UI: the Cilium flow dashboard and who can reach it.
- ClickStack vs Loki+Prom Comparison: a pointer to ClickStack. The stack it compared no longer runs.
Applications and AI
Section titled “Applications and AI”- LiteLLM: the LLM proxy, its lanes, its virtual keys and their owners, and the procedures for keys and credentials.
- MCP Gateway Clients: how Claude Code and Claude Desktop connect to the MCP servers behind
llm.fzymgc.house. - Headroom: the proxy that compresses LLM request context for Claude Code.
- Life agent: the personal Hermes agent, its egress proxies, and how to read their decisions.
- Octopus: the self-hosted pull-request review bot, its configuration, and how to read a review record.
- Home Assistant: calgary, the host of the Home Assistant VM, its connection model, its reboot policy, and the failure domains of the lighting.
- Temporal: workflow orchestration, the workers, and the task queues.
- NATS: the messaging cluster, its accounts, and its operations.
- hl-assets Uploader: the upload endpoint at
drop.fzymgc.net, its four clients, and how to take a link down. - RustDesk Clients: how to configure a desktop or mobile client for the self-hosted RustDesk server.
- Qdrant Memory Backup and Restore: the two backup paths for the memory store and how to restore from either.
- NAS: the TrueNAS host under Ansible, its roles, its custom apps, and the connection model.
- RustFS: the S3 object store on the NAS, its endpoints and consumers, and its backup.
Records
Section titled “Records”- Incidents and Corrections of Error: the incident reports.
Common Operations
Section titled “Common Operations”| Task | Guide |
|---|---|
| Rotate a secret | Vault Operations |
| Update DNS records | Cloudflare DNS |
| Run Terraform | HCP Terraform |
| Execute a workflow | Temporal Operations |
| Check cluster health | Temporal Operations |