Skip to content

Operations

Each page in this section states the current state of one part of the estate, the procedures that touch it, and a check for each procedure.

  • Internal DNS: the resolver pair, its floating addresses, the PowerDNS tier, Kea and DDNS, and the UCG-Fiber gateway beside it.
  • dns-control: the read-only control plane over the resolver pair, its sign-in, its image pin, and the recovery for each data alert.
  • Lab Service VLAN: the tagged Lab leg on every node and the dual-stack LoadBalancer VIPs on it.
  • Tailscale Subnet Routers: the three tailnet nodes that advertise the LAN routes, and how to change or check a route.
  • Cilium: the CNI, its settings and where each is set, and the network policies.
  • Cilium kube-proxy Replacement: the settings that replace kube-proxy, and the commands that replace the iptables workflow.
  • UniFi: the switching fabric, what tf/unifi manages, and the controller settings that can veto work elsewhere.
  • Terraform Provider Fork (UniFi): our UniFi provider fork, how to consume it, and how to release it to the private registry.
  • heimdall: the out-of-band recovery host, its role, its gated self-reboot, and the rescue path.
  • Vault: the secret store, its auth methods, the PKI, and the procedures to unseal, rotate and add a path.
  • Cloudflare: the DNS records, zones, tunnels, Access applications, service tokens and R2 buckets that tf/cloudflare owns.
  • HCP Terraform: the workspaces, how a pull request plans and a merge applies, and the procedures to read and fix a run.
  • cluster-bootstrap: the tf/cluster-bootstrap module that installs cert-manager, External Secrets, Longhorn and ArgoCD.
  • ArgoCD: GitOps deployments, application sync, and the GitHub webhook.
  • etcd Snapshot Restore: how to take a control-plane backup and restore it on the first server.
  • Longhorn: the block storage, its classes, the backup target, and the procedures to add a disk and restore a volume.
  • GitHub Credential Operations: the GitHub Apps and OAuth clients the estate uses, where each credential lives, and how to rotate one.
  • Renovate: the dependency-update worker, the constraints on its configuration, and the procedures an operator runs.
  • App Releases: how dns-control and hl-assets-uploader get a version, the changelog fragments, and the release pull request.
  • Outbound Email: how applications send email through Cloudflare Email Sending with one token per app.
  • Pushover: the mobile notification provider and the applications that send through it.
  • ClickStack: the logs, metrics and traces backend, its tables, its feeders, and its query forms.
  • Alerting: the HyperDX alerts, how they are defined in Git, and how they reach Pushover.
  • Hubble UI: the Cilium flow dashboard and who can reach it.
  • ClickStack vs Loki+Prom Comparison: a pointer to ClickStack. The stack it compared no longer runs.
  • LiteLLM: the LLM proxy, its lanes, its virtual keys and their owners, and the procedures for keys and credentials.
  • MCP Gateway Clients: how Claude Code and Claude Desktop connect to the MCP servers behind llm.fzymgc.house.
  • Headroom: the proxy that compresses LLM request context for Claude Code.
  • Life agent: the personal Hermes agent, its egress proxies, and how to read their decisions.
  • Octopus: the self-hosted pull-request review bot, its configuration, and how to read a review record.
  • Home Assistant: calgary, the host of the Home Assistant VM, its connection model, its reboot policy, and the failure domains of the lighting.
  • Temporal: workflow orchestration, the workers, and the task queues.
  • NATS: the messaging cluster, its accounts, and its operations.
  • hl-assets Uploader: the upload endpoint at drop.fzymgc.net, its four clients, and how to take a link down.
  • RustDesk Clients: how to configure a desktop or mobile client for the self-hosted RustDesk server.
  • Qdrant Memory Backup and Restore: the two backup paths for the memory store and how to restore from either.
  • NAS: the TrueNAS host under Ansible, its roles, its custom apps, and the connection model.
  • RustFS: the S3 object store on the NAS, its endpoints and consumers, and its backup.
Task Guide
Rotate a secret Vault Operations
Update DNS records Cloudflare DNS
Run Terraform HCP Terraform
Execute a workflow Temporal Operations
Check cluster health Temporal Operations