Skip to content
fzymgc-house Cluster Docs

Router-Hosts Monitoring Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Enable full observability (metrics, traces, alerts) for router-hosts on Firewalla.

Architecture: router-hosts exports OTEL to local Alloy, which relays to K8s Prometheus/Tempo. Grafana alerts monitor service health.

Tech Stack: Grafana Alloy, OpenTelemetry, Prometheus, GrafanaAlertRuleGroup CRD

Task 1: Add OTLP Receiver to Alloy

Section titled “Task 1: Add OTLP Receiver to Alloy”

Files:

  • Modify: ansible/roles/alloy/templates/config.alloy.j2

Step 1: Add OTLP receiver section

Insert after the // Metrics Collection section (around line 211), before // OTLP Export:

// =============================================================================
// OTLP Receiver (for local services like router-hosts)
// =============================================================================


// Receive OTEL from local services (router-hosts exports here)
otelcol.receiver.otlp "local" {
  grpc {
    endpoint = "127.0.0.1:4317"
  }
  output {
    metrics = [otelcol.processor.batch.default.input]
    traces  = [otelcol.processor.batch.default.input]
  }
}

Step 2: Update batch processor to handle traces

Find otelcol.processor.batch "default" (around line 238) and add traces output:

Change:

otelcol.processor.batch "default" {
  output {
    logs    = [otelcol.exporter.otlphttp.relay.input]
    metrics = [otelcol.exporter.otlphttp.relay.input]
  }
}

To:

otelcol.processor.batch "default" {
  output {
    logs    = [otelcol.exporter.otlphttp.relay.input]
    metrics = [otelcol.exporter.otlphttp.relay.input]
    traces  = [otelcol.exporter.otlphttp.relay.input]
  }
}

Step 3: Commit

Terminal window
git add ansible/roles/alloy/templates/config.alloy.j2
git commit -m "feat(alloy): add OTLP receiver for local services


Enables router-hosts and other local services to export OTEL
telemetry to Alloy for relay to the K8s observability stack.


- Add otelcol.receiver.otlp on localhost:4317
- Add traces output to batch processor"

Task 2: Add OTEL Export to router-hosts Config

Section titled “Task 2: Add OTEL Export to router-hosts Config”

Files:

  • Modify: ansible/roles/router-hosts/templates/server.toml.j2

Step 1: Add metrics.otel section

Append to end of file:

[metrics.otel]
endpoint = "http://127.0.0.1:4317"
service_name = "router-hosts"
export_metrics = true
export_traces = true

Step 2: Commit

Terminal window
git add ansible/roles/router-hosts/templates/server.toml.j2
git commit -m "feat(router-hosts): enable OTEL metrics and traces export


Configures router-hosts to export metrics and traces to local
Alloy instance via OTEL/gRPC.


Metrics available:
- router_hosts_requests_total
- router_hosts_request_duration_seconds
- router_hosts_storage_operations_total
- router_hosts_hook_executions_total
- router_hosts_hosts_entries"

Task 3: Create Grafana Alert Rules Directory

Section titled “Task 3: Create Grafana Alert Rules Directory”

Files:

  • Create: argocd/app-configs/router-hosts-alerts/kustomization.yaml
  • Create: argocd/app-configs/router-hosts-alerts/grafana-alerts.yaml

Step 1: Create kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization


resources:
  - grafana-alerts.yaml

Step 2: Create grafana-alerts.yaml with folder and alert group

---
# Grafana folder for router-hosts alerts
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaFolder
metadata:
  name: router-hosts-alerts
  namespace: grafana
spec:
  title: "Router Hosts Alerts"
  instanceSelector:
    matchLabels:
      dashboards: grafana
---
# router-hosts alert rules
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaAlertRuleGroup
metadata:
  name: router-hosts-alerts
  namespace: grafana
spec:
  folderRef: router-hosts-alerts
  instanceSelector:
    matchLabels:
      dashboards: grafana
  interval: 1m
  rules:
    # Critical: Service is down (no metrics received)
    - uid: router-hosts-down
      title: "Router Hosts Service Down"
      condition: B
      for: 5m0s
      execErrState: Alerting
      noDataState: Alerting
      annotations:
        summary: "router-hosts service is not reporting metrics"
        description: "No metrics received from router-hosts for 5 minutes. The service may be down or telemetry pipeline is broken."
      labels:
        severity: critical
        service: router-hosts
      data:
        - refId: A
          datasourceUid: prometheus
          relativeTimeRange:
            from: 600
            to: 0
          model:
            datasource:
              type: prometheus
              uid: prometheus
            expr: up{job="router-hosts"} OR vector(0)
            instant: true
            intervalMs: 1000
            refId: A
        - refId: B
          datasourceUid: __expr__
          relativeTimeRange:
            from: 600
            to: 0
          model:
            conditions:
              - evaluator:
                  params: [1]
                  type: lt
                operator:
                  type: and
                query:
                  params: [A]
                reducer:
                  params: []
                  type: last
                type: query
            datasource:
              type: __expr__
              uid: __expr__
            expression: A
            type: threshold
            refId: B


    # Critical: High error rate (>5% errors over 5 minutes)
    - uid: router-hosts-high-error-rate
      title: "Router Hosts High Error Rate"
      condition: C
      for: 5m0s
      execErrState: Error
      noDataState: NoData
      annotations:
        summary: "router-hosts gRPC error rate is high"
        description: "Error rate is above 5%. Current rate: {{ $values.A.Value | printf \"%.1f\" }}%"
      labels:
        severity: critical
        service: router-hosts
      data:
        - refId: A
          datasourceUid: prometheus
          relativeTimeRange:
            from: 300
            to: 0
          model:
            datasource:
              type: prometheus
              uid: prometheus
            expr: |
              100 * (
                sum(rate(router_hosts_requests_total{status="error"}[5m])) OR vector(0)
              ) / (
                sum(rate(router_hosts_requests_total[5m])) > 0 OR vector(1)
              )
            instant: true
            intervalMs: 1000
            refId: A
        - refId: C
          datasourceUid: __expr__
          relativeTimeRange:
            from: 300
            to: 0
          model:
            conditions:
              - evaluator:
                  params: [5]
                  type: gt
                operator:
                  type: and
                query:
                  params: [A]
                reducer:
                  params: []
                  type: last
                type: query
            datasource:
              type: __expr__
              uid: __expr__
            expression: A
            type: threshold
            refId: C


    # Warning: Hook execution failures
    - uid: router-hosts-hook-failures
      title: "Router Hosts Hook Failures"
      condition: C
      for: 5m0s
      execErrState: Error
      noDataState: NoData
      annotations:
        summary: "router-hosts post-edit hooks are failing"
        description: "Hook failures detected in the last 5 minutes. DNS updates may not be taking effect."
      labels:
        severity: warning
        service: router-hosts
      data:
        - refId: A
          datasourceUid: prometheus
          relativeTimeRange:
            from: 300
            to: 0
          model:
            datasource:
              type: prometheus
              uid: prometheus
            expr: increase(router_hosts_hook_executions_total{status="error"}[5m])
            instant: true
            intervalMs: 1000
            refId: A
        - refId: C
          datasourceUid: __expr__
          relativeTimeRange:
            from: 300
            to: 0
          model:
            conditions:
              - evaluator:
                  params: [0]
                  type: gt
                operator:
                  type: and
                query:
                  params: [A]
                reducer:
                  params: []
                  type: last
                type: query
            datasource:
              type: __expr__
              uid: __expr__
            expression: A
            type: threshold
            refId: C


    # Warning: Certificate expiring soon (file mtime > 23 days)
    # Note: This is a heuristic - native cert metrics planned (router-hosts#236)
    - uid: router-hosts-cert-expiring
      title: "Router Hosts Certificate Expiring Soon"
      condition: C
      for: 1h0m0s
      execErrState: Error
      noDataState: NoData
      annotations:
        summary: "router-hosts TLS certificate may be expiring soon"
        description: "Certificate file has not been renewed in over 23 days (30-day TTL). Check Vault Agent certificate renewal."
      labels:
        severity: warning
        service: router-hosts
      data:
        - refId: A
          datasourceUid: prometheus
          relativeTimeRange:
            from: 3600
            to: 0
          model:
            datasource:
              type: prometheus
              uid: prometheus
            expr: |
              (time() - node_textfile_mtime_seconds{file=~".*router-hosts.*cert.*"}) / 86400
            instant: true
            intervalMs: 1000
            refId: A
        - refId: C
          datasourceUid: __expr__
          relativeTimeRange:
            from: 3600
            to: 0
          model:
            conditions:
              - evaluator:
                  params: [23]
                  type: gt
                operator:
                  type: and
                query:
                  params: [A]
                reducer:
                  params: []
                  type: last
                type: query
            datasource:
              type: __expr__
              uid: __expr__
            expression: A
            type: threshold
            refId: C

Step 3: Commit

Terminal window
git add argocd/app-configs/router-hosts-alerts/
git commit -m "feat(alerts): add router-hosts Grafana alert rules


Alerts for router-hosts service health:
- RouterHostsDown: no metrics for 5m (critical)
- RouterHostsHighErrorRate: >5% gRPC errors (critical)
- RouterHostsHookFailures: DNS reload failures (warning)
- RouterHostsCertExpiring: cert file >23 days old (warning)


Note: Cert alert uses file mtime heuristic pending router-hosts#236"

Task 4: Register Alert App with ArgoCD

Section titled “Task 4: Register Alert App with ArgoCD”

Files:

  • Create: argocd/cluster-app/templates/router-hosts-alerts.yaml

Step 1: Create Helm template for Application

Create argocd/cluster-app/templates/router-hosts-alerts.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: router-hosts-alerts
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  project: core-services
  sources:
    - repoURL: https://github.com/fzymgc-house/selfhosted-cluster
      targetRevision: HEAD
      path: argocd/app-configs/router-hosts-alerts
  destination:
    server: https://kubernetes.default.svc
    namespace: grafana
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true

Step 2: Commit

Terminal window
git add argocd/cluster-app/templates/router-hosts-alerts.yaml
git commit -m "feat(argocd): register router-hosts-alerts application


Enables GitOps sync of router-hosts Grafana alerts to cluster.
Uses sync-wave 2 to ensure Grafana Operator is ready first."

Task 5: Deploy and Verify Alloy Changes

Section titled “Task 5: Deploy and Verify Alloy Changes”

Step 1: Deploy Alloy config to router

Terminal window
cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster
ansible-playbook -i ansible/inventory/hosts.yml ansible/router-playbook.yml --tags alloy --check --diff
# Review changes, then apply:
ansible-playbook -i ansible/inventory/hosts.yml ansible/router-playbook.yml --tags alloy

Step 2: Verify OTLP receiver is listening

Terminal window
ssh pi@192.168.20.1 "docker logs alloy 2>&1 | grep -i 'otlp\|4317' | tail -5"

Expected: Log lines showing OTLP receiver started on 127.0.0.1:4317

Step 3: Commit verification notes

No commit needed - deployment verification only.

Task 6: Deploy and Verify router-hosts Changes

Section titled “Task 6: Deploy and Verify router-hosts Changes”

Step 1: Deploy router-hosts config

Terminal window
ansible-playbook -i ansible/inventory/hosts.yml ansible/router-playbook.yml --tags router-hosts --check --diff
# Review changes, then apply:
ansible-playbook -i ansible/inventory/hosts.yml ansible/router-playbook.yml --tags router-hosts

Step 2: Restart router-hosts to pick up config

Terminal window
ssh pi@192.168.20.1 "cd /home/pi/.firewalla/run/docker/router-hosts && docker compose restart router-hosts"

Step 3: Verify OTEL export

Terminal window
ssh pi@192.168.20.1 "docker logs router-hosts 2>&1 | grep -i 'otel\|metrics' | tail -5"

Expected: Log lines showing OTEL exporter initialized

Task 7: Verify End-to-End Telemetry

Section titled “Task 7: Verify End-to-End Telemetry”

Step 1: Check metrics in Prometheus

Open Grafana → Explore → Prometheus datasource:

router_hosts_requests_total

Expected: Metric with method/status labels

Step 2: Check traces in Tempo

Open Grafana → Explore → Tempo datasource:

  • Search: service.name = "router-hosts"

Expected: Traces showing gRPC operations

Step 3: Verify alerts registered

Open Grafana → Alerting → Alert rules

  • Look for “Router Hosts Alerts” folder
  • Verify 4 rules present

Task 8: Create PR

Section titled “Task 8: Create PR”

Step 1: Push branch

Terminal window
git push -u origin feat/router-hosts-monitoring

Step 2: Create PR

Terminal window
gh pr create --title "feat(observability): add router-hosts monitoring and alerting" --body "## Summary
- Enable OTEL telemetry export from router-hosts to Alloy
- Add OTLP receiver to Alloy for local services
- Create Grafana alert rules for service health


## Alerts Added
- **RouterHostsDown** (critical): No metrics for 5m
- **RouterHostsHighErrorRate** (critical): >5% gRPC errors
- **RouterHostsHookFailures** (warning): DNS reload failures
- **RouterHostsCertExpiring** (warning): Cert file >23 days old


## Testing
- [ ] Alloy receives OTLP on localhost:4317
- [ ] router-hosts exports metrics/traces
- [ ] Metrics visible in Prometheus
- [ ] Traces visible in Tempo
- [ ] Alerts registered in Grafana


Closes: selfhosted-cluster-gon
Related: fzymgc-house/router-hosts#236"

Verification Checklist

Section titled “Verification Checklist”
  • Alloy OTLP receiver listening on 127.0.0.1:4317
  • router-hosts exporting to OTEL endpoint
  • router_hosts_requests_total visible in Prometheus
  • Traces with service.name=router-hosts in Tempo
  • 4 alert rules in “Router Hosts Alerts” folder
  • Alerts in OK state (no false positives)