Firewalla MSP MCP Server Design

For Claude: This is a design spec. After a READY verdict from design-reviewer, the next step is dev-flow:writing-plans, then plan-to-beads. Tracking bead: hl-zply (designs implementation bead hl-vbvu.1, epic hl-vbvu).

Goal: Give cluster AI agents programmatic, queryable access to Firewalla state so DNS/firewall incidents (the class of hl-gy4x, which cost hours of manual SSH spelunking) can be triaged — and later managed — through the Model Context Protocol instead of by hand. v1 delivers read-only triage; scoped writes (rules + target-lists, the hl-12z4 enabler) follow on a separate route.

Architecture: A self-built Go MCP server (engram pattern: own repo, distroless image, CI-published) wraps the Firewalla MSP API v2. It is deployed in-cluster and exposed only through the existing agentgateway (mcp-gw.fzymgc.house), reusing the clickhouse-mcp deployment + routing pattern. Read and (future) write capabilities are split across two gateway routes, so the read-only blast-radius guarantee is enforced by topology, not by trusting in-server logic.

Tech Stack: Go + github.com/modelcontextprotocol/go-sdk (GA), StreamableHTTP transport, Task + goreleaser + cobra, distroless, ghcr.io/fzymgc-house/firewalla-mcp; ArgoCD + Kustomize; agentgateway (AgentgatewayBackend/HTTPRoute/AgentgatewayPolicy); External Secrets + Vault; Firewalla MSP API v2 (docs.firewalla.net).

---

1. Scope & Decisions

Decision	Choice	Rationale
Build vs adopt	Build (engram pattern)	Full control of tool surface + write-safety; no third-party code holding a firewall-management token. (Off-the-shelf amittell/firewalla-mcp-server exists and was considered.)
Hosting	In-cluster behind agentgateway	Matches every other cluster MCP server; available to all gateway clients (cloud agents + Claude Code) with central auth/audit.
Write scope (v1)	Read-only; scoped writes as fast-follow	Proves triage value and limits blast radius before exposing firewall mutation.
Repo	fzymgc-house/firewalla-mcp	Org-consistent with the cluster.
Auth	VK + static upstream bearer (clickhouse-style)	Sufficient for a read-only tool; lighter than engram’s per-user OIDC.
MSP hosting	Cloud-hosted (<domain>.firewalla.net)	Drives an internet-egress NetworkPolicy.

Non-goals (v1): write/mutation tools, per-user OIDC identity, a firewalla-rw route, the sibling triage skill (hl-vbvu.2), and codifying the ATF allow rule (hl-12z4) — all explicitly deferred.

2. Firewalla MSP API (verified against docs.firewalla.net, v2)

• Base URL: https://<msp_domain>/v2
• Auth: request header Authorization: Token <personal_access_token>
• Limits: 100 requests/minute; 30s timeout per request
• Read endpoints (v1 surface):
  • GET /v2/boxes — list Firewalla boxes
  • GET /v2/devices — list devices (filters: box, group)
  • GET /v2/alarms, GET /v2/alarms/:gid/:aid — alarms list + detail
  • GET /v2/flows — flows (query string: query, groupBy, sortBy, limit<=500, cursor)
  • GET /v2/rules — rules (e.g. query=box.id:<gid>)
  • GET /v2/target-lists — target lists
• Write endpoints (designed, deferred to fast-follow): POST /v2/rules (block/allow, MSP 2.10.0+), POST /v2/rules/:id/{pause,resume}, target-list create/update/delete, DELETE /v2/alarms/:gid/:aid, PATCH /v2/boxes/:gid/devices/:id.
• “DNS/ACL state” clarification: the MSP v2 API exposes no dedicated DNS/ACL endpoint. That policy is represented through rules (dnsOnly flag; target types domain/category) and target-lists. v1 surfaces it via list_rules / list_target_lists; there is no separate tool.

3. Server Repository (fzymgc-house/firewalla-mcp, engram pattern)

firewalla-mcp/
├── cmd/firewalla-mcp/        # cobra entrypoint; flags/env → config
├── internal/
│   ├── config/               # env parsing + validation (fail fast on missing MSP creds)
│   ├── msp/                  # typed MSP API v2 client (rate limit, retry, pagination)
│   ├── tools/                # MCP tool definitions (typed In/Out) → msp client calls
│   └── server/              # StreamableHTTP wiring, bearer-auth middleware, /health
├── Taskfile.yml              # build/test/lint/release tasks
├── .goreleaser.yaml          # distroless multi-arch image → ghcr.io/fzymgc-house/firewalla-mcp
└── ... (CI workflow, README, LICENSE)

• MCP SDK: github.com/modelcontextprotocol/go-sdk — mcp.NewServer(&mcp.Implementation{...}, nil), mcp.AddTool[In,Out] (JSON-schema inference from typed args), and the mcp.NewStreamableHTTPHandler(func(*http.Request) *mcp.Server, opts) constructor for HTTP transport (GA-idiomatic form; the bare &mcp.StreamableHTTPHandler{} struct also exists).
• Transport: StreamableHTTP served at /mcp, bind :8080. /health returns 200 on valid config + process health (it does not call the MSP API, to avoid burning the rate limit).
• Image: distroless, non-root, read-only root FS; pinned by digest in the cluster manifests.

4. MSP Client (internal/msp)

• Construct base https://<FIREWALLA_MSP_ID>/v2; attach Authorization: Token <token> on every request.
• Rate limiting: client-side token-bucket limiter at 100/min; 30s per-request timeout.
• Retry: bounded exponential backoff on 429 (honor Retry-After if present) and 5xx; never retry 4xx other than 429.
• Pagination: cursor-based helper following next_cursor for flows/alarms up to a caller-supplied cap.
• Errors: typed error values mapped to actionable messages (see §7). No silent failures.

5. Tool Surface (read-only v1)

Each tool is a typed AddTool[In, Out] handler delegating to the MSP client.

Tool	MSP call	Key inputs
list_boxes	GET /v2/boxes	—
list_devices	GET /v2/devices	box?, group?, online?
get_device	GET /v2/devices (filter by id)	id
list_alarms	GET /v2/alarms	query?, groupBy?, sortBy?, limit?, cursor?
get_alarm	GET /v2/alarms/:gid/:aid	gid, aid
query_flows	GET /v2/flows	query?, groupBy?, sortBy?, limit?, cursor?
list_rules	GET /v2/rules	query? (e.g. box.id:<gid>)
get_rule	GET /v2/rules (filter by id)	id
list_target_lists	GET /v2/target-lists	—
get_target_list	GET /v2/target-lists (filter by id)	id

Deferred write tools (fast-follow bead, exposed only on a future /mcp/firewalla-rw route): create_rule, pause_rule, resume_rule, create_target_list, update_target_list, delete_target_list.

6. Configuration & Secrets

Environment:

Var	Purpose
FIREWALLA_MSP_ID	MSP domain, e.g. <domain>.firewalla.net
FIREWALLA_MSP_TOKEN	MSP personal access token (PAT)
FIREWALLA_BOX_ID	default box GID (optional)
MCP_AUTH_TOKEN	static bearer the gateway injects (gateway→server auth)
HTTP_ADDR	listen address (default :8080)
LOG_LEVEL	log verbosity

Vault: secret/fzymgc-house/cluster/firewalla-mcp with properties msp_id, token, box_id. The static bearer (firewalla_mcp_auth_token) and the client VK (vk_firewalla) are stored under the existing agentgateway Vault path. No new Vault policy or k8s-auth role is required — the server never reads Vault directly; it consumes env from an ESO-materialized Secret, and the shared external-secrets-operator policy already grants wildcard secret/data/* read (confirmed: no per-app policy exists for clickhouse-mcp either). These three Vault properties are populated manually via the Vault CLI (the established pattern for agentgateway VK/upstream secrets — the LIVE-GATE ExternalSecrets stay unsynced until the property exists), not via Terraform.

ExternalSecret: argocd/app-configs/firewalla-mcp/external-secret.yaml materializes FIREWALLA_* into a namespace Secret consumed by the Deployment. The gateway-side bearer (agentgateway-mcp-firewalla) and the client VK (vk_firewalla) are added as new stanzas in argocd/app-configs/agentgateway/secrets.yaml — the established home for all agentgateway-mcp-* upstream-auth and VK ExternalSecrets, not the mcp-firewalla.yaml route file. Define both the Vault property name (vk_firewalla under fzymgc-house/cluster/agentgateway) and the rendered Kubernetes Secret key (e.g. firewalla-ro in the agentgateway-vkeys Secret), mirroring the vk_clickhouse_ro → clickhouse-ro convention.

7. Auth Model (clickhouse-style, three layers)

1. Client → gateway: AgentgatewayPolicy apiKeyAuthentication (Strict) against agentgateway-vkeys; client presents vk_firewalla.
2. Gateway → server: upstream AgentgatewayPolicy injects a static bearer (agentgateway-mcp-firewalla); our server’s middleware rejects any request missing it. This, plus a NetworkPolicy restricting ingress to the gateway namespace, prevents in-cluster bypass of the gateway to a server that holds firewall credentials.
3. Server → MSP: the MSP PAT (from Vault) authenticates the server to the Firewalla MSP API.

8. Cluster Wiring (mirrors clickhouse-mcp)

• Namespace: firewalla-mcp.
• argocd/app-configs/firewalla-mcp/: namespace.yaml, deployment.yaml (read-only; resources ~50m/64Mi req, 200m/256Mi lim; securityContext non-root, RO-rootfs, drop ALL), service.yaml (:8080), external-secret.yaml, networkpolicy.yaml, kustomization.yaml.
• argocd/cluster-app/templates/firewalla-mcp.yaml: ArgoCD Application with argocd.argoproj.io/sync-wave: "6" (mirrors clickhouse-mcp, so Vault/ESO come up first) and ignoreDifferences for ExternalSecret controller-injected fields (conversionStrategy, decodingStrategy, metadataPolicy) to avoid a perpetual OutOfSync.
• argocd/app-configs/agentgateway/mcp-firewalla.yaml: AgentgatewayBackend (static host firewalla-mcp.firewalla-mcp.svc.cluster.local:8080, path: /mcp, protocol: StreamableHTTP); HTTPRoute on the mcp-gw listener at /mcp/firewalla-ro; AgentgatewayPolicy for VK client auth + upstream bearer (the bearer/VK ExternalSecrets themselves live in secrets.yaml, per §6).
• Accept-list: add vk_firewalla to argocd/app-configs/agentgateway/secrets.yaml — both the Vault data source entry and the rendered VK-JSON template entry.
• NetworkPolicy: egress to the cloud MSP over HTTPS/443 (<domain>.firewalla.net) + DNS; ingress only from the agentgateway namespace.
• Velero: add firewalla-mcp to excludedNamespaces (stateless).
• Vault: populate secret/fzymgc-house/cluster/firewalla-mcp (msp_id/token/box_id) and add firewalla_mcp_auth_token + vk_firewalla to secret/fzymgc-house/cluster/agentgateway via the Vault CLI. No Terraform policy change (ESO wildcard read covers it).

9. Error Handling

• MSP errors map to MCP error results with actionable text: 401→“MSP token invalid/expired”; 429→“rate limited, retry after N”; 5xx→bounded retry then surface; timeout→explicit timeout error.
• Config validation fails fast at startup (missing FIREWALLA_MSP_ID/FIREWALLA_MSP_TOKEN ⇒ exit non-zero, no half-initialized server).
• No fallback that masks failures; tools never return empty-on-error.

10. Testing

• Unit (MSP client): httptest mock of v2 endpoints — auth header present, pagination cursor followed, 429/Retry-After handling, error mapping, timeout behavior.
• Unit (tools): typed In/Out + schema, delegating to a mocked client.
• Manifests: kustomize build argocd/app-configs/firewalla-mcp validates; kubeconform if available.
• Optional live integration: build-tagged test hitting a real MSP when FIREWALLA_MSP_* present; skipped in CI.

11. Rollout & Verification

1. CI in fzymgc-house/firewalla-mcp publishes ghcr.io/fzymgc-house/firewalla-mcp (digest-pinned).
2. Vault properties populated via CLI (no Terraform policy needed); ArgoCD syncs the app.
3. Verify: pod healthy (/health 200); from a gateway client, list tools at mcp-gw.fzymgc.house/mcp/firewalla-ro with vk_firewalla; call list_boxes → 200 with the box.
4. Outcome: read-only Firewalla triage is live for agents; unblocks the write fast-follow and the sibling triage skill (hl-vbvu.2).

12. Risks & Open Items

• MSP HTTP-transport server protection: our server is the trust boundary; the bearer middleware
  • NetworkPolicy ingress lock-down must both be in place before exposing it.
• Rate limit (100/min) is global per token: triage bursts (e.g. paging many flows) must respect the client-side limiter; document this for the triage skill.
• Egress to cloud MSP: depends on cluster egress to *.firewalla.net; confirm the exact MSP domain at implementation time for the NetworkPolicy.
• get_device/get_rule/get_target_list by id: the v2 API filters via query/list rather than always offering a /:id GET; the client may implement these as filtered list calls.