Skip to content

Design: live screens in dns-control

See openspec/changes/archive/2026-10-11-dns-control-live-screens/proposal.md for the reason and the scope. The specs of this change are under its specs/ folder there, and the main specs dns-control-live-events, dns-control-web-ui and dns-query-log now hold them. A Go path or a web path in this document is relative to apps/dns-control/. The facts that shape the design:

  • dns-control runs one replica. The sign-in sessions live in the memory of the process (internal/authn/session.go), and argocd/app-configs/dns-control/deployment.yaml says that a second replica needs sticky sessions.
  • The Temporal worker runs inside the dns-control process (cmd/dns-control/main.go). So the activities that end a change (Finalize, ExportSink in internal/change/activities.go) and an operations action (FinalizeAudit in internal/ops/activities.go) run in the process that serves the browser.
  • The node poller (internal/ops/poller.go) reads each node every 30 seconds. The index builder (internal/refresh/refresh.go) rebuilds every 30 seconds and publishes each index through an atomic pointer, whether or not its content changed.
  • internal/events publishes dns.change.* and dns.ops.* to NATS JetStream after a change or an action ends. Nothing in dns-control consumes them, and they carry no export state, no health reading and no index change.
  • The router is http.ServeMux behind withRequestTimeout (30 seconds), otelhttp and apierr.RecoverMiddleware. newServer sets ReadTimeout 15 seconds, WriteTimeout 150 seconds and IdleTimeout 120 seconds. statusTrackingWriter has Unwrap but no Flush.
  • RequireReader takes the session cookie first and a bearer token second, and verifies the token on each request. The web client sends the cookie only.
  • The IngressRoute of dns-control on websecure has no middleware. The websecure entry point serves HTTP/2 and HTTP/3, and the repository sets no Traefik timeout, so the defaults apply.

Goals:

  • One stream per tab, whatever the number of screens that tab visits.
  • No event carries data that the route it points to does not give the same person.
  • The load on ClickStack does not grow with the number of open tabs.

Non-Goals:

  • No second replica. The hub serves one process.
  • No change to an existing route but GET /api/querylog, which gains parameters and response fields and keeps the meaning of its old parameters. The MCP tools stay as they are.
  • No event history. A tab that missed events reads once when its stream opens again.
  • No change to the retry of the index-building state on Search and Name detail. It ends with the first index, about 30 seconds after a start.

Decision 1: One stream per tab, and an event is a signal to read

Section titled “Decision 1: One stream per tab, and an event is a signal to read”

The Shell opens one EventSource on /api/events once /api/me answers. An event names its type and carries ids and states only. The screen reads its existing route on the event, so each route keeps its role check, its shape and its tests.

Event Data Source
change id, subject, outcome, export_state Finalize and ExportSink
ops id, action, outcome FinalizeAudit
health none the end of each Poller.Poll
index none RebuildOnce, when the content digest changes
querylog newest, the time of the newest row the newest-row read
signed-out none the credential check at a heartbeat

Alternative: one stream per screen. Rejected, because a tab that moves between screens then closes and opens streams, and each screen needs its own route.

Alternative: events that carry the new data. Rejected, because each event type then needs its own serialization and its own role filter, and a slow read of a large answer blocks the stream.

A new package internal/live holds the hub. A source calls Publish, which never blocks. Each stream has a buffer of 16 events. If a buffer is full, the hub ends that stream, and the tab opens it again and reads once. A Temporal retry of an activity can publish an event twice, which costs one extra read.

Alternative: a NATS consumer of DNS_EVENTS. Rejected, because the subjects carry no export state, health or index event, and the hub is still necessary for those.

Alternative: Postgres LISTEN and NOTIFY. Rejected for now, because one process holds every source and every stream. If dns-control runs a second replica, the hub moves behind NOTIFY, and the event contract stays.

Decision 3: The route and the limits of the server

Section titled “Decision 3: The route and the limits of the server”

/api/events is registered through apiRoute, so it answers GET only, carries Cache-Control: no-store, and runs RequireReader at the start. The handler:

  1. Sets Content-Type: text/event-stream and writes retry: 3000.
  2. Before each write, sets a write deadline of 30 seconds from now, and flushes through the response controller. The controller reaches the connection through Unwrap, also behind statusTrackingWriter and otelhttp.
  3. Writes : ping every 15 seconds while no event is due.
  4. At each heartbeat, checks the credential again with the lookup of RequireReader: the session must exist and give a valid token, or the bearer token must still verify. On a failure, it writes signed-out and returns.

The handler does not clear its read deadline. The HTTP/1.1 server of Go 1.26 and Go 1.27 clears it when it starts the background read of a request with no body, so the 15-second ReadTimeout does not end a stream.

withRequestTimeout passes /api/events through without a deadline. otelhttp leaves the route out through otelhttp.WithFilter, so a stream of an hour is not one span of an hour and does not skew the duration metric of the requests. main.go registers hub.Close with srv.RegisterOnShutdown, so a shutdown ends every stream at once.

A Go test runs the full router behind a real http.Server with ReadTimeout 1 second and WriteTimeout 2 seconds, and reads a stream for 5 seconds. This is the proof that step 2 and the pass-through of withRequestTimeout work, because the timeouts of newServer are what cut a stream. Without step 2, the stream ends after 2 seconds.

Decision 4: The index sends an event only when its content changes

Section titled “Decision 4: The index sends an event only when its content changes”

RebuildOnce computes a sha256 digest over the zones, the record sets with their writers, the findings and the node agreement of the new index, in a fixed order, with no time field. It publishes index when the digest differs from the digest of the last build. The first build always differs, so a tab that waits for the first index gets the event.

Alternative: an event on every rebuild. Rejected, because Search then reads again every 30 seconds while a person reads the results.

Decision 5: One newest-row read for every stream

Section titled “Decision 5: One newest-row read for every stream”

The hub starts a loop when the first stream opens and stops it when the last stream ends. Every 15 seconds the loop runs this query through the reader of internal/querylog:

SELECT max(Timestamp) FROM default.otel_logs
WHERE ServiceName IN (<the ServiceNames of the query log reader>)
AND Timestamp > now() - INTERVAL 1 HOUR
SETTINGS log_comment = 'dns_control_querylog_newest'

The ServiceNames come from the same constant as the query of the query log, so a rename of the resolver services, as in the change dns-policy-as-rpz, changes both. The loop publishes querylog when the time is newer than the time of the last event. The first read after a start only sets that time.

lib/events.svelte.ts holds the one EventSource and a state: connecting, live, reconnecting or signed-out. A screen subscribes to its event types in an $effect, and the call returns the unsubscribe function.

  • On open, also after a reconnect, the client calls every subscriber once, so each screen reads once.
  • On error with readyState CONNECTING, the state is reconnecting, and the browser retries after 3 seconds.
  • On error with readyState CLOSED, which follows a 401 or a 403, the client reads /api/me. A 401 sends the app to its signed-out state through onaccess.
  • On signed-out, the client closes the stream and does the same.

A screen reads at most once per 500 milliseconds, so a burst of events costs one read. A hidden tab notes the event and reads once on visibilitychange. Health and Changes lose their timers. Their 1-second clocks for the ages stay.

StatusBar takes the state and shows Live, Reconnecting or Paused as text, with the ok tone for Live and the warn tone for the other two. Health shows the same text in its header, because it has no list.

Up to 0.21.0, the screen read the current filters again on each querylog event and replaced the rows. On 2026-10-11 Sean chose new rows above the rows on screen, a Pause control, and no more than one read every 30 seconds. The screen now works as follows:

  1. On a querylog event, the screen reads 30 seconds after the first event of a burst, so two live reads are at least 30 seconds apart. The subscription helper of Decision 6 takes the interval from the screen, so the other screens keep 500 milliseconds.
  2. The read sets after to the cursor of the newest row on screen. The rows of the answer go above the rows on screen. If the answer has an older cursor, more rows are new than one read holds, and the answer replaces the rows on screen.
  3. After each live read, the screen drops the rows older than the start of the range and the rows past 1,000. The cursor of the new last row is then the place of the next scroll read. Scroll reads stop at 1,000 rows, and the status bar says so.
  4. If the list is scrolled down, the screen adds the height of the new rows to scrollTop, so the rows on screen stay in place. The frame sets overflow-anchor: none. Chrome and Firefox anchor the scroll without help, but Safari does not, and one rule for every browser is easier to test.

While the screen is paused, it ignores querylog events and the status bar reads Paused. Resume reads once, as in step 2.

Alternative: pause the screen while the list is scrolled down. Sean chose the new rows above instead.

Alternative: replace the rows on each read, as 0.21.0 does. Rejected, because a replace drops the older rows that a scroll read.

GET /api/querylog sorts by Timestamp and then by a row key, both descending. The row key is cityHash64 of the node, the LogAttributes and the Body of the row. A cursor is the base64url form of the time of a row in nanoseconds and its key. before reads the rows below the cursor in that order, and after reads the rows above it. Both stay inside the range of the request, and the range starts at the time of each read. Each REST row carries its cursor, so the screen reads after its newest row and keeps a cursor for its new last row when it drops rows.

On 2026-10-11, 5,428 of the 21,292 rows of one hour shared their timestamp with another row, and up to 6 rows shared one timestamp. A cursor of the time alone skips or repeats those rows at the end of a page.

The screen asks for as many rows as two frame heights hold at 32 pixels a row, from 20 to 200. The frame height is the larger of the height of the frame and the room from its top to the bottom of the window, because a frame with no rows is as short as its header. A scroll handler on the frame starts the next read with before when the end of the rows comes within one frame height of the view. A read that leaves the frame short of rows starts one more. The check of the handler reads three numbers of the frame, and a test sets them, which a test cannot do for an IntersectionObserver in jsdom.

Alternative: offset with a fixed end time. Rejected, because a row that ClickHouse stores late moves every later offset, and a row then shows twice.

Two rows with the same time, node, attributes and body have one key, so one of them can drop at the end of a page. Such rows differ in nothing that the screen shows.

Decision 9: The filters and the name pattern

Section titled “Decision 9: The filters and the name pattern”

querylog.Query gains a name pattern, a result, a resolver and a node. The REST handler checks qname and mode as internal/httpapi/search.go does. It turns a glob into an anchored RE2 expression and adds (?i) to a regex. In glob mode it completes the trailing dot as the screen does, so qname=nas.fzymgc.house keeps its meaning. Go compiles the expression first, and ClickHouse then runs match() on the name with the expression as a parameter. ClickHouse uses RE2 too, so a pattern that Go accepts has the same meaning in ClickHouse.

result is an uppercase word of at most 16 letters, and type keeps the check of WithQType. resolver is one of the ServiceNames of the query log, and node is one of the nodes of legs.Config, which the answer also lists in nodes for the Node menu. The Result menu offers the five types that Blocky logs: Resolved, Cached, Conditional, Blocked and Special. The Type menu offers the common question types. Each menu adds its current value when the value is not in the list.

The exact QName of explain and of the MCP tool stays. An exact name is a glob with no operator, so a link from dns_query_log opens the same rows on the screen.

The client filter sits behind the funnel of the Client header, as a field with an Apply button, because a menu cannot list every client address. The name pattern stays the one text field above the table, as on Search.

The Result card opens from the Result label only, and holds the response code and the reason. The Name cell becomes the trigger of a Name card, which opens only when the cell is narrower than its name, so scrollWidth is greater than clientWidth. The Name cell loses its title.

Version 0.21.0 opened the Result card on a hover anywhere in the row. With the pointer on a name, the card showed under the Result label, far from the pointer, and the browser showed the title of the Name cell at the same time.

The REST handler parses range with time.ParseDuration and accepts whole minutes from 1 minute to 24 hours. 24 hours stays the bound of the scan of otel_logs. The screen sends 5m when the hash holds no range. An empty range on the route still reads as 1h, and the MCP tools keep their four keys, so an agent sees no change.

  • [Traefik or the browser cuts a stream that carries no event] → The heartbeat every 15 seconds, and the acceptance reads a stream through Traefik for 5 minutes.
  • [A burst of events makes every tab read many times] → Each screen reads at most once per 500 milliseconds, and an event carries no data.
  • [Health reads ClickHouse and Postgres on each read] → One health event every 30 seconds gives the same rate as the poll of today, and a hidden tab does not read.
  • [A source in another process does not reach the hub] → Not true today. The read on each open of the stream covers a lost event, and Decision 2 names NOTIFY for a second process.
  • [Sessions live in memory, so a deploy signs every tab out] → This is true today. The stream shows it within 30 seconds, not at the next click.
  • [The newest-row read fails] → The loop logs the error and tries again at the next tick, and the Query log keeps its rows. A submit of the filters still reads the log.
  • [A regex pattern makes ClickHouse test every name of the range] → The range bounds the scan to 24 hours, a pattern has at most 254 bytes, and RE2 does not backtrack.
  • [A busy hour adds about 175 rows to a live Query log every 30 seconds] → The list holds at most 1,000 rows, and the oldest rows go first.
  • [Two equal rows share one cursor] → One of them can drop at the end of a page. Decision 8 names the cost.

No data changes. The release and the pin follow docs/operations/releases.md. The pin is the only live change. To roll back, revert the pin commit and merge the revert:

Terminal window
git revert <commit of the pin pull request>

ArgoCD then syncs the image of the previous release, and Health and Changes poll again.