LiteLLM D-50 Migration-Job Churn Cessation

Measured: 2026-08-30T07:18:35Z Requirement: OBS-01. Phase: 03-llm-key-management, plan 03-21 (gap closure over 03-04). Detector: tools/litellm-verify/tests/test_val01_arch_otel_prisma.py::test_migration_job_is_not_recreated_hourly, selected by the migration_churn marker. Command: LITELLM_REQUIRE_LIVE=1 task litellm-verify:live -- -k migration_churn → 1 passed, 99 deselected.

What this records

03-04 measured the before-number honestly and refused to claim the after-number, stating that it could only come from this detector once the fix merged and ArgoCD synced. This is that number.

A clean kubectl get pods is NOT this measurement and must not be substituted for it. A completed Job that keeps reappearing is invisible to a point-in-time listing: each incarnation ran for roughly 22 seconds and the TTL controller deleted it again, so the pre-fix cluster showed nothing wrong at any instant while opening ~300 Prisma connections a day against the production CNPG instance. Only a query over a trailing window sees the recreations.

Provenance — what the number was measured against

Merge carrying ttlSecondsAfterFinished: null	1dc6fe3cf891c577bf5b57f7735eb8db1ab7ef73 — feat(litellm): LiteLLM Return milestone — phases 1-2 complete, phase 3 waves 1-4 (#1860), 2026-08-16T14:41:27Z, confirmed an ancestor of origin/main
Live at	argocd/app-configs/litellm-chart/values.yaml:100
ArgoCD litellm Application	Synced / Healthy; last sync operation Succeeded, started 2026-08-29T19:27:05Z, finished 2026-08-29T19:28:51Z; reconciledAt 2026-08-30T07:14:34Z
Elapsed sync → reading	11h 50m — the trailing 6h window is entirely post-sync
Deployment path	ArgoCD only. No kubectl apply was used to place the change on the cluster

The running Job reflects the synced fix: kubectl get job litellm-migrations -n litellm reports .spec.ttlSecondsAfterFinished unset, not 120.

The measurement

Both numbers are from the same run at 2026-08-30T07:18:35Z, over the same trailing CHURN_WINDOW_HOURS = 6 window. They are meaningless apart: the churn count alone cannot distinguish “the loop stopped” from “the telemetry stopped”.

d50-churn-cadence — distinct litellm-migrations-* pods per hour over the trailing 6h: no migration pods in the window (peak 0, threshold 1); 12978 litellm log rows in the same window

	Pre-fix (2026-08-16)	Post-fix (2026-08-30)
Per-hour distinct litellm-migrations-* pods	8, 16, 12, 12, 13, 13, 12, 12, 13, 15, 12, 14, 15, 14, 12, 14, 12, 12, 12, 12, 12, 12, 12 (24h)	(empty series — none in the window)
Peak	16/hour	0/hour
Threshold CHURN_MAX_PODS_PER_HOUR	1	1
Verdict	RED	PASS

The detector’s own series is literally empty. Expanded per hour, with the vacuity guard’s rows broken out over the same window by the same helper (supplementary query, 2026-08-30T07:19Z):

Hour (UTC)	litellm-migrations-* pods	litellm namespace log rows
2026-08-30 01:00	0	1,149
2026-08-30 02:00	0	1,452
2026-08-30 03:00	0	2,112
2026-08-30 04:00	0	1,501
2026-08-30 05:00	0	1,455
2026-08-30 06:00	0	3,736
2026-08-30 07:00 (partial)	0	1,575

Why the window is long enough

At the pre-fix cadence of 12-16 recreations per hour sustained across a full day, a six-hour window would contain roughly 72-96 distinct migration pods. It contains zero. The loop cannot have been running during this window and produced this reading.

The vacuity guard is satisfied continuously, not in a burst: every one of the seven hour-buckets carries between 1,149 and 3,736 litellm log rows, so the pipeline delivered throughout the window rather than in a single spike that a total alone would hide.

The two positive controls, which are not the same control

namespace_rows proves the pipeline delivers. It does not prove the litellm-migrations-% LIKE matcher could still match a migration pod name — a matcher whose subject-format assumption has drifted returns a flattering zero against a live pipeline. This estate has the precedent: a generic matcher on this same telemetry once returned zero against 284 real NATS [ERR] lines.

03-04’s pre-fix 24h series IS the litellm-migrations-% MATCHER’s positive control. The same detector, the same litellm-migrations-% prefix and the same k8s.pod.name subcolumn took 8, 16, 12, 12, 13, 13, 12, 12, 13, 15, 12, 14, 15, 14, 12, 14, 12, 12, 12, 12, 12, 12, 12 distinct pods per hour on 2026-08-16 — the matcher has been shown to read a known positive, on this cluster, against this schema.

That control is re-confirmed contemporaneously, not merely cited from history: the identical query shape widened to a trailing 336h window on 2026-08-30 returns 91 distinct litellm-migrations-* pod names, the most recent being litellm-migrations-phr5r at 2026-08-29T19:28:14Z. The matcher matches migration pod names today, against today’s schema, on today’s ingest path. A zero over six hours is therefore a zero the matcher was capable of exceeding.

Attribution

Cessation is attributed to two config properties, each held statically by a named gate:

Property	Gate	Lane
migrationJob.ttlSecondsAfterFinished is null, so no TTL applies and the completed Job persists	test_migration_job_has_no_ttl	tests/test_phase03_config_gates.py
The migration Job is ordered by sync wave, not by a PreSync hook (which is measured to deadlock a cold install)	test_migration_job_ordered_by_wave_not_presync_hook	tests/test_fnd_static_gates.py

Both pass in task litellm-verify:gates (34 passed). The causal chain they break: the TTL controller deleted the Job two minutes after completion → ArgoCD’s selfHeal saw a managed resource missing → the Replace=true sync option recreated it → every reconcile, forever.

Limitation: a 24h window would read peak 3, and why that is not churn

Reported rather than smoothed. The same query over a trailing 24h window returns peak 3, above the threshold of 1. Seven migration pods appear, all on 2026-08-29 between 17:42Z and 19:28Z, and every one of them sits 30-90 seconds behind a merge that changed the LiteLLM chart config:

Merge (UTC)	Commit	Migration pod	First log
17:35:37	dd474d875	litellm-migrations-2ch8w	17:42:36
17:43:09	fd4fd2629	litellm-migrations-n9qk4	17:44:05
17:43:23	c6fd722ff	litellm-migrations-dgp9b	17:45:22
18:16:28	ad88166df	litellm-migrations-f2ftp	18:17:30
—	(unattributed)	litellm-migrations-s4hg6	18:18:53
19:06:54	9193ee883	litellm-migrations-fprdr	19:07:46
19:26:58	a774fa3f7	litellm-migrations-phr5r	19:28:14

Six config-changing merges in 111 minutes produced seven recreations. This is Replace=true doing what it is supposed to do — recreate the Job when the values checksum changes — not the reconcile loop. The intervals are irregular (2min, 1min, 32min, 49min, 20min); the pre-fix loop was clockwork at a fixed 5min, then 3.2min, matching ArgoCD’s reconcile interval rather than any human action.

Two honest residuals fall out of this:

1. One pod is unattributed. litellm-migrations-s4hg6 starts 83 seconds after f2ftp with no merge between them. The likeliest reading is a Job-level retry producing a second pod from one Job — which the detector cannot distinguish, because it counts distinct pod names, not Job UIDs.
2. The threshold’s own assumption is what a deploy burst exceeds. CHURN_MAX_PODS_PER_HOUR = 1 encodes “a legitimate values-checksum change recreates the Job exactly once”, which silently assumes at most one config change per hour. Six merges in one hour breaks that assumption, not the fix. The threshold was not raised — raising it to make a run pass is prohibited; a failing peak is recorded as the result.

Reopen condition. If a 6h run ever reports a peak above 1 that cannot be matched one-for-one to a merge touching argocd/app-configs/litellm-chart/, the loop has resumed and D-50 reopens. If a run reports peak > 1 during a deploy burst, correlate against git log on that path before concluding anything.

Corroboration, not the measurement

Stated separately because it must not be mistaken for the number. The Job created at 2026-08-29T19:27:12Z completed at 19:28:50Z, and its pod litellm-migrations-phr5r was still present in kubectl get pods -n litellm at 2026-08-30T07:18Z — 11h 50m later. Under the pre-fix ttlSecondsAfterFinished: 120 it would have been deleted at ~19:30:50Z. The TTL controller is demonstrably not firing. This corroborates the mechanism; the trailing-window count above is the evidence.

Standing detection

This detector is what would surface a regression, by marker:

export CLICKHOUSE_PASSWORD=$(vault kv get -field=mcp_readonly_password secret/fzymgc-house/cluster/clickstack)
LITELLM_REQUIRE_LIVE=1 task litellm-verify:live -- -k migration_churn

It needs the mcp_readonly reader credential, so it runs on operator demand rather than in CI. LITELLM_REQUIRE_LIVE=1 converts a missing credential from a BLOCK into a FAILURE — without it the run exits 0 having measured nothing, which reads as a clean measurement. The credential travels in the environment and never in argv, a URL, or a committed file.

The migration_churn marker exists because -k matches marker names and migration_churn is not a substring of the test’s node id (test_migration_job_is_not_recreated_hourly). Without the marker the documented selector would select nothing and report a green zero-test run.