[critical] [daily-error-review] workers.restart_count (1 → 3; service instability)

[viktor-shcherb]

Summary

The worker-3 Docker restart counter increased from 1 in the 2026-07-09 review to 3 in the 2026-07-10 snapshot, meeting the daily-review host incident threshold. All three HTTP workers were healthy at collection time, so the impact is intermittent rather than an active full outage. Window: 2026-07-09 09:08 UTC → 2026-07-10 09:08 UTC. A deployment recreated the crawler stack around 23:13 UTC, which limits causal attribution and makes container-generation continuity uncertain.

Signal

day	restart-count snapshot for worker-3
-4	unavailable
-3	unavailable
-2	unavailable
-1	1
today	3

The unit is Docker's cumulative RestartCount at collection time, not a log-event count. Today's bundle also shows worker-1=1 and worker-2=1, no OOM kills, and all expected services running.

Sample log line

{"service":"worker-3","OOMKilled":false,"status":"running","restart_count":3,"finished_at":"2026-07-09T23:40:49Z"}

Root cause hypothesis

1. The health check or restart policy in apps/crawler/docker-compose.yml may restart a saturated worker during post-deploy load; correlate health-check timing and worker startup/shutdown logging.
2. The worker process path under apps/crawler/src/workers/ may exit without an application-level fatal line, leaving Docker as the only restart evidence; improve termination-cause logging.

Impact

Transient loss of one HTTP worker's capacity and possible claim delays during restarts. No OOM, missing service, or sustained zero-log condition was observed at collection time.

[viktor-shcherb]

2026-07-11 recurrence

Window: 2026-07-10 09:04 UTC → 2026-07-11 09:04 UTC.

All three HTTP-worker restart counters increased from the prior review snapshot:

worker	2026-07-10	2026-07-11	last finish UTC
worker-1	1	4	2026-07-10 17:20:28
worker-2	1	3	2026-07-10 17:10:04
worker-3	3	4	2026-07-10 17:40:10

All expected services were running and workers were healthy at collection time. OOMKilled=false for every expected service; root disk was 49%, swap was not configured, and 15-minute load was 4.69 on 8 CPUs (incident threshold 16). The bundle's crawler logs begin at the 11:04 UTC deployment boundary, leaving the first two hours unavailable; no retained fatal application line establishes the restart cause.

[viktor-shcherb]

Root-cause reconstruction from retained production telemetry:

• The only crawler deployment on 2026-07-10 completed at 11:05 UTC (run 29087859681); the observed worker exits at 17:10:04, 17:20:28, and 17:40:10 were therefore not deploy recreation.
• Loki shows worker-2 producing normal work through 17:10:03.777 and a fresh cli.starting at 17:10:07.032. There is no cli.shutting_down, cli.stopped, pipeline.stopped, pipeline.worker.early_exit, traceback, or fatal application event. The other workers show the same abrupt metric reset/start pattern.
• Prometheus shows worker-2 RSS peaking at 868,757,504 bytes before its pair of restarts, but worker-1 and worker-3 were only ~338 MB and ~295 MB before their last exits. A common 1 GB cgroup OOM therefore does not explain all three. The daily bundle also had OOMKilled=false and no kernel OOM line.
• No health auto-restart/watchtower unit, cron, systemd timer, GitHub deploy, or retained SSH command accounts for the staggered kills.

This rules out the original deployment-generation and Python graceful-exit hypotheses. The exits were externally/native-abrupt (no Python finally path), but the July Docker event stream/exit codes were not retained, so the exact signal sender or native exit code cannot be recovered retroactively. #5760 now records container identity and cgroup oom_kill counters; the remaining attribution gap is Docker lifecycle events/exit codes. I am keeping this issue open while adding that durable evidence instead of declaring the symptom fixed.

[viktor-shcherb]

Reopening after production verification of #5793 found a fresh, previously invisible recurrence at 2026-07-21 11:49:50 UTC. The new lifecycle watcher recovered the daemon events from its five-minute startup window: worker-3 emitted oom, then die with exit code 137, then start on the same container ID with RestartCount=1. This proves the immediate mechanism is a container cgroup OOM, even though Docker State.OOMKilled had already reset to false after restart. #5793 resolves the evidence-retention root cause, but the worker memory root cause remains and will be fixed before this issue is closed again.

[viktor-shcherb]

2026-07-21 post-fix recurrence

Window: 2026-07-20 13:14 UTC → 2026-07-21 13:14 UTC.

The durable lifecycle stream covers only 2026-07-21 11:49:50–13:12:32 UTC, but within that partial interval it proves 8 distinct cgroup OOM sequences after deduplication by container ID, action, and event timestamp. Each sequence is oom → die with exit code 137 → start on the same container ID.

worker	verified OOMs	first UTC	last UTC
worker-1	3	12:08:03	13:12:32
worker-2	2	11:55:42	12:49:47
worker-3	3	11:49:50	13:06:35

Four recurrences happened after #5794 merged and #5102 closed at 12:22 UTC: worker-1 and worker-2 on v0.13.134, then worker-3 and worker-1 on v0.13.135. The v0.13.135 application slice shows several overlapping high-cardinality monitor completions before the latest kills, including 21,873-item Amazon and 7,198-item Oracle HCM runs on worker-1, and 9,975-item Nextdata, 7,665-item API-sniffer, and 3,463-item Workday runs on worker-3.

All three workers were running and healthy at collection time. Current snapshots show no cgroup counter increment because those post-restart snapshots do not preserve the completed kill sequence; the timestamped lifecycle events are the authoritative evidence. The lifecycle journal and current crawler logs cover only the final 82 and 22 minutes respectively, so this is a lower bound rather than a supported full-window count.

Two remaining hypotheses:

1. Several large result/diff sets can still overlap under the 1 GiB worker limit even at the restored 5-monitor concurrency; inspect retention and weighted concurrency in apps/crawler/src/processing/board.py and apps/crawler/src/core/monitor.py.
2. The compaction added in apps/crawler/src/core/monitors/api_sniffer.py covers explicit rich mappings, while other high-volume monitor paths can retain full result sets through diff/enqueue; extend bounded-memory treatment across those paths.

Impact remains intermittent worker-capacity loss and retry delay for in-flight work; no sustained full-service outage was visible at collection.

[viktor-shcherb]

Root-cause fix is now in draft PR #5798.

The production recurrence is not one remaining rich-monitor implementation retaining a single oversized result. It is an admission-control failure: MONITOR_CONCURRENCY=5 assigns identical cost to tiny and 7k–22k-job boards, so several independently streamed/compacted monitors can still overlap inside one 1 GiB cgroup. Allocator high-water memory then carries into the next large board.

#5798 adds a fair weighted budget based on each board's existing rolling discovery counts, treats unknown boards as exclusive for their first run, and reclaims cyclic/libc heap memory before releasing the reservation. The observed Amazon (~21.9k) and JPMorgan (7.2k) production metadata already classify at the full four-unit HTTP-worker budget. Full crawler validation is green (5,688 passed, 37 skipped). I will keep this issue open through CI, merge, deployment, and a production OOM-free soak.