[observability] Stop Grafana series rejection and Alloy OOM telemetry loss

[viktor-shcherb]

Audit finding

Both production telemetry delivery paths are losing data during the current incidents.

Host Alloy remote write

On 2026-08-01 the host collector repeatedly received HTTP 429 with err-mimir-max-active-series: the Grafana tenant exceeded its 15,000 active-series limit. Remote-write batches were rejected/stalled.

Crawler compose Alloy

The compose collector is constrained to 256 MiB and was repeatedly OOM-killed approximately every two hours on August 2-3. Audit-time restart count: 21.

Impact

Metrics and logs are incomplete exactly when PostgreSQL, Typesense, backup, reconciliation, and crawler services are failing. Process-up checks and dashboards cannot be trusted to demonstrate coverage/freshness while ingestion is rejecting series or the collector is restarting.

Scope

• Inventory active series by job/label and identify the cardinality growth source.
• Establish an explicit series budget per surface and remove unbounded/high-cardinality labels.
• Restore successful host remote write with headroom below the tenant limit.
• Profile compose Alloy memory, queue/backpressure, discovery, and log-volume behavior; fix the cause and right-size the limit.
• Add collector self-monitoring for 429s, queue age, dropped samples/logs, memory pressure, restart count, and last successful remote write.
• Verify alert routing independently from the same impaired telemetry path.
• Add deploy/capacity checks that fail before cardinality or memory limits become hard outages.

Acceptance criteria

• No err-mimir-max-active-series responses under representative load.
• Active-series count stays within a documented budget with alerting headroom.
• Compose Alloy runs for at least seven days without OOM/restart.
• Host and compose metrics/logs have bounded freshness visible from the destination.
• A forced ingestion failure reaches an independent operator channel.
• Dashboards identify coverage gaps instead of silently displaying stale/partial data.
• Cardinality and memory regression tests/checks are part of deployment.

Related

• Epic: Harden and recover all non-Murmur Hetzner systems #5922
• Monitor every Hetzner host and correct false exporter alerts #5926
• Give the Hetzner Codex error-review service scoped metrics access #5948
• audit-observability: web app emits zero metrics; Typesense vs Postgres fallback ratio is unmeasurable #3182
• [critical] PostgreSQL data volume is full and production database is crash-looping #6117
• [critical] Recover Typesense from persistent leaderless Raft state and prevent descriptor exhaustion #6119

Found during the 2026-08-03 full repository/service audit. Murmur, Stripe, and Resend are outside scope.

[viktor-shcherb]

2026-08-03 daily error review evidence\n\nWindow: 2026-08-02T09:04:00Z to 2026-08-03T09:04:00Z.\n\nThe lifecycle bundle proves 11 Alloy OOM → exit 137 → restart sequences from 2026-08-02T10:50:22Z through 2026-08-03T07:35:54Z. The current container reports restart count 21 and a 256 MiB limit; it was running at collection.\n\nFull-window Alloy logs also record 28 stale/out-of-order Prometheus sample rejections and 11 final Loki batch drops with no retries left, establishing telemetry loss rather than harmless restarts. Six stale deleted-container inspection errors were also present.\n\nThis is classified as a critical incident and deduplicated here. Candidate code/config surfaces are apps/crawler/docker-compose.yml, apps/crawler/alloy.river, and deploy/observability/alloy-host.alloy.

[viktor-shcherb]

Production remediation is live and PR #6217 is ready for review: #6217

Immediate acceptance:

• protected fleet deployment passed end to end: https://github.com/colophon-group/jobseek/actions/runs/30817867323
• active series reduced to 5,226 (35% of the 15,000 provider limit; below the new 12,000 deploy ceiling)
• crawler 1,026/2,000, Redis 58/200, Unix/textfile 1,061/2,000
• all four Alloy collectors ready with current send timestamps
• zero recent HTTP 429s, no queue above 386/4,000, and no failed/dropped Prometheus samples after rollout
• native Alloy has zero restarts on crawler, PostgreSQL, and Typesense
• Compose Alloy is running at 512 MiB with zero restarts and OOM=false; all six crawler workloads retained their start times during its isolated recreation
• root-only rollback/acceptance evidence: /var/lib/jobseek-incidents/6126/20260803T132410Z

The fix removes the two unbounded series multipliers (full Alloy self-scrapes and per-egress-host circuit metrics), allowlists Redis operational metrics, bounds remote-write concurrency/queues, and adds independent alerts for rejection, staleness, backlog, loss, memory, restart/OOM, and renewed series growth.

The seven-day stability criterion cannot honestly be completed on deployment day. The new independent host sampler and rules now retain that evidence without depending on the Compose collector itself; the immediate recovery and prevention gates pass.