0.2.14 dapi followups: Host Events Long-Poll & ML-Node Capacity Fallback

[a-kuprin]

Summary

Standalone devshardd no longer depends on continuous DAPI and inference node reachability for two critical paths: warming escrow sessions and bounding direct-HTTP fallback when AcquireMLNode fails. DAPI listens for chain escrow and maintenance events (scheduled / canceled), publishes them plus per-escrow acquire load over a long-poll RPC (GetHostEvents); devshardd consumes them and gates fallback concurrency with observed node capacity.

This branch builds on upgrade-0.2.14 follow-ups: the long-poll + capacity feature commits, plus hardening for correctness under outages, high load, and unrelated event churn.

Motivation

When inference node or DAPI is slow or unreachable:

1. Cold escrows — sessions that only exist after a chain query / lazy create
   miss traffic until something warms them.
2. Unbounded fallback — doWithFallbackNodes can hammer ML nodes with no
   awareness of broker locks or how many active escrows share a node.

When the inference node (DAPI path) is not available, there is also no
request-time lazy-init of escrow sessions: the client cannot round-trip through
DAPI to create/warm a session on demand. Instead devshardd actively
creates sessions from polled escrow-created host events, so traffic can proceed
on the direct-HTTP fallback path without waiting for a lazy create that cannot
happen.

The design observes capacity and load while DAPI is healthy, then uses that
observation to bound fallback during an outage — without inventing a second
control plane.

Architecture (conceptual)

  chain events ──► DAPI HostEventRing ──► GetHostEvents (long-poll)
                         │                      │
                         │                      ▼
                  EscrowLoadTracker      standalone hostevents.Run
                  (EMA rates)                   │
                         │              ┌───────┴────────┐
                         └──────────────► LoadMap        │
                                        │                ▼
                         ListNodeCapacity ──► NodeCapacityCache
                                        │                │
                                        └──────► doWithFallbackNodes
                                                 (TryAcquire / Unknown)

Host events. DAPI’s event listener appends escrow created/settled and
maintenance scheduled/canceled into a bounded HostEventRing. Clients
long-poll GetHostEvents with a subscribe set and cursor — so maintenance
updates are available the same way as escrow updates (clients that only care
about escrows subscribe to escrow kinds and are not woken by maintenance).
devshardd warms / tears down sessions from escrow events and refreshes its
load map from every reply (including unchanged timeouts).

Capacity fallback. While DAPI is up, devshardd polls ListNodeCapacity
and consumes escrow_load. Fallback slots are
effectiveMax − chargedLocks − localInFlight, with
effectiveMax = maxConcurrent / max(activeEscrows, 4). When DAPI is down,
last-seen capacity is retained (stale) and the local semaphore still bounds
concurrency.

Behavior changes

Area	Behavior
Warm path	Escrow events warm sessions without a request-time chain round-trip
Maintenance	DAPI listens for maintenance scheduled/canceled and serves them via the same long-poll
Long-poll wake	Waiters wake only on subscribed kinds — maintenance churn cannot reset an escrow client's deadline or stall escrow_load delivery
Fallback bound	Enforced when capacity has been observed; old DAPI (Unimplemented) stays unbounded
Unknown nodes	Nodes returned by PickNode but never listed in capacity get a synthetic budget (UnknownMaxConcurrent / Divisor()), not an unbounded bypass
Chain blips	Transient chain query failures map to HTTP 503 (chain_unavailable); true not-found stays a hard error
Load tracking	Per-escrow rates use a decaying EMA (O(escrows)), not a 30-minute timestamp list (O(RPS × window))
Event ring	Fixed-capacity ring buffer — overwrite in place on wrap (no sliding-slice growth / GC churn)