Harden RPC-fault reliability: bound locks/timeouts and cap fan-out (7 fixes)

[kAIPraxisBot]

Summary

Hardens a class of reliability bugs where a transient RPC / chain-node failure (503, timeout, connection refused, tx error) turns into a disproportionate, long-lived outage. Three shapes recur across the services:

• a shared lock held across a chain RPC that has no timeout, so one slow query stalls every caller;
• an epoch-scoped or unbounded latch that turns a transient failure into an epoch-long degradation;
• unbounded goroutine/tx fan-out with no backpressure that self-amplifies under RPC failure.

The recurring aggravator is context.Background() on chain calls (the cometbft/cosmos clients set no default timeout). Each fix is one commit; every changed path is covered by a test, and the concurrency changes were run under -race.

Fixes

1. epoch_group_cache.go — don't hold the cache lock across the chain query (HIGH)

GetCurrentEpochGroupData held the write lock c.mu across CurrentEpochGroupData(context.Background()). At an epoch boundary every in-flight request misses at once; one hung query pins the mutex and stalls all inference validation. Now: singleflight coalesces concurrent misses into one query, the query runs with a bounded context outside the lock, and the lock is taken only for the brief cache read/write. The same shape in GetEpochGroupData is fixed too. Tested: single-RPC coalescing, mutex not held across the query, bounded context — under -race.

2. authzcache/cache.go — don't hold the cache lock across the signer queries (HIGH)

getSigners held the write lock across two sequential chain queries (GranteesByMessageType + AccountByAddress), so one slow granter throttled signature verification for every granter, and the 2-minute TTL made the misses recur. Now: singleflight + both queries run outside the lock under a bounded context. Tested: single-pair coalescing, mutex not held across the queries, fetch-path correctness/caching, error propagation — under -race.

3. devshard/gateway.go — bound concurrent escrow-rotation chain transactions (HIGH)

scheduleDepletedEscrowReplacement and scheduleAutoSettlement spawned one goroutine and chain tx per escrow with only per-id dedup and no global cap. A mass-depletion event (100+ escrows) fanned out that many broadcasts at once, driving the node into 503s and a self-amplifying crash loop. Now: a bounded escrowTxSem acquired around the actual broadcast (released during settlement back-off so slots are not starved); the helpers are nil-safe. Tested: concurrency reaches but never exceeds the cap; nil-safety.

4. proxy/sidecar/main.go — don't wipe the validator whitelist on a transient resolution failure (MEDIUM)

The once-per-epoch whitelist sync overwrote the nginx whitelist with whatever resolved. A transient DNS outage at that single moment resolved zero IPs, yet syncWhitelist returned nil, so the caller advanced BlockHeightSynced and did not re-sync until the next epoch — de-whitelisting every validator and accruing fail2ban bans. The 404 path already returns an error to preserve state; this mirrors it: when participants that advertise a URL exist but none resolve to a public IP, return an error so the whitelist is preserved and the sync retries within the same epoch. Tested: the guard fires on an all-unresolvable set.

5. devshard/escrow_rotator.go — bound the finalizeMu hold across the settlement Finalize (MEDIUM)

settleDevshardOnChain held the process-wide finalizeMu across the multi-round network Finalize. finalizeMu serialises every finalize (user /v1/finalize included), so a hung Finalize blocked all of them — and the admin settle path passes only the request context, which has no server-side deadline, so the hold could be indefinite. Now: finalizeUnderLock still holds the lock but bounds the Finalize with rotationFinalizeTimeout, capping the worst-case hold without changing the serialisation invariant. Tested: the timeout cuts off a hung finalize and the mutex is released.

6. post_chat_handler.go — bound the request-path Status() lookups with a deadline (MEDIUM)

handleTransferRequest and validateTimestampNonce called recorder.Status(context.Background()); the cometbft client has no default timeout, so a stalled node could pile up request goroutines/connections (line 361 runs before the bandwidth limiter, so it is not capacity-capped). Both calls now carry a statusQueryTimeout deadline (request-scoped where an echo context is available). Tested: the Status context now carries a deadline.

7. inference-chain/internal/rpc/client.go — bound the RPC HTTP client (LOW)

getStatus, GetBlockHash and DownloadGenesis used http.Get (http.DefaultClient), which sets no timeout, so a node that accepts the connection but never responds hangs the caller. Now: a shared client bounds dial + response headers, and each request additionally carries a per-request context deadline covering the whole request including the body read (short for the tiny status/block responses, generous for a genesis download), so a headers-then-body-stall is bounded too. Tested: happy/error paths, a never-responds node, and a headers-then-body-stall.

Testing

• Every changed path has a test; the concurrency changes (F1, F2, F3, F5) were verified under -race.
• Full package suites pass for the touched packages in decentralized-api, devshard, inference-chain, and the proxy/sidecar binary.
• The change set went through two rounds of independent adversarial review; all findings are folded in: bound the whole request including the body read (not just headers); run the coalesced cache queries under a decoupled context via DoChan so one caller's cancellation cannot fail the shared flight while each caller can still abandon its own wait; and key the whitelist-preserve guard on actual DNS resolution failures so an all-private participant set does not loop.