feat: endpoint failover for unavailable data

[barnabasbusa]

Endpoint failover for unavailable data

Problem

Dugtrio picks one endpoint per call (sticky per session) and streams back whatever it returns. For data whose availability differs per endpoint — most notably historical states, which only nodes with state archiving serve — this means requests keep failing even when another endpoint in the pool has the data:

• eth/v1/beacon/states/{old-slot}/validators → 404 "State not found" (pruned) or 500 "Historical state regen is not enabled" (lodestar without --chain.serveHistoricalState), while e.g. nimbus endpoints serve it fine.
• Worse, some endpoints hang on these queries (observed on glamsterdam-devnet-6: a prysm node blocks indefinitely on historical state queries, and a lodestar node's nginx hangs on error responses when the request has Accept-Encoding: gzip), so the client just burns the whole callTimeout. These nodes pass all pool health checks (version/syncing/head), so they never get evicted — the hang only manifests on the failing request path.

Change

When an upstream cannot serve the requested data, retry the call on other ready endpoints instead of returning the failure:

• Trigger: calls matching failoverPaths (default: ^/eth/v[0-9]+/beacon/states/, ^/eth/v[0-9]+/debug/beacon/states/) that fail with 404/500/501/503 or a connection error.
• Sweep order: remaining ready endpoints interleaved by client type, so every client implementation is covered as early as possible (e.g. the archiving client is found within the first few attempts rather than after trying all instances of a pruning client). Nothing is persisted between calls — each failover sweeps fresh, so load balancing is preserved.
• Hedging instead of per-attempt timeouts: fast upstream failures advance to the next candidate immediately. An attempt that stays silent for failoverHedgeDelay (default 20s) is not cancelled — it keeps running while the next candidate is raced in parallel, bounded by failoverMaxParallel (default 2). The first acceptable response wins; the other attempts are cancelled. So slow-but-capable endpoints keep the full call timeout to load a state (mainnet/hoodi state queries regularly take a minute+), while hanging endpoints only cost latency, not time budget — no cascade.
• Fallback: if no endpoint can serve the data, the last upstream failure response is passed through to the client instead of a generic 500.
• Scope: GET/HEAD/POST only (POST bodies up to 1MB are buffered for replay). Calls pinned to an explicit endpoint via X-Dugtrio-Next-Endpoint are excluded; client-type filters (prefix proxies / type filter) are honored during the sweep. Sticky session endpoints are not changed by failover. Rate limiting still charges 1 call.
• Served failover responses expose X-Dugtrio-Failover-Attempts.

New config (all optional, feature is on by default for state paths; disable with failoverDisabled: true):

proxy:
  #failoverDisabled: false
  #failoverPaths:
  #  - ^/eth/v[0-9]+/beacon/states/
  #  - ^/eth/v[0-9]+/debug/beacon/states/
  #failoverMaxAttempts: 8
  #failoverHedgeDelay: 20s
  #failoverMaxParallel: 2

Review updates

• Removed persistence of the retried endpoint (0c06f1f): the first version remembered the last endpoint that served a path class and tried it first, which would concentrate failover traffic from all sessions on the same node. Load balancing is prioritized instead (thanks @pk910).
• Hedging replaces the hard per-attempt timeout (d6874a6): the first version cancelled attempts after failoverAttemptTimeout, which on bigger networks (where state loads take a minute+) would cascade — no endpoint got enough time to load the state (thanks @pk910). failoverAttemptTimeout is replaced by failoverHedgeDelay/failoverMaxParallel.

Testing

Unit tests (proxy/proxycall_test.go, run with -race in CI) cover the failover coordinator against mock upstreams: fast-failure sweep, hedging on silent endpoints, slow endpoint still winning after hedges started (the cascade regression), fallback passthrough when nobody has the data, the parallelism cap, POST body replay, and the non-failover passthrough path.

Manual testing of the original version against all 45 glamsterdam-devnet-6 nodes (6 client types; nimbus is the only one serving historical states there, and the pool includes endpoints that hang on these queries):

Case	Result
GET /eth/v1/beacon/states/10000/validators	200 from nimbus-erigon-1, X-Dugtrio-Failover-Attempts: 4, hanging endpoints skipped
GET /eth/v2/debug/beacon/states/10000 (SSZ)	200, 3.7MB body (5 attempts)
POST .../validators with JSON body	200, body replayed, 1 attempt, 0.2s
GET /eth/v1/beacon/headers/head	untouched — no failover header, normal sticky endpoint
nonexistent state (slot 99999999)	bounded sweep (8 attempts), upstream error returned — no hang
?dugtrio-next-endpoint=<node> (pinned)	no failover, response passed through as before
?dugtrio-next-endpoint=lodestar (type filter)	sweep stayed within lodestar endpoints, upstream error passed through

go build clean; golangci-lint reports no new findings (2 pre-existing on master).