record delivered-but-unsettled requests as delivered_pending_settlement

[qdanik]

Summary

A gateway request whose winner streamed a full response to the client was accounted as failed with winner_nonce=0 whenever the settlement handshake couldn't finalize (validators halted → state signature unverifiable), so dashboards and devshard_gateway_critical_user_failures_total counted a paid, delivered request as a critical failure. This PR stops dropping the executor's work on a transient validator halt, so such requests settle and are accounted as success, and closes two settlement-liveness gaps found along the way. Fixes #1387.

Root cause

On an unverifiable validator state signature, Session.processResponse returned ErrInvalidStateSig early — before queuing the executor's receipt (MsgConfirmStart) and finish (MsgFinishInference) and before setting outcome.finished. So the executor's already-produced finish was dropped, the nonce never finished, and the request fell into finishRaceOutcome's failure branch (reached on the dominant route via doneCh → winningInflightTerminalFailure → forceTreatAsFailure), which hardcoded failed/0. Separately, settleDevshardOnChain deactivated the devshard before Finalize, so a Finalize that fell short of quorum left the escrow inactive-and-unsettled with no retry.

What changes

• processResponse no longer drops the executor's finish on an unverifiable state signature. The invalid/unexpected-key signature is logged and skipped (not counted toward the settlement quorum); the receipt and finish are still queued and outcome.finished is still set. The ErrStateHashMismatch drop (real state divergence) is kept as the dividing line. As a result the Gateway in-flight long chat during validator halt: client success vs request outcome failed #1387 halt now resolves to a normal success, and a latent liveness bug is fixed too: previously one transient bad state signature aborted the entire Finalize.

• The delivered_pending_settlement outcome becomes a narrow fallback. Since the finish is no longer dropped, the label now only covers the remaining case — the winner streamed clean content but produced no applied finish, so the nonce never finished. winnerDeliveredPendingSettlement requires err == nil && contentChunks > 0 && !isFailedStreamAttempt, guarded by !IsNonceFinished and !ErrStateHashMismatch; resolveTerminalDecision records the real winner nonce and this outcome across accounting, the gateway request metric, and the settled log. A genuinely broken winner (transport error, empty/error stream) still records failed/0. This relabels only accounting/metrics — escrow and the returned error are untouched.

• Settlement is retried instead of abandoned on a transient quorum shortfall. settleDevshardOnChain now always calls Finalize (its PhaseSettlement branch re-collects a short quorum, so a retry after a validator returns can complete instead of re-broadcasting the same short proof). A settlementRetry set on the Gateway (devshard id → deadline epoch) is populated on a retire failure (deadline epoch+1) and cleared on a successful settle (centralized in settleDevshardOnChain, so admin/auto/rotation/retry paths all clear); retrySettlements runs each rotation tick — outside the pre-PoC prepare window and while PoC is inactive — re-attempting each pending settlement under a bounded timeout until it succeeds or currentEpoch passes the deadline, after which the on-chain reaper takes over.

Safety

• Consensus: the finish and receipt carry their own executor signatures, verified independently on apply (applyFinishInference, applyConfirmStart); malformed txs are dropped by ApplyLocalBestEffort; and on-chain settlement independently re-verifies the state root, signatures, quorum, and the Settled flag (VerifyDevshardSettlement). No unverified signature is ever stored, so the change is quorum-neutral-or-stricter locally.
• Idempotency: on-chain MsgSettleDevshardEscrow checks escrow.Settled before any payout, so a "broadcast landed but the gateway read an error → re-broadcast" cannot double-pay — the re-broadcast is rejected.

Known limitations (not blockers)

• The settlementRetry set is in-memory; a gateway restart mid-retry loses it (the devshard stays persisted-inactive and is not retried). Funds are still protected by the on-chain reaper; persisting the set is a follow-up.
• If a settlement failure leaves the session at PhaseFinalizing, Finalize returns "already in progress" and cannot recover on retry — pre-existing, unchanged here.
• Retries may run one epoch past the on-chain settle window (escrow.EpochIndex+1) and get rejected until the deadline — harmless log noise (funds safe); clearing on a terminal chain error is a follow-up (avoids error-string matching).

Escrow economics (for reviewers)

If the executor never returns, its dropped finish is unrecoverable (there is no cross-host re-pick): the nonce goes unpaid to the executor and the client is refunded at devshard finalize; if settlement slips past the E+1 window, the reaper pays the group validators and the client loses the refund. This PR makes the transient-halt case settle normally (executor paid); permanent non-return still falls back to the reaper.

Testing

go build, gofmt, go vet ./... clean; user, types, and cmd/devshardctl packages green. New/updated tests: the apply path (bad sig + receipt + finish → nonce finishes, txs queued; rejected key not in quorum), the narrowed predicate (8 cases), forced-route delivered → delivered_pending_settlement, broken winner → failed, and the retry reconciler (success-after-failures clears, expiry past deadline, no-op when settlement disabled, earliest-deadline kept). End-to-end retry recovery and post-upgrade settlement are covered by Testermint.