feat(relayer): optionally defer fills sourced from late-arriving origin blocks

[droplet-rl]

On chains with fixed slot times, a block delivered late within its own slot was published after that slot's attestation deadline and is materially more likely to be replaced at the same height by the next proposer. The existing deposit confirmation gate compares block heights, so a same-height replacement is invisible to it.

• SpokePoolClient (listener mixin) records, per block, the interval between the block's own timestamp and its arrival. Bounded to the most recent BLOCK_ARRIVAL_HISTORY blocks.
• Relayer.originBlockUnsettled() withholds a deposit whose origin block arrived at or beyond the configured threshold until a subsequent block has been built on top of it.
• Per-origin-chain control via RELAYER_MAX_ORIGIN_BLOCK_LATENESS_<chainId>, disabled by default.

Blocks not observed live have no recorded arrival time and are treated as settled, so backfilled and restarted state is unaffected.

Threshold selection is deliberately left to config: a useful value depends on the operator's own event-delivery latency, which has to be measured per deployment rather than assumed. Off by default for that reason.

🤖 Generated with Claude Code

[pxrl]

Test is failing - needs a fix.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 392f5bf4bc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[droplet-rl]

Fixed in b9f1145.

originBlockUnsettled() indexed config.maxOriginBlockLateness directly, but the Relayer tests cast a partial object literal to RelayerConfig, so it was undefined and every deposit evaluation threw. Now optional-chained, matching the minFillTime lookup above it. Repro'd the three failing files locally (Relayer.BasicFill, Relayer.SlowFill, Relayer.TokenShortfall) — all pass.

Added test/Relayer.LateOriginBlock.ts; the feature had no coverage, which is why this shipped. The last case omits the config field entirely and does fail without the fix.

One thing to flag: LATE_BLOCK_MIN_CONFIRMATIONS = 2 counts blocks built on top (same convention as the MDC check at Relayer.ts:440), so it requires two, not the one its comment claimed. I corrected the comment rather than the value — say the word if you'd rather it were 1.

[droplet-rl]

🔎 View trace

[pxrl]

@droplet review all comments.

[droplet-rl]

Reviewed all 6 Codex comments — I'd missed them earlier because I only pulled the review-scoped endpoint for your review, which was empty. Replied on each thread; 5 were real and are fixed in the new commit.

• Same-height replacement not recorded (P1) — valid. A replacement takes the misordered branch, so its own arrival was never recorded. Now recorded before the ordering check, and never lowered for a height. The old behaviour was already conservative when the original was the late block; the hole was a late replacement of a punctual one.
• SVM lateness always ~0 (P1) — valid, and the nastiest of the six. The SVM listener passes getCurrentTime() as the slot timestamp, so a Solana threshold would look configured and never fire. Rejected at startup. EVM and TVM use the real block timestamp.
• NaN / negative threshold (P2) — valid. Number("oops") silently disabled the check. Now asserted finite and non-negative, matching the deposit-confirmation check in the same file.
• Missing docs (P1) — valid per AGENTS.md. Documented in the relayer README, including the RELAYER_EXTERNAL_LISTENER dependency.
• Dry-run early exit (P2) — valid. Now mirrors the confirmation gate's simulation exception.
• LATE_BLOCK_MIN_CONFIRMATIONS = 2 (P2) — same thing I flagged to you. I've kept 2 and fixed the comment: an operator opting in is trading latency for finality safety, and the successor's own attestations haven't landed. Say the word and it's a one-line change to 1 — tests are written against the constant.

Tests grew to 15; the two new same-height cases both fail without the fix. 66 passing across the affected files, lint and typecheck clean.

One thing to be aware of: .env.example and src/monitor/.env.sample show as deleted in my working tree and were already deleted when the session started. Not mine, so I left them unstaged out of both commits.

[droplet-rl]

🔎 View trace

[pxrl]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 33d1e17400

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[droplet-rl]

Codex came back with one P2 and it was a good one — fixed, plus a worse instance of the same bug it didn't spot.

What it found: the EVM listener reads the head block, scrapes historical events, then posts the block update. So backfill duration was counted as block-arrival lateness, and a slow startup scrape would falsely mark the head as late.

What it missed: TRON has the same shape but inside the poll loop (RelayerSpokePoolListenerTVM.ts:133-169) — read head, run range query, post. So on TRON the query duration inflated every block, not just the startup head.

Why I didn't fix it the way Codex suggested. Stamping the observation time before the scrape corrects the interval, but the deeper problem is that a polled block has no meaningful lateness at all: the interval measures poll phase, not publication lateness. A punctual block polled 8s into a 12s slot reads as 8s late. Recording that would false-positive at a high rate — worse than not measuring.

So the block update now carries an optional observedAt and lateness is recorded only when present. Only the EVM live watchBlocks subscription sets it; the startup scrape and TRON poll omit it and fall through to the existing "not observed live → settled" path. Net effect: the gate only ever fires on chains where we're actually pushed blocks, which is what it was always meant to mean.

Also corrected the README — my earlier claim that restarted state was unaffected was wrong.

23 tests in the feature file, 63 passing across affected files, lint and typecheck clean. Two new cases: polled updates record nothing, and lateness comes from the observation rather than message receipt.

Still open for you: LATE_BLOCK_MIN_CONFIRMATIONS — I've kept it at 2, one line to change if you want 1.

Worth noting one residual: watchBlocks uses emitOnBegin, so the first block after a subscription starts is effectively a poll and can read as late. It's one block per listener start and errs toward deferring, so I left it.

[droplet-rl]

🔎 View trace

[droplet-rl]

Two follow-up commits, in response to the review.

LATE_BLOCK_MIN_CONFIRMATIONS 2 → 3. The review is right that the fixed cap is the weak point, but the binding quantity is the notification latency rather than the lateness magnitude. A deposit is only backed out once the listener reports the removal, and that report trails the replacement by more than one block time often enough that two confirmations can release a deposit while its removal is still in flight. Three covers the observed range. The tests reference the constant symbolically, so no test changes were needed for the bump.

Not taking the receivedAt reformulation, for two reasons:

1. The scaling term only exceeds the confirmation floor once lateness passes roughly three block times. Measured lateness on blocks that were actually replaced sits well inside that, so on the cases we can observe the two formulations are behaviourally identical — the floor does all the work either way.
2. A plain overwrite of the (timestamp, receivedAt) pair reintroduces the same-height-replacement gap that #33d1e17 closed: δ recomputes against the replacement's timestamp, falls under the threshold, and the height is treated as settled while the removal is still pending. receivedAt is monotonic, but the threshold test reads δ, which isn't — so the monotonicity fix doesn't disappear, it becomes a sticky "was this height ever late" flag. Worth doing if the scaling ever earns its keep, but it isn't a free deletion.

Also folded in the standing items: the externalListener assert (matching the SVM one — previously documented but unenforced), the shadowed blockNumber in the retention sweep, and the README's restart claim, which overstated things: the first block pushed after a (re)connect can still record an inflated interval and be held.

I left the sweep unconditional. Gating it on map size looked like a cheap win but changes eviction from height-distance to size-based, which breaks the retention test's contract for no real gain — the map is bounded by the sweep itself.

[droplet-rl]

Dropped the external-listener assertion in fc7866b, replied on the thread.

You're right that it was disproportionate — the threshold is merely inert without the listener, not broken, and hard-failing startup makes it awkward to carry in a shared config across deployments that don't all run one. Back to documenting it as inert; the test now pins that a threshold parses and is retained with the listener disabled.

I kept the two adjacent asserts in that block — malformed/negative values, and SVM where there's no slot timestamp to compare against. Those are genuinely broken input rather than a no-op, but say the word if you want either of those gone too.

Note I was a few commits behind when this came in (your comment referenced maxDelay, which I didn't have), so I fast-forwarded onto 098a2512 first — the rename and the three-confirmation change are intact. 18 tests in the feature file, 69 passing across affected files, lint and typecheck clean.

[droplet-rl]

🔎 View trace

[droplet-rl]

Fixed in 26bbe8b4. The flag is right, and the second consequence is the one that matters: a punctual block filled at its MDC tier and then re-orged is a real invalid fill, and the log would have reported it as having confirmations to spare — reassurance during exactly the incident class the instrumentation exists to catch.

The log now reports measured facts only, folded into the existing removal warn:

arrivalDelay   how long after its own timestamp the block reached us
confirmations  accrued on the origin block at the time of the report
noticeDelay    seconds from the block's timestamp to the report

Absent entirely for blocks not observed live, so their absence is itself meaningful. Establishing whether the gate applied is left to whoever knows the configured threshold.

One step further than proposed: I also dropped confirmationsToSpare rather than only neutralising the message. The field name carries the same unfounded claim, and a dashboard or alert rule keying on it would read the false reassurance regardless of how the message is worded. confirmations plus the constant recovers it for anyone who has first established that the gate applied.

Agreed on logging all removals rather than gating — that distribution is what produced the arrival-delay figures in the first place. Also took the noise point: one warn per removal now, fields adjacent to the deposit they describe.

Tests re-pointed from message text to fields, including negative assertions that the log does not say "late-arriving" or "to spare", plus a case covering a removal for a block that was never observed live. 20/20 locally.

[droplet-rl]

Thanks — nothing to action, no inline comments on the approval.

Approved commit 26bbe8b4 is the current head, all six checks pass, and the PR is mergeable. Leaving the merge to you.

[droplet-rl]

🔎 View trace

[droplet-rl]

2d4457fc makes this on-by-default for mainnet at 5 seconds, via DEFAULT_MAX_ORIGIN_BLOCK_ARRIVAL_DELAY in Constants.ts. Chains without an entry stay disabled; an explicitly configured 0 still disables.

This is a behaviour change worth a reviewer's attention: the gate was opt-in and is now active wherever the external listener runs, third-party operators included. It remains inert without a listener.

Sizing, on mainnet measurements: 5s sits at roughly the 90th percentile of arrival delays, so it withholds on the order of 10% of mainnet-origin deposits until their origin block has LATE_BLOCK_MIN_CONFIRMATIONS on top — a few blocks, after which they fill normally. Nothing is dropped. Against that, it covers close to 90% of the arrival-delay range in which blocks that were subsequently replaced have actually been observed.

That is deliberately tighter than the point that maximises protection per deposit withheld, which sits nearer 7s. The rationale is that fill latency is the recoverable cost and an invalid fill is not, and that the current competitive baseline is the wrong reference if comparable mitigations become common. Loosening is a config change, not a code change.

The figures come from indexer ingestion timestamps, which run slightly later than the listener's own observation, so the effective threshold in production will be marginally tighter than these percentiles imply. The arrivalDelay reported with each removal is the signal to re-tune from once there is deployment data.

Tests: 22/22, covering the chain default, no-default chains, and explicit-zero override.

[droplet-rl]

Thanks — nothing to action, no inline comments again.

Confirming the approval covers the on-by-default change: it's against 2d4457fc, which is the current head, and it landed two minutes after that commit was flagged here as a behaviour change. So mainnet-origin deposits will be withheld at a 5s arrival delay wherever the external listener runs, third-party operators included, on the understanding that's intended.

All six checks pass, PR is mergeable. Leaving the merge to you.

[droplet-rl]

🔎 View trace