Summary
The execution-layer cannon (canonical_execution_* derivers, added in #850 stops making progress a short time after startup and never resumes until the process is restarted. Its finalized_block / ceiling stays pinned to the value observed at startup even though the beacon node reports a much newer finalized block.
As a result canonical_execution_block / _transaction /_balance_diffs (etc.) ingest one initial catch-up burst and then stall indefinitely.
Root cause: the EL block iterator resolves its ceiling with GetBeaconBlock(ctx, "finalized"), and the beacon-block cache stores results keyed by the raw identifier string — so the moving "finalized" alias gets cached like an immutable block, with a hard-coded 1h TTL that is additionally kept alive forever by touch-on-hit.
Affected versions
- Reproduced on
v1.22.0.
- The relevant code is unchanged on
master at time of writing.
Observed behaviour
finalized_block in the "Caught up to CL-finalized ceiling, sleeping" log stays constant for hours (verified across many epoch ticks).
- Meanwhile the beacon node's
/eth/v2/beacon/blocks/finalized returns a block ~hundreds of blocks newer (verified directly and via the LB the cannon uses).
- Restarting the pod clears the cache → the ceiling jumps to the current finalized block, processes the accumulated gap in one burst → then re-freezes at the new value.
- The consensus cannon is unaffected: its checkpoint iterator derives finality from the wallclock, not from
GetBeaconBlock("finalized").
Summary
The execution-layer cannon (
canonical_execution_*derivers, added in #850 stops making progress a short time after startup and never resumes until the process is restarted. Itsfinalized_block/ ceiling stays pinned to the value observed at startup even though the beacon node reports a much newer finalized block.As a result
canonical_execution_block/_transaction/_balance_diffs(etc.) ingest one initial catch-up burst and then stall indefinitely.Root cause: the EL block iterator resolves its ceiling with
GetBeaconBlock(ctx, "finalized"), and the beacon-block cache stores results keyed by the raw identifier string — so the moving"finalized"alias gets cached like an immutable block, with a hard-coded 1h TTL that is additionally kept alive forever by touch-on-hit.Affected versions
v1.22.0.masterat time of writing.Observed behaviour
finalized_blockin the "Caught up to CL-finalized ceiling, sleeping" log stays constant for hours (verified across many epoch ticks)./eth/v2/beacon/blocks/finalizedreturns a block ~hundreds of blocks newer (verified directly and via the LB the cannon uses).GetBeaconBlock("finalized").