Skip to content

EL cannon finalized ceiling freezes: GetBeaconBlock("finalized") is cached under the moving alias key #878

Description

@selfuryon

Summary

The execution-layer cannon (canonical_execution_* derivers, added in #850 stops making progress a short time after startup and never resumes until the process is restarted. Its finalized_block / ceiling stays pinned to the value observed at startup even though the beacon node reports a much newer finalized block.

As a result canonical_execution_block / _transaction /_balance_diffs (etc.) ingest one initial catch-up burst and then stall indefinitely.

Root cause: the EL block iterator resolves its ceiling with GetBeaconBlock(ctx, "finalized"), and the beacon-block cache stores results keyed by the raw identifier string — so the moving "finalized" alias gets cached like an immutable block, with a hard-coded 1h TTL that is additionally kept alive forever by touch-on-hit.

Affected versions

  • Reproduced on v1.22.0.
  • The relevant code is unchanged on master at time of writing.

Observed behaviour

  • finalized_block in the "Caught up to CL-finalized ceiling, sleeping" log stays constant for hours (verified across many epoch ticks).
  • Meanwhile the beacon node's /eth/v2/beacon/blocks/finalized returns a block ~hundreds of blocks newer (verified directly and via the LB the cannon uses).
  • Restarting the pod clears the cache → the ceiling jumps to the current finalized block, processes the accumulated gap in one burst → then re-freezes at the new value.
  • The consensus cannon is unaffected: its checkpoint iterator derives finality from the wallclock, not from GetBeaconBlock("finalized").

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions