Prune block_hashes at every checkpoint - #6599
Open
afck wants to merge 7 commits into
Open
Conversation
A checkpoint only guarantees the availability of the blocks it vouches for in its `outbox_block_hashes`, so a node that bootstraps from it never learns the other pre-checkpoint blocks. Drop those from `block_hashes` when executing a checkpoint, so a node that has been following the chain all along converges on the same set. Retained below the checkpoint are the vouched-for blocks, which still carry unacknowledged messages, and any block still queued in an outbox: a queue is drained by a delivery confirmation from the recipient's validators, which is a separate path from the acknowledgement that empties `unfinalized_message_blocks`, so it can outlive the checkpoint's guarantee. For the vouched-for set to cover every `previous_message_blocks` anchor, the checkpoint also drops the anchor of each recipient that has acknowledged everything we sent it. Such an anchor would otherwise dangle even without any pruning, breaking a bootstrapped node on its next block to that recipient. Finally, `process_outgoing_messages` now treats a missing `block_hashes` entry for the outbox's previous height as "no predecessor" instead of a corrupted chain state -- the same state a bootstrapped node's rebuilt outbox is in.
Extend the ack-cycle test past the producer's second checkpoint: reset the producer to that checkpoint (the unit-test stand-in for a node that bootstrapped from it) and send to the recipient again. This reproduces two failures observed on devnet-2026-07-08 with unpruned anchors: the proposal failed locally with CorruptedChainState because the recipient's previous_message_blocks anchor pointed below the checkpoint, and a full-history sender's message on the same lane was never delivered to the checkpointed recipient's inbox.
pull Bot
pushed a commit
to Spencerx/linera-protocol
that referenced
this pull request
Jul 14, 2026
## Motivation `ChainClient::checkpoint` exists since the checkpoint feature was merged, but there was no way to trigger a checkpoint manually: its only callers are unit tests, and neither the CLI nor the node-service GraphQL API exposes it. ## Proposal Add a `linera checkpoint [CHAIN_ID]` command that publishes a checkpoint of the given chain (defaulting to the wallet's default chain), retrying via `apply_client_command` like the other operation commands, and printing the height of the resulting block. ## Test Plan CI, plus a manual end-to-end test against the `devnet-2026-07-08` devnet (validators at 13206a2, client from this branch): * Two wallets with faucet chains A and B; several transfers in both directions with inbox processing. * `linera checkpoint` on A (with a still-unreceived A→B transfer in flight) — checkpoint block certified by the devnet validators; B then received the pre-checkpoint message. * Full `CheckpointAck` cycle: checkpoint on B, A received the ack in a block and checkpointed again. * Fresh third wallet, `linera wallet follow-chain A --sync`: bootstrapped from the *latest* checkpoint without replaying history — local state and balance match the owner wallet, and the local node stores only the checkpoint block (a `blocks` GraphQL query walks back from the tip and stops at the checkpoint block because its parent was never downloaded). * `change-ownership`/`assign` to the third wallet, which then successfully proposed blocks (a transfer to a fresh chain, and another checkpoint) on the bootstrapped chain. The test also reproduced two known limitations on `main` (addressed by the upcoming linera-io#6599): * The bootstrapped wallet cannot send to a recipient whose `previous_message_blocks` anchor predates the checkpoint (`Corrupted chain state: missing entry in block_hashes`, locally in the proposing client), and its local node can no longer sync the chain once another owner does so. * After both sender and recipient have checkpointed, a new message on the previously-used A→B lane is never delivered to the recipient's inbox on the validators (a transfer to a fresh recipient sent at the same time was delivered normally). ## Release Plan - Nothing to do / These changes follow the usual release cycle.
…ssing Fixes for three review findings on checkpoint pruning: * handle_revert_confirm walked the previous_message_blocks links down into the acknowledged prefix and hard-errored on the first pruned height, before re-adding anything to the outbox - so after a recipient reset, wiped in-flight bundles were never resent. The walk now stops at the first missing hash instead: acknowledgements cover a prefix of each lane, so everything from there down is already included in the recipient's own latest checkpoint and needs no retransmission. * The simple-protocol proxy served DownloadCertificatesByHeights through the chain worker's pruned block_hashes map, silently under-serving requests for pruned heights and making honest validators look faulty to syncing clients. It now reads the unpruned storage index, like the gRPC proxy. The in-memory test validator took the same shortcut and got the same fix. * The process_outgoing_messages tolerance for a missing predecessor hash was untested. The reachable scenario is an outbox that drained while ahead of the tip: it survives, pointing past its last scheduled block, which a later checkpoint then prunes once every recipient has acknowledged it. A new worker test builds that shape end-to-end with real certificates, and the tolerance comment now describes it.
After a checkpoint prunes a fully acknowledged lane, the next message on it carries previous_height: None. A node that never received one of the settled bundles — it bootstrapped the sender chain from a later checkpoint, or missed the push — could not reconcile its consumption: once the new bundle was added, consuming the missed one hit UnexpectedBundle (and in the reverse order, the anticipated entry made every future push fail), permanently wedging the lane. Recoverable only by abandoning the chain and bootstrapping it. The inbox now records the no-predecessor point in a new sender_pruned_cursor register: the sender's checkpoint settled everything below it, so nothing there is re-pushable, and consumption reaches this node only through the receiver's own certified blocks — which carry the bundle in their body. Below the cursor, consumptions that cannot be reconciled are ignored instead of failing, nothing is anticipated, anticipated entries are dropped when the cursor advances, and re-deliveries are dropped. Bundles actually present in added_bundles are unaffected: a pending bundle below the cursor — e.g. a CheckpointAck sent by the checkpoint block itself, which no anchor tracks — still reconciles or errors exactly as before, and the ignored case reports the bundle as not present, so block proposals are still rejected unless the validator really holds their bundles.
A CheckpointAck bundle has no availability guarantee: no anchor tracks it, no checkpoint vouches for its block, and once delivered it is never resendable. A validator that bootstrapped the sender chain from a checkpoint later than the ack-sending block therefore cannot hold the bundle, and a proposal consuming it is rejected by that validator (MissingCrossChainUpdate) while others may vote for it. In the fast round the votes are confirmations, so such a split cannot be resolved by re-proposing without the ack. Clients now propose ack-consuming blocks in a slower round - the same rule that already applies to blocks using oracles - and validators enforce it, mirroring FastBlockUsingOracles.
When a validator rejects a proposal with MissingCrossChainUpdate, the updater first tries to heal it by pushing the sender chain. That cannot work for a CheckpointAck bundle on a validator that bootstrapped the sender chain from a checkpoint later than the ack-sending block: the block sits below its tip, is vouched for by nothing, and is never resent. With enough such validators (plus ordinary unavailability), a proposal consuming the ack can never be certified, and the client would keep re-proposing it forever. The client now settles exactly the acknowledgement bundles named by such rejections in its local inbox - removing them from the pending bundles and advancing the lane's sender_pruned_cursor past them, the state a node that never received them is in - and retries the proposal without them. Skipping them is always legal: they are Simple, zero-grant bundles, and validators that do hold one discard it when the lane's next bundle is consumed; if another owner's quorum does consume it, the certified block reconciles as a settled no-op. The settlement is part of the chain state, so it survives client restarts and one-shot CLI commands instead of costing a failed proposal round-trip each time. A leftover pending proposal from an interrupted call gets the same treatment rather than wedging every future proposal. If nothing else is pending, no block is proposed at all.
afck
force-pushed
the
prune-block-hashes-at-checkpoint
branch
from
July 16, 2026 14:38
905bfaa to
534455b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
A checkpoint only guarantees the availability of the blocks it vouches for in its
outbox_block_hashes, so a node that bootstraps from it never learns the other pre-checkpoint blocks.However, a node that just executes a block with a checkpoint, without bootstrapping from it, has its
block_hashescollection growing indefinitely.Proposal
Drop those from
block_hasheswhen executing a checkpoint. Retained below the checkpoint are the vouched-for blocks, which still carry unacknowledged messages, and any block still queued in an outbox.For the vouched-for set to cover every
previous_message_blocksanchor, the checkpoint also drops the anchor of each recipient that has acknowledged everything we sent it. Such an anchor would otherwise dangle even without any pruning, breaking a bootstrapped node on its next block to that recipient.The rest of the codebase is made to survive pruned block hashes:
process_outgoing_messagestreats a missingblock_hashesentry for the outbox's previous height as "no predecessor" instead of a corrupted chain state — the same state a bootstrapped node's rebuilt outbox is in.handle_revert_confirmstops re-adding heights at the first pruned one instead of failing: acknowledgements cover a lane prefix, so everything below the pruned height is already acknowledged in the recipient's own checkpoint.DownloadCertificatesByHeightsdirectly from storage instead of going through the (now prunable)block_hashesmap.Settled lanes. When the first bundle of a cross-chain update declares no predecessor, the sender's checkpoint settled the lane below it, and the sender will never push anything below that bundle again. The receiving inbox records this in a new
sender_pruned_cursor: below it, consumptions that cannot be reconciled withadded_bundlesare ignored instead of raisingUnexpectedBundle, nothing is anticipated, and late re-deliveries are dropped. Without this, a node that replays the recipient chain without having seen the pre-checkpoint bundles would corrupt its inbox cursor. Bundles actually present in the inbox still reconcile (or get skipped) normally.CheckpointAckliveness. Acknowledgements are the one bundle class with no availability story: checkpoints don't vouch for them, they are excluded fromprevious_message_blocksandunfinalized_message_blocks, and bootstrapping doesn't restore them to the outbox. A validator that bootstrapped the recipient chain from a later checkpoint can therefore never obtain a pre-checkpoint ack, and no one can make it validate a block consuming that ack. So:CheckpointAck: the worker rejects such proposals, and the client routes ack-consuming blocks to a non-fast round. Otherwise a fast proposal confirmed by half the validators and unvalidatable by the other half would brick the chain.MissingCrossChainUpdatefor an ack bundle even after the updater has sent the validator the sender chain's information, the client retries the proposal without any ack bundles.added_bundlesand advancingsender_pruned_cursorpast it — so a restart or a laterlinera process-inboxdoesn't propose it again, while a certified block from another chain owner that does consume the ack still reconciles as a no-op.This also fixes the issues found in #6609.
Test Plan
Tests were extended: inbox unit tests for the settled-lane marker, a two-worker test driving a full sender/receiver checkpoint-ack cycle across pruned lanes, and client tests for revert-confirm on pruned heights, the fast-round exclusion, and skipping an ack a validator cannot hold.
Release Plan
Links