Skip to content

Worldstate database corruption. #10749

Description

@Yoldark34

Steps to Reproduce

Don' t know

Expected behavior: No worldstate database corruption.

**Actual behavior:**Worldstate database corruption.

Frequency: first time

Logs

2026-07-05 07:45:13.973+0200 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | FCU(VALID) | head: 2c502.....7b345 | safe: 9c64e.....6da98 | finalized: ed6f2.....7a214
2026-07-05 07:45:24.938+0200 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #25,464,316  (556b4.....f65b6)|  186 tx ( 69.4% parallel)| 16 ws|  5 blobs|  94.39 mwei bfee|  33,073,697 ( 55.1%) gas used| 364.5ms ex>
2026-07-05 07:45:25.155+0200 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | FCU(VALID) | head: 556b4.....f65b6 | safe: 9c64e.....6da98 | finalized: ed6f2.....7a214
2026-07-05 07:45:37.157+0200 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #25,464,317  (eb4af.....08988)|  552 tx ( 89.9% parallel)| 16 ws|  3 blobs|  95.60 mwei bfee|  25,634,839 ( 42.7%) gas used| 98.3ms exe>
2026-07-05 07:45:37.360+0200 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | FCU(VALID) | head: eb4af.....08988 | safe: 9c64e.....6da98 | finalized: ed6f2.....7a214
2026-07-05 07:45:49.759+0200 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #25,464,318  (bf7a9.....d470d)|  204 tx ( 78.9% parallel)| 16 ws|  0 blobs|  93.86 mwei bfee|  18,741,412 ( 31.2%) gas used| 100.6ms ex>
2026-07-05 07:45:49.939+0200 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | FCU(VALID) | head: bf7a9.....d470d | safe: 9c64e.....6da98 | finalized: ed6f2.....7a214
2026-07-05 07:46:01.330+0200 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #25,464,319  (abb0f.....97bb1)|  182 tx ( 83.5% parallel)| 16 ws|  3 blobs|  89.46 mwei bfee|  15,948,462 ( 26.6%) gas used| 75.9ms exe>
2026-07-05 07:46:01.514+0200 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | FCU(VALID) | head: abb0f.....97bb1 | safe: 9c64e.....6da98 | finalized: ed6f2.....7a214
2026-07-05 07:46:13.092+0200 | vert.x-worker-thread-0 | INFO  | DefaultSynchronizer | 
####################################################################################################
#                                                                                                  #
# Besu has identified a problem with its worldstate database.                                      #
# Your node will fetch the correct data from peers to repair the problem.                          #
# Starting the sync pipeline...                                                                    #
#                                                                                                  #
####################################################################################################
2026-07-05 07:46:13.097+0200 | vert.x-worker-thread-0 | INFO  | TransactionPoolFactory | Disabling transaction handling during re-sync
2026-07-05 07:46:13.105+0200 | vert.x-worker-thread-0 | WARN  | BonsaiWorldStateProvider | Invalid node for account 0xeacd761dc4a3db736d24403c2d4961839a2bb4a5 at location 0x0c040704060207
2026-07-05 07:46:13.106+0200 | vert.x-worker-thread-0 | INFO  | BonsaiFlatDbStrategyProvider | setting FlatDbStrategy to PARTIAL
2026-07-05 07:46:13.107+0200 | vert.x-worker-thread-0 | INFO  | FlatDbStrategyProvider | Flat db mode found PARTIAL
2026-07-05 07:46:13.276+0200 | vert.x-worker-thread-0 | INFO  | DefaultSynchronizer | Starting synchronizer.
2026-07-05 07:46:13.277+0200 | vert.x-worker-thread-0 | INFO  | SnapSyncDownloader | Starting pivot-based sync
2026-07-05 07:46:13.285+0200 | vert.x-worker-thread-0 | INFO  | SnapWorldStateDownloader | Downloading world state from peers for pivot block 25464271 (0x9c64e0bfe573a87ff7b3fd1e926eb3aef42514cf803f801abb9fbf9fe356da98). State root 0xe>
2026-07-05 07:46:13.946+0200 | EthScheduler-Services-27729 | INFO  | SnapSyncChainDownloader | Created initial chain sync state: ChainSyncState{pivotBlockNumber=25464271, pivotBlockHash=0x9c64e0bfe573a87ff7b3fd1e926eb3aef42514cf803f801>
2026-07-05 07:46:13.949+0200 | EthScheduler-Services-27729 | INFO  | SnapSyncChainDownloader | Starting two-stage fast sync chain download from pivot block 0x9c64e0bfe573a87ff7b3fd1e926eb3aef42514cf803f801abb9fbf9fe356da98, 25464271, a>
2026-07-05 07:46:13.949+0200 | EthScheduler-Services-27729 | INFO  | SnapSyncChainDownloadPipelineFactory | Creating backward header download pipeline from upper=25464271 down to lower=0, parallelism=8, batchSize=192, peers=49
2026-07-05 07:46:14.177+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 0.00%
2026-07-05 07:46:44.709+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 2.27%
2026-07-05 07:47:13.363+0200 | EthScheduler-Services-27718 (batchPersistTrieNodeData) | INFO  | SnapSyncMetricsManager | Healed 63021 world state trie nodes, Peer count: 50
2026-07-05 07:47:14.711+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 5.57%
2026-07-05 07:47:44.900+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 8.76%
2026-07-05 07:48:15.089+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 12.09%
2026-07-05 07:48:45.092+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 15.44%
2026-07-05 07:49:16.932+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 18.62%
2026-07-05 07:49:49.917+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 20.46%
2026-07-05 07:50:19.933+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 23.21%
2026-07-05 07:50:49.935+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 26.82%
2026-07-05 07:51:20.157+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 30.06%
2026-07-05 07:51:52.258+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 33.11%
2026-07-05 07:52:22.506+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 36.09%
2026-07-05 07:52:52.685+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 39.35%
2026-07-05 07:53:22.686+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 42.54%
2026-07-05 07:53:53.351+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 44.90%
2026-07-05 07:54:23.518+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 46.95%
2026-07-05 07:54:53.553+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 48.99%
2026-07-05 07:55:23.793+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 51.69%
2026-07-05 07:55:54.085+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 54.94%
2026-07-05 07:56:24.202+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 59.17%
2026-07-05 07:56:54.697+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 62.43%
2026-07-05 07:57:24.698+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 65.97%
2026-07-05 07:57:54.699+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 70.53%
2026-07-05 07:58:24.699+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 75.71%
2026-07-05 07:58:54.700+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 81.17%
2026-07-05 07:59:24.700+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 84.87%
2026-07-05 07:59:54.948+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 91.08%
2026-07-05 08:00:25.588+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 93.37%
2026-07-05 08:00:56.047+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 96.25%
2026-07-05 08:01:26.609+0200 | EthScheduler-Services-27732 (importHeadersStep) | INFO  | ImportHeadersStep | Header import progress 98.72%
2026-07-05 08:01:47.371+0200 | EthScheduler-Services-27778 (importBlocks) | INFO  | ImportSyncBlocksStep | Block import progress: 15537522 of 25464271 (61%), Peer count: 46
2026-07-05 08:02:17.647+0200 | EthScheduler-Services-27778 (importBlocks) | INFO  | ImportSyncBlocksStep | Block import progress: 15553778 of 25464271 (61%), Peer count: 48

Versions (Add all that apply)

  • Software version: besu/v26.6.1/linux-x86_64/openjdk-java-25

  • Java version: openjdk version "25.0.3" 2026-04-21
    OpenJDK Runtime Environment (build 25.0.3+9-2-24.04.2-Ubuntu)
    OpenJDK 64-Bit Server VM (build 25.0.3+9-2-24.04.2-Ubuntu, mixed mode, sharing)

  • OS Name & Version: DISTRIB_ID=Ubuntu
    DISTRIB_RELEASE=24.04
    DISTRIB_CODENAME=noble
    DISTRIB_DESCRIPTION="Ubuntu 24.04.4 LTS"
    PRETTY_NAME="Ubuntu 24.04.4 LTS"
    NAME="Ubuntu"
    VERSION_ID="24.04"
    VERSION="24.04.4 LTS (Noble Numbat)"
    VERSION_CODENAME=noble
    ID=ubuntu
    ID_LIKE=debian
    HOME_URL="https://www.ubuntu.com/"
    SUPPORT_URL="https://help.ubuntu.com/"
    BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
    PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
    UBUNTU_CODENAME=noble
    LOGO=ubuntu-logo

  • Kernel Version: Linux new-staker 6.8.0-134-generic #134-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun 26 18:43:11 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux

  • Consensus Client & Version if using Proof of Stake: [e.g. Teku, Lighthouse, Prysm, Nimbus, Lodestar]

  • Lighthouse/v8.2.0-120c3c6

Additional Information (Add any of the following or anything else that may be relevant)

I was not doing anything on the machine

lighthouse beacon chain logs

Jul 05 07:45:41.002 INFO  Synced                                        peers: "61", exec_hash: "0xeb4a…8988 (verified)", finalized_root: 0xbbe4…8cf7, finalized_epoch: 459389, epoch: 459391, block: "0xc1f9…7e1f", slot: 14700526
Jul 05 07:45:49.578 INFO  New block received                            slot: 14700527, root: 0x3c368ba04ead37b31cd47375dda136963a7a3e2125bed5fdac2858cec805b46e
Jul 05 07:45:53.001 INFO  Synced                                        peers: "66", exec_hash: "0xbf7a…470d (verified)", finalized_root: 0xbbe4…8cf7, finalized_epoch: 459389, epoch: 459391, block: "0x3c36…b46e", slot: 14700527
Jul 05 07:46:01.153 INFO  New block received                            slot: 14700528, root: 0xdcd89bb5e99914aca381e0dd5a043a5444e5153cc701e20b9627004afde76f21
Jul 05 07:46:05.001 INFO  Synced                                        peers: "66", exec_hash: "0xabb0…7bb1 (verified)", finalized_root: 0xbbe4…8cf7, finalized_epoch: 459389, epoch: 459391, block: "0xdcd8…6f21", slot: 14700528
Jul 05 07:46:12.531 INFO  New block received                            slot: 14700529, root: 0xf29844c85ba022fd83311ca15977b6db4a09339ce49ab285639744035d2eef26
Jul 05 07:46:13.927 WARN  Execution engine call failed                  error: ServerMessage { code: -32603, message: "Internal error" }
Jul 05 07:46:13.927 WARN  Error whilst processing payload status        error: Api { error: ServerMessage { code: -32603, message: "Internal error" } }
Jul 05 07:46:17.001 INFO  Synced                                        peers: "64", exec_hash: "0xabb0…7bb1 (verified)", finalized_root: 0xbbe4…8cf7, finalized_epoch: 459389, epoch: 459391, block: "   …  empty", slot: 14700529
Jul 05 07:46:17.503 INFO  New RPC block received                        slot: 14700529, hash: 0xf29844c85ba022fd83311ca15977b6db4a09339ce49ab285639744035d2eef26
Jul 05 07:46:24.447 INFO  New block received                            slot: 14700530, root: 0xda8d8c442abbec10c0a29de9003d371f302add048e45b548fc49c96a8f4fff52
Jul 05 07:46:29.002 WARN  Head is optimistic                            info: "chain not fully verified, block and attestation production disabled until execution engine syncs", execution_block_hash: 0x5b7e7ec813635894544ab9ebd49053162>
Jul 05 07:46:29.002 INFO  Synced                                        peers: "66", exec_hash: "0x5b7e…3d0d (unverified)", finalized_root: 0xbbe4…8cf7, finalized_epoch: 459389, epoch: 459391, block: "0xda8d…ff52", slot: 14700530
Jul 05 07:46:36.778 INFO  New block received                            slot: 14700531, root: 0x32d605decf9656ee4495fcdb80a17c33f81ee8f0c4fd84799d57e1e36bba675d
Jul 05 07:46:41.001 WARN  Head is optimistic                            info: "chain not fully verified, block and attestation production disabled until execution engine syncs", execution_block_hash: 0x3f43bcdda1ce9bfd1844216c76eb12f2d>
Jul 05 07:46:41.001 INFO  Synced                                        peers: "60", exec_hash: "0x3f43…5124 (unverified)", finalized_root: 0xbbe4…8cf7, finalized_epoch: 459389, epoch: 459391, block: "0x32d6…675d", slot: 14700531
Jul 05 07:46:49.760 INFO  New block received                            slot: 14700532, root: 0x1933c9abf248d7059cfd42fea776afb46c317a3f73b12c9ab41870bec76b36bc
Jul 05 07:46:53.001 WARN  Head is optimistic                            info: "chain not fully verified, block and attestation production disabled until execution engine syncs", execution_block_hash: 0xada328f7f8781071aab9afea1f4f5cf43>
Jul 05 07:46:53.001 INFO  Synced                                        peers: "65", exec_hash: "0xada3…daa3 (unverified)", finalized_root: 0xbbe4…8cf7, finalized_epoch: 459389, epoch: 459391, block: "0x1933…36bc", slot: 14700532
Jul 05 07:47:05.001 WARN  Head is optimistic                            info: "chain not fully verified, block and attestation production disabled until execution engine syncs", execution_block_hash: 0xada328f7f8781071aab9afea1f4f5cf43>
Jul 05 07:47:05.002 INFO  Synced                                        peers: "66", exec_hash: "0xada3…daa3 (unverified)", finalized_root: 0xbbe4…8cf7, finalized_epoch: 459389, epoch: 459391, block: "   …  empty", slot: 14700533
Jul 05 07:47:12.402 INFO  New block received                            slot: 14700534, root: 0xfd92903b1c90176035b6320abfdfccdf9c335849bc39c06cadce0afb0e5144e9

####################################################################################################
#                                                                                                  #
# Besu version 26.6.1                                                                              #
#                                                                                                  #
# Configuration:                                                                                   #
# Network: Mainnet                                                                                 #
# Network Id: 1                                                                                    #
# Data storage: Bonsai                                                                             #
# Sync mode: Snap                                                                                  #
#   SNAP Sync server disabled                                                                      #
# Sync min peers: 5                                                                                #
# RPC HTTP APIs: ETH,NET,WEB3                                                                      #
# RPC HTTP port: 8545                                                                              #
# Engine APIs: ENGINE,ETH                                                                          #
# Engine port: 8551                                                                                #
# Engine JWT: /var/lib/jwtsecret/jwt.hex                                                           #
# Using LAYERED transaction pool implementation                                                    #
# Using STACKED worldstate update mode                                                             #
# Opcode optimizations enabled                                                                     #
# Parallel transaction processing enabled                                                          #
# Limit trie logs enabled: retention: 512; prune window: 5000                                      #
# BAL pruning enabled (retained BALs: 113056)                                                      #
# Target Gas Limit: 60M                                                                            #
#                                                                                                  #
# Host:                                                                                            #
# Java: openjdk-java-25                                                                            #
# Maximum heap size: 7.76 GB                                                                       #
# OS: linux-x86_64                                                                                 #
# glibc: 2.39                                                                                      #
# Total memory: 31.03 GB                                                                           #
# CPU cores: 28                                                                                    #
#                                                                                                  #
# Plugin Registration Summary:                                                                     #
# No plugins have been registered.                                                                 #
# TOTAL = 0 of 0 plugins successfully registered.                                                  #
#                                                                                                  #
####################################################################################################

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions