Bug Description
When CommunityPool claims more funds than the bank account actually holds,
BeginBlocker: Passes the pre-check (reads CommunityPool)- Fails the bank transfer (reads real balance)
- Returns a raw error (not wrapped by
haltSchedule)
- Aborts
FinalizeBlock → CometBFT logs CONSENSUS FAILURE and panics
the function InitGenesis logs the error when bankBalance == poolAmount and does nothing :
func (k Keeper) InitGenesis(ctx sdk.Context, data types.GenesisState) {
if err := k.RewardPool.Set(ctx, data.RewardPool); err != nil {
panic(err)
}
if err := k.Params.Set(ctx, data.Params); err != nil {
panic(err)
}
if err := k.ReleaseSchedule.Set(ctx, data.ReleaseSchedule); err != nil {
panic(err)
}
// Consistency check: warn if the module bank balance for the configured denom
// does not match the CommunityPool accounting
denom := data.Params.TokenDenom
moduleAddr := authtypes.NewModuleAddress(types.ModuleName)
bankBalance := k.bankKeeper.GetBalance(ctx, moduleAddr, denom)
poolAmount := data.RewardPool.CommunityPool.AmountOf(denom).TruncateInt()
if !bankBalance.Amount.Equal(poolAmount) {
k.Logger(ctx).Error(
"rewards module bank balance does not match CommunityPool accounting at genesis", //@audit just logs the error
"denom", denom,
"bank_balance", bankBalance.Amount,
"community_pool", poolAmount,
)
}
}So here the code runs as usual with just logging the error and then if :
bank balance = 500,000 ukii
CommunityPool = 1,000,000 ukii ← phantom 500k
schedule: Active, TotalAmount=1,000,000, distributes ~5,000/block
The precheck in BeginBlocker
// Verify the CommunityPool has sufficient balance for the denom before transferring
poolAmount := rewardPool.CommunityPool.AmountOf(amountToDistribute.Denom)
if math.LegacyNewDecFromInt(amountToDistribute.Amount).GT(poolAmount) {
return k.haltSchedule(ctx, schedule, fmt.Errorf("community pool has insufficient balance for %s: pool has %s, need %s",
amountToDistribute.Denom, poolAmount, amountToDistribute.Amount))
}
// Send to distribution pool
if err := k.bankKeeper.SendCoinsFromModuleToModule(ctx, types.ModuleName, k.feeCollectorName, coinsToDistribute); err != nil {
return err//@audit-issue when SendCoinsFromModuleToModule fails it returns raw error and not haltSchedule error
}BeginBlocker line 66 — pre-check uses CommunityPool:
x/rewards/keeper/abci.go lines 65-68
poolAmount := rewardPool.CommunityPool.AmountOf(amountToDistribute.Denom)
if math.LegacyNewDecFromInt(amountToDistribute.Amount).GT(poolAmount) {
return k.haltSchedule(ctx, schedule, fmt.Errorf(...))
}poolAmount = 500,000. amountToDistribute = 5,000. 5,000 > 500,000 → false → pre-check PASSES.
BeginBlocker line 73 — bank send uses actual balance:
x/rewards/keeper/abci.go lines 72-74
if err := k.bankKeeper.SendCoinsFromModuleToModule(ctx, types.ModuleName, k.feeCollectorName,
coinsToDistribute); err != nil {
return err
}x/bank checks module account: 0 ukii. Needs 5,000 ukii. Returns ErrInsufficientFunds.
return err → BeginBlocker fails → FinalizeBlock returns error → CometBFT panics → all validators halt
simultaneously.
Steps to Reproduce
Provide deterministic steps to reproduce the issue.
Add a file e2e_rewards_chain_halt_test.go in the test folder and then run the test with following command to see the chain halt : go test -mod=readonly -timeout=20m -v ./tests/e2e -tags=test \ -run TestRewardsDualLedgerChainHaltE2E
package e2e
import (
"context"
"encoding/json"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
"github.com/ory/dockertest/v3"
"github.com/ory/dockertest/v3/docker"
"github.com/spf13/viper"
"github.com/stretchr/testify/require"
tmconfig "github.com/cometbft/cometbft/config"
tmjson "github.com/cometbft/cometbft/libs/json"
rpchttp "github.com/cometbft/cometbft/rpc/client/http"
"github.com/cosmos/cosmos-sdk/server"
srvconfig "github.com/cosmos/cosmos-sdk/server/config"
sdk "github.com/cosmos/cosmos-sdk/types"
authtypes "github.com/cosmos/cosmos-sdk/x/auth/types"
genutiltypes "github.com/cosmos/cosmos-sdk/x/genutil/types"
evmconfig "github.com/cosmos/evm/server/config"
kiichain "github.com/kiichain/kiichain/v7/app"
rewardstypes "github.com/kiichain/kiichain/v7/x/rewards/types"
)
// TestRewardsDualLedgerChainHaltE2E demonstrates the rewards dual-ledger bug on a live node:
// CommunityPool (KV) says 1M akii while bank holds 500k akii. BeginBlock passes the
// accounting pre-check, then SendCoinsFromModuleToModule fails → CONSENSUS FAILURE → halt.
//
// Prerequisites: Docker running, `make docker-build-debug` (kiichain/kiichaind-e2e image).
//
// Run only this test:
//
// SKIP_IBC_TESTS=true go test -mod=readonly -timeout=20m -v ./tests/e2e -tags=test \
// -run TestRewardsDualLedgerChainHaltE2E
func TestRewardsDualLedgerChainHaltE2E(t *testing.T) {
if os.Getenv("SKIP_CHAIN_HALT_E2E") == "true" {
t.Skip("SKIP_CHAIN_HALT_E2E=true")
}
cfg := defaultDualLedgerHaltConfig()
moduleAddr := authtypes.NewModuleAddress(rewardstypes.ModuleName)
t.Log("╔══════════════════════════════════════════════════════════════╗")
t.Log("║ E2E: Rewards dual-ledger → chain halt ║")
t.Log("╚══════════════════════════════════════════════════════════════╝")
t.Logf(" Real bank balance: %s akii", cfg.BankAmount)
t.Logf(" Phantom CommunityPool: %s akii", cfg.PoolAmount)
t.Logf(" Rewards module address: %s", moduleAddr)
t.Logf(" Schedule end: %s", cfg.ScheduleEnd.Format(time.RFC3339))
c, dkrPool, dkrNet, resource, rpcURL := setupHaltChain(t, cfg)
defer teardownHaltChain(t, dkrPool, dkrNet, resource, c)
t.Log("Phase 1: waiting for chain to produce blocks...")
rpcClient := waitForBlocks(t, rpcURL, resource.Container.ID, 5, 5*time.Minute)
chainEndpoint := fmt.Sprintf("http://%s", resource.GetHostPort("1317/tcp"))
poolBefore, err := queryRewardPool(chainEndpoint)
require.NoError(t, err)
poolAmt := poolBefore.RewardPool.CommunityPool.AmountOf(akiiDenom)
t.Logf(" CommunityPool at REST (phantom ledger): %s", poolAmt)
heightBeforeHalt := mustBlockHeight(t, rpcClient)
t.Logf(" Block height before halt watch: %d", heightBeforeHalt)
t.Log("Phase 2: monitoring for CONSENSUS FAILURE / insufficient funds (bank exhausted)...")
haltHeight, logSnippet := waitForChainHalt(t, rpcClient, resource, heightBeforeHalt, 12*time.Minute)
t.Log("╔══════════════════════════════════════════════════════════════╗")
t.Log("║ CHAIN HALT CONFIRMED ║")
t.Log("╚══════════════════════════════════════════════════════════════╝")
t.Logf(" Last height before stall: %d", heightBeforeHalt)
t.Logf(" Halt observed near height: %d", haltHeight)
t.Logf(" Container log excerpt:\n%s", logSnippet)
requireChainHaltEvidence(t, logSnippet)
require.True(t, haltHeight >= 2,
"halt should occur after block 1 (schedule timestamp init)")
}
func setupHaltChain(
t *testing.T,
cfg dualLedgerHaltConfig,
) (*chain, *dockertest.Pool, *dockertest.Network, *dockertest.Resource, string) {
t.Helper()
c, err := newChain()
require.NoError(t, err)
require.NoError(t, c.createAndInitValidators(1))
val := c.validators[0]
addr, err := val.keyInfo.GetAddress()
require.NoError(t, err)
require.NoError(t, modifyGenesis(val.configDir(), "", initBalanceStr, []sdk.AccAddress{addr}, akiiDenom))
writeHaltGenesis(t, c, cfg)
require.NoError(t, configureHaltValidator(val.configDir()))
dkrPool, err := dockertest.NewPool("")
require.NoError(t, err)
dkrNet, err := dkrPool.CreateNetwork(fmt.Sprintf("halt-%s", c.id))
require.NoError(t, err)
runOpts := &dockertest.RunOptions{
Name: val.instanceName(),
NetworkID: dkrNet.Network.ID,
Mounts: []string{
fmt.Sprintf("%s/:%s", val.configDir(), kiichainHomePath),
},
Repository: "kiichain/kiichaind-e2e",
PortBindings: map[docker.Port][]docker.PortBinding{
"1317/tcp": {{HostIP: "", HostPort: "13170"}},
"26657/tcp": {{HostIP: "", HostPort: "26670"}},
},
}
require.NoError(t, exec.Command("chmod", "-R", "0777", val.configDir()).Run()) //nolint:gosec // test fixture
resource, err := dkrPool.RunWithOptions(runOpts, noRestart)
require.NoError(t, err)
rpcURL := fmt.Sprintf("tcp://%s", resource.GetHostPort("26657/tcp"))
t.Logf(" Chain ID: %s", c.id)
t.Logf(" RPC: %s", rpcURL)
t.Logf(" REST: http://%s", resource.GetHostPort("1317/tcp"))
t.Logf(" Container: %s", resource.Container.ID[:12])
return c, dkrPool, dkrNet, resource, rpcURL
}
func writeHaltGenesis(t *testing.T, c *chain, cfg dualLedgerHaltConfig) {
t.Helper()
val := c.validators[0]
serverCtx := server.NewDefaultContext()
config := serverCtx.Config
config.SetRoot(val.configDir())
genFilePath := config.GenesisFile()
appGenState, genDoc, err := genutiltypes.GenesisStateFromGenFile(genFilePath)
require.NoError(t, err)
require.NoError(t, injectRewardsDualLedgerGenesis(appGenState, cfg))
require.NoError(t, ensureGenesisEvmAndBankMetadata(appGenState))
createValmsg, err := val.buildCreateValidatorMsg(stakingAmountCoin)
require.NoError(t, err)
signedTx, err := val.signMsg(createValmsg)
require.NoError(t, err)
txRaw, err := cdc.MarshalJSON(signedTx)
require.NoError(t, err)
var genUtilGenState genutiltypes.GenesisState
require.NoError(t, cdc.UnmarshalJSON(appGenState[genutiltypes.ModuleName], &genUtilGenState))
genUtilGenState.GenTxs = []json.RawMessage{txRaw}
appGenState[genutiltypes.ModuleName], err = cdc.MarshalJSON(&genUtilGenState)
require.NoError(t, err)
genDoc.AppState, err = json.MarshalIndent(appGenState, "", " ")
require.NoError(t, err)
bz, err := tmjson.MarshalIndent(genDoc, "", " ")
require.NoError(t, err)
require.NoError(t, writeFile(filepath.Join(val.configDir(), "config", "genesis.json"), bz))
}
func configureHaltValidator(configDir string) error {
tmCfgPath := filepath.Join(configDir, "config", "config.toml")
vpr := viper.New()
vpr.SetConfigFile(tmCfgPath)
if err := vpr.ReadInConfig(); err != nil {
return err
}
valConfig := tmconfig.DefaultConfig()
if err := vpr.Unmarshal(valConfig); err != nil {
return err
}
valConfig.P2P.ListenAddress = "tcp://0.0.0.0:26656"
valConfig.P2P.AddrBookStrict = false
valConfig.RPC.ListenAddress = "tcp://0.0.0.0:26657"
valConfig.Consensus.TimeoutPropose = 500 * time.Millisecond
valConfig.Consensus.TimeoutPrevote = 300 * time.Millisecond
valConfig.Consensus.TimeoutPrecommit = 300 * time.Millisecond
valConfig.Consensus.TimeoutCommit = 800 * time.Millisecond
valConfig.LogLevel = "info"
tmconfig.WriteConfigFile(tmCfgPath, valConfig)
appCfgPath := filepath.Join(configDir, "config", "app.toml")
appConfig := srvconfig.DefaultConfig()
appConfig.API.Enable = true
appConfig.API.Address = "tcp://0.0.0.0:1317"
appConfig.MinGasPrices = fmt.Sprintf("%s%s", minGasPrice, akiiDenom)
appConfig.GRPC.Address = "0.0.0.0:9090"
evmCfg := evmconfig.DefaultConfig()
evmCfg.EVM.EVMChainID = kiichain.KiichainID
evmCfg.Config = *appConfig
srvconfig.SetConfigTemplate(srvconfig.DefaultConfigTemplate + evmconfig.DefaultEVMConfigTemplate)
srvconfig.WriteConfigFile(appCfgPath, evmCfg)
return nil
}
func waitForBlocks(
t *testing.T,
rpcURL string,
containerID string,
minHeight int64,
timeout time.Duration,
) *rpchttp.HTTP {
t.Helper()
rpcClient, err := rpchttp.New(rpcURL, "/websocket")
require.NoError(t, err)
deadline := time.Now().Add(timeout)
for time.Now().Before(deadline) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
status, err := rpcClient.Status(ctx)
cancel()
if err == nil && !status.SyncInfo.CatchingUp && status.SyncInfo.LatestBlockHeight >= minHeight {
return rpcClient
}
time.Sleep(time.Second)
}
t.Logf("node failed to reach height %d; docker logs:\n%s", minHeight, tailContainerLogs(containerID, 200))
require.FailNow(t, "node failed to produce blocks")
return rpcClient
}
func mustBlockHeight(t *testing.T, rpcClient *rpchttp.HTTP) int64 {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
status, err := rpcClient.Status(ctx)
require.NoError(t, err)
return status.SyncInfo.LatestBlockHeight
}
func waitForChainHalt(
t *testing.T,
rpcClient *rpchttp.HTTP,
resource *dockertest.Resource,
startHeight int64,
timeout time.Duration,
) (int64, string) {
t.Helper()
deadline := time.Now().Add(timeout)
var lastHeight int64 = startHeight
stallSince := time.Time{}
var lastLogs string
for time.Now().Before(deadline) {
logs := tailContainerLogs(resource.Container.ID, 80)
lastLogs = logs
lower := strings.ToLower(logs)
if strings.Contains(lower, "consensus failure") ||
strings.Contains(lower, "insufficient funds") ||
strings.Contains(lower, "failed to apply block") {
h := mustBlockHeight(t, rpcClient)
return h, logs
}
h := mustBlockHeight(t, rpcClient)
if h > lastHeight {
lastHeight = h
stallSince = time.Time{}
t.Logf(" block height %d", h)
} else if h >= 3 && stallSince.IsZero() {
stallSince = time.Now()
} else if !stallSince.IsZero() && time.Since(stallSince) > 25*time.Second {
if strings.Contains(strings.ToLower(lastLogs), "insufficient") ||
strings.Contains(strings.ToLower(lastLogs), "consensus failure") {
return h, lastLogs
}
}
time.Sleep(2 * time.Second)
}
t.Fatalf("timed out waiting for chain halt; last logs:\n%s", lastLogs)
return 0, ""
}
func tailContainerLogs(containerID string, lines int) string {
out, err := exec.Command( //nolint:gosec // test-only docker logs
"docker", "logs", "--tail", fmt.Sprintf("%d", lines), containerID,
).CombinedOutput()
if err != nil {
return string(out)
}
return string(out)
}
func teardownHaltChain(
t *testing.T,
pool *dockertest.Pool,
net *dockertest.Network,
resource *dockertest.Resource,
c *chain,
) {
t.Helper()
if os.Getenv("KIICHAIN_E2E_SKIP_CLEANUP") != "" {
t.Log("KIICHAIN_E2E_SKIP_CLEANUP set — leaving containers and datadir")
return
}
if resource != nil {
_ = pool.Purge(resource)
}
if net != nil {
_ = pool.RemoveNetwork(net)
}
if c != nil {
_ = os.RemoveAll(c.dataDir)
}
}
=== RUN TestRewardsDualLedgerChainHaltE2E
e2e_rewards_chain_halt_test.go:53: ╔══════════════════════════════════════════════════════════════╗
e2e_rewards_chain_halt_test.go:54: ║ E2E: Rewards dual-ledger → chain halt ║
e2e_rewards_chain_halt_test.go:55: ╚══════════════════════════════════════════════════════════════╝
e2e_rewards_chain_halt_test.go:56: Real bank balance: 500000 akii
e2e_rewards_chain_halt_test.go:57: Phantom CommunityPool: 1000000 akii
e2e_rewards_chain_halt_test.go:58: Rewards module address: kii1245yut9zht8q4hz39sd0lzqtzkuw5us5rermc4
e2e_rewards_chain_halt_test.go:59: Schedule end: 2026-05-27T15:17:42Z
e2e_rewards_chain_halt_test.go:61: Chain ID: localchain_1336-641011424771825542
e2e_rewards_chain_halt_test.go:61: RPC: tcp://localhost:26670
e2e_rewards_chain_halt_test.go:61: REST: http://localhost:13170
e2e_rewards_chain_halt_test.go:61: Container: 513d959e5c82
e2e_rewards_chain_halt_test.go:64: Phase 1: waiting for chain to produce blocks...
e2e_rewards_chain_halt_test.go:71: CommunityPool at REST (phantom ledger): 968664.000000000000000000
e2e_rewards_chain_halt_test.go:74: Block height before halt watch: 5
e2e_rewards_chain_halt_test.go:76: Phase 2: monitoring for CONSENSUS FAILURE / insufficient funds (bank exhausted)...
e2e_rewards_chain_halt_test.go:77: block height 8
e2e_rewards_chain_halt_test.go:77: block height 10
e2e_rewards_chain_halt_test.go:77: block height 13
e2e_rewards_chain_halt_test.go:77: block height 15
e2e_rewards_chain_halt_test.go:77: block height 18
e2e_rewards_chain_halt_test.go:77: block height 20
e2e_rewards_chain_halt_test.go:77: block height 23
e2e_rewards_chain_halt_test.go:77: block height 25
e2e_rewards_chain_halt_test.go:77: block height 28
e2e_rewards_chain_halt_test.go:77: block height 30
e2e_rewards_chain_halt_test.go:77: block height 33
e2e_rewards_chain_halt_test.go:77: block height 36
e2e_rewards_chain_halt_test.go:77: block height 38
e2e_rewards_chain_halt_test.go:77: block height 41
e2e_rewards_chain_halt_test.go:77: block height 43
e2e_rewards_chain_halt_test.go:77: block height 45
e2e_rewards_chain_halt_test.go:77: block height 48
e2e_rewards_chain_halt_test.go:77: block height 50
e2e_rewards_chain_halt_test.go:77: block height 53
e2e_rewards_chain_halt_test.go:77: block height 55
e2e_rewards_chain_halt_test.go:77: block height 58
e2e_rewards_chain_halt_test.go:77: block height 60
e2e_rewards_chain_halt_test.go:77: block height 63
e2e_rewards_chain_halt_test.go:77: block height 66
e2e_rewards_chain_halt_test.go:77: block height 68
e2e_rewards_chain_halt_test.go:77: block height 71
e2e_rewards_chain_halt_test.go:77: block height 73
e2e_rewards_chain_halt_test.go:77: block height 76
e2e_rewards_chain_halt_test.go:77: block height 78
e2e_rewards_chain_halt_test.go:77: block height 81
e2e_rewards_chain_halt_test.go:77: block height 83
e2e_rewards_chain_halt_test.go:77: block height 86
e2e_rewards_chain_halt_test.go:77: block height 88
e2e_rewards_chain_halt_test.go:77: block height 91
e2e_rewards_chain_halt_test.go:77: block height 94
e2e_rewards_chain_halt_test.go:77: block height 96
e2e_rewards_chain_halt_test.go:77: block height 99
e2e_rewards_chain_halt_test.go:77: block height 101
e2e_rewards_chain_halt_test.go:77: block height 104
e2e_rewards_chain_halt_test.go:77: block height 106
e2e_rewards_chain_halt_test.go:77: block height 109
e2e_rewards_chain_halt_test.go:79: ╔══════════════════════════════════════════════════════════════╗
e2e_rewards_chain_halt_test.go:80: ║ CHAIN HALT CONFIRMED ║
e2e_rewards_chain_halt_test.go:81: ╚══════════════════════════════════════════════════════════════╝
e2e_rewards_chain_halt_test.go:82: Last height before stall: 5
e2e_rewards_chain_halt_test.go:83: Halt observed near height: 109
e2e_rewards_chain_halt_test.go:84: Container log excerpt:
3:16PM INF indexed block events height=100 module=txindex server=node
3:16PM INF Timed out dur=792.661959 height=101 module=consensus round=0 server=node step=RoundStepNewHeight
3:16PM INF received proposal module=consensus proposal="Proposal{101/0 (C6FA33C916DA34EC5CDF6BEA0379ED59FA901F3EEAD40FC6F9854825C59254A6:1:853FCF2B2506, -1) D74A0CCBE50E @ 2026-05-27T15:16:07.060669927Z}" proposer=39AAFA02D13D7F71A7C78B92F6CECE4B94CFE154 server=node
3:16PM INF received complete proposal block hash=C6FA33C916DA34EC5CDF6BEA0379ED59FA901F3EEAD40FC6F9854825C59254A6 height=101 module=consensus server=node
3:16PM INF finalizing commit of block hash=C6FA33C916DA34EC5CDF6BEA0379ED59FA901F3EEAD40FC6F9854825C59254A6 height=101 module=consensus num_txs=0 root=019525A570A425B7C320FCEB208999EA0DF18E8A91059AB29C8F95461ACDFDEC server=node
3:16PM INF Aggregating exchange rates module=x/oracle
3:16PM INF finalized block block_app_hash=A7E22E5C3283E175F08D46A6022DF0F454334975828EA873A62EDEC9D340D233 height=101 module=state num_txs_res=0 num_val_updates=0 server=node
3:16PM INF executed block app_hash=A7E22E5C3283E175F08D46A6022DF0F454334975828EA873A62EDEC9D340D233 height=101 module=state server=node
3:16PM INF committed state block_app_hash=019525A570A425B7C320FCEB208999EA0DF18E8A91059AB29C8F95461ACDFDEC height=101 module=state server=node
3:16PM INF indexed block events height=101 module=txindex server=node
3:16PM INF Timed out dur=782.364083 height=102 module=consensus round=0 server=node step=RoundStepNewHeight
3:16PM INF received proposal module=consensus proposal="Proposal{102/0 (52F43ED92D3C59FE72BFD2230737D7D19213358DC07B36DFE7818A83E5376E9F:1:E2DB9B5AFE2A, -1) 0FF31022FB48 @ 2026-05-27T15:16:07.881020261Z}" proposer=39AAFA02D13D7F71A7C78B92F6CECE4B94CFE154 server=node
3:16PM INF received complete proposal block hash=52F43ED92D3C59FE72BFD2230737D7D19213358DC07B36DFE7818A83E5376E9F height=102 module=consensus server=node
3:16PM INF finalizing commit of block hash=52F43ED92D3C59FE72BFD2230737D7D19213358DC07B36DFE7818A83E5376E9F height=102 module=consensus num_txs=0 root=A7E22E5C3283E175F08D46A6022DF0F454334975828EA873A62EDEC9D340D233 server=node
3:16PM INF finalized block block_app_hash=04DED6162141F6DBFD2B549E0A0E050516171FBE4CA646329BE5CF0F7204FCA8 height=102 module=state num_txs_res=0 num_val_updates=0 server=node
3:16PM INF executed block app_hash=04DED6162141F6DBFD2B549E0A0E050516171FBE4CA646329BE5CF0F7204FCA8 height=102 module=state server=node
3:16PM INF committed state block_app_hash=A7E22E5C3283E175F08D46A6022DF0F454334975828EA873A62EDEC9D340D233 height=102 module=state server=node
3:16PM INF indexed block events height=102 module=txindex server=node
3:16PM INF Timed out dur=789.143167 height=103 module=consensus round=0 server=node step=RoundStepNewHeight
3:16PM INF received proposal module=consensus proposal="Proposal{103/0 (62DE7353EE6FC5074749D0419E93B0D9417E21BB5C62C1A9F751EECDDF453795:1:AD1CFF95C3FB, -1) 901715B4A15A @ 2026-05-27T15:16:08.704576928Z}" proposer=39AAFA02D13D7F71A7C78B92F6CECE4B94CFE154 server=node
3:16PM INF received complete proposal block hash=62DE7353EE6FC5074749D0419E93B0D9417E21BB5C62C1A9F751EECDDF453795 height=103 module=consensus server=node
3:16PM INF finalizing commit of block hash=62DE7353EE6FC5074749D0419E93B0D9417E21BB5C62C1A9F751EECDDF453795 height=103 module=consensus num_txs=0 root=04DED6162141F6DBFD2B549E0A0E050516171FBE4CA646329BE5CF0F7204FCA8 server=node
3:16PM INF Aggregating exchange rates module=x/oracle
3:16PM INF finalized block block_app_hash=C44FAC717D1CDCADC4B8B3109D76E29B3104FDA3BCBEBA35EC66739A1ED7223B height=103 module=state num_txs_res=0 num_val_updates=0 server=node
3:16PM INF executed block app_hash=C44FAC717D1CDCADC4B8B3109D76E29B3104FDA3BCBEBA35EC66739A1ED7223B height=103 module=state server=node
3:16PM INF committed state block_app_hash=04DED6162141F6DBFD2B549E0A0E050516171FBE4CA646329BE5CF0F7204FCA8 height=103 module=state server=node
3:16PM INF indexed block events height=103 module=txindex server=node
3:16PM INF Timed out dur=788.945834 height=104 module=consensus round=0 server=node step=RoundStepNewHeight
3:16PM INF received proposal module=consensus proposal="Proposal{104/0 (7F5A21D3E272FCAE78926A36568E914DC2524C34685B8596CFF9B334658AA3DF:1:EA7331B2BB46, -1) E227A798579D @ 2026-05-27T15:16:09.516950803Z}" proposer=39AAFA02D13D7F71A7C78B92F6CECE4B94CFE154 server=node
3:16PM INF received complete proposal block hash=7F5A21D3E272FCAE78926A36568E914DC2524C34685B8596CFF9B334658AA3DF height=104 module=consensus server=node
3:16PM INF finalizing commit of block hash=7F5A21D3E272FCAE78926A36568E914DC2524C34685B8596CFF9B334658AA3DF height=104 module=consensus num_txs=0 root=C44FAC717D1CDCADC4B8B3109D76E29B3104FDA3BCBEBA35EC66739A1ED7223B server=node
3:16PM INF finalized block block_app_hash=41F513FDD4B5CAA9CA1663739A00146A1EDCD3F404078F0A229750AA7B741859 height=104 module=state num_txs_res=0 num_val_updates=0 server=node
3:16PM INF executed block app_hash=41F513FDD4B5CAA9CA1663739A00146A1EDCD3F404078F0A229750AA7B741859 height=104 module=state server=node
3:16PM INF committed state block_app_hash=C44FAC717D1CDCADC4B8B3109D76E29B3104FDA3BCBEBA35EC66739A1ED7223B height=104 module=state server=node
3:16PM INF indexed block events height=104 module=txindex server=node
3:16PM INF Timed out dur=795.24 height=105 module=consensus round=0 server=node step=RoundStepNewHeight
3:16PM INF received proposal module=consensus proposal="Proposal{105/0 (06BA004DFF3ACBFB4D66F24A7DDACD2F90220B3803D8838919E075634C23ED04:1:6ED0C5661457, -1) 78A5CE51FA72 @ 2026-05-27T15:16:10.324991762Z}" proposer=39AAFA02D13D7F71A7C78B92F6CECE4B94CFE154 server=node
3:16PM INF received complete proposal block hash=06BA004DFF3ACBFB4D66F24A7DDACD2F90220B3803D8838919E075634C23ED04 height=105 module=consensus server=node
3:16PM INF finalizing commit of block hash=06BA004DFF3ACBFB4D66F24A7DDACD2F90220B3803D8838919E075634C23ED04 height=105 module=consensus num_txs=0 root=41F513FDD4B5CAA9CA1663739A00146A1EDCD3F404078F0A229750AA7B741859 server=node
3:16PM INF Aggregating exchange rates module=x/oracle
3:16PM INF finalized block block_app_hash=75C5970ABA97FFEC26508A4B5AF8F1A8253F70F13F9A2114DB054111C2EB39BF height=105 module=state num_txs_res=0 num_val_updates=0 server=node
3:16PM INF executed block app_hash=75C5970ABA97FFEC26508A4B5AF8F1A8253F70F13F9A2114DB054111C2EB39BF height=105 module=state server=node
3:16PM INF committed state block_app_hash=41F513FDD4B5CAA9CA1663739A00146A1EDCD3F404078F0A229750AA7B741859 height=105 module=state server=node
3:16PM INF indexed block events height=105 module=txindex server=node
3:16PM INF Timed out dur=790.947708 height=106 module=consensus round=0 server=node step=RoundStepNewHeight
3:16PM INF received proposal module=consensus proposal="Proposal{106/0 (A12AF3599C3D3FA99165C0D3EFEFBA7462A4022B377F53E9ED7D3E79408937D5:1:C41BBB434CF3, -1) 9151954EAF75 @ 2026-05-27T15:16:11.137049804Z}" proposer=39AAFA02D13D7F71A7C78B92F6CECE4B94CFE154 server=node
3:16PM INF received complete proposal block hash=A12AF3599C3D3FA99165C0D3EFEFBA7462A4022B377F53E9ED7D3E79408937D5 height=106 module=consensus server=node
3:16PM INF finalizing commit of block hash=A12AF3599C3D3FA99165C0D3EFEFBA7462A4022B377F53E9ED7D3E79408937D5 height=106 module=consensus num_txs=0 root=75C5970ABA97FFEC26508A4B5AF8F1A8253F70F13F9A2114DB054111C2EB39BF server=node
3:16PM INF finalized block block_app_hash=26B6D3CA23EC7349FDF0CCBCF115571BCC651261EAF1DAD4F0C9716FE156A783 height=106 module=state num_txs_res=0 num_val_updates=0 server=node
3:16PM INF executed block app_hash=26B6D3CA23EC7349FDF0CCBCF115571BCC651261EAF1DAD4F0C9716FE156A783 height=106 module=state server=node
3:16PM INF committed state block_app_hash=75C5970ABA97FFEC26508A4B5AF8F1A8253F70F13F9A2114DB054111C2EB39BF height=106 module=state server=node
3:16PM INF indexed block events height=106 module=txindex server=node
3:16PM INF Timed out dur=791.047292 height=107 module=consensus round=0 server=node step=RoundStepNewHeight
3:16PM INF received proposal module=consensus proposal="Proposal{107/0 (4A755A0988EDEA9F14827AC718CE42EE87F3F28C0A736D32867A812CAE94B439:1:8C3CD7E343A2, -1) 84A8A416BF9C @ 2026-05-27T15:16:11.946907721Z}" proposer=39AAFA02D13D7F71A7C78B92F6CECE4B94CFE154 server=node
3:16PM INF received complete proposal block hash=4A755A0988EDEA9F14827AC718CE42EE87F3F28C0A736D32867A812CAE94B439 height=107 module=consensus server=node
3:16PM INF finalizing commit of block hash=4A755A0988EDEA9F14827AC718CE42EE87F3F28C0A736D32867A812CAE94B439 height=107 module=consensus num_txs=0 root=26B6D3CA23EC7349FDF0CCBCF115571BCC651261EAF1DAD4F0C9716FE156A783 server=node
3:16PM INF Aggregating exchange rates module=x/oracle
3:16PM INF finalized block block_app_hash=EEB514611AC942421D109D400DBCF665920828CD96FBD21F8224634C6F2398BA height=107 module=state num_txs_res=0 num_val_updates=0 server=node
3:16PM INF executed block app_hash=EEB514611AC942421D109D400DBCF665920828CD96FBD21F8224634C6F2398BA height=107 module=state server=node
3:16PM INF committed state block_app_hash=26B6D3CA23EC7349FDF0CCBCF115571BCC651261EAF1DAD4F0C9716FE156A783 height=107 module=state server=node
3:16PM INF indexed block events height=107 module=txindex server=node
3:16PM INF Timed out dur=792.70425 height=108 module=consensus round=0 server=node step=RoundStepNewHeight
3:16PM INF received proposal module=consensus proposal="Proposal{108/0 (BF86EDC885ADD41C1067033A279A20F4C25F0448AA27C8929738867CB3371184:1:C935A74195DD, -1) B813F96779A4 @ 2026-05-27T15:16:12.756035972Z}" proposer=39AAFA02D13D7F71A7C78B92F6CECE4B94CFE154 server=node
3:16PM INF received complete proposal block hash=BF86EDC885ADD41C1067033A279A20F4C25F0448AA27C8929738867CB3371184 height=108 module=consensus server=node
3:16PM INF finalizing commit of block hash=BF86EDC885ADD41C1067033A279A20F4C25F0448AA27C8929738867CB3371184 height=108 module=consensus num_txs=0 root=EEB514611AC942421D109D400DBCF665920828CD96FBD21F8224634C6F2398BA server=node
3:16PM INF finalized block block_app_hash=392A89DD8718E90856E78C72DAE44CA37D3B4DD40C7B07689ACBEDB4B34864EE height=108 module=state num_txs_res=0 num_val_updates=0 server=node
3:16PM INF executed block app_hash=392A89DD8718E90856E78C72DAE44CA37D3B4DD40C7B07689ACBEDB4B34864EE height=108 module=state server=node
3:16PM INF committed state block_app_hash=EEB514611AC942421D109D400DBCF665920828CD96FBD21F8224634C6F2398BA height=108 module=state server=node
3:16PM INF indexed block events height=108 module=txindex server=node
3:16PM INF Timed out dur=793.970292 height=109 module=consensus round=0 server=node step=RoundStepNewHeight
3:16PM INF received proposal module=consensus proposal="Proposal{109/0 (5F5AF0C79B65305C1C88EE4D9AE176906C9D0B095FB3A3B7F5364C8529AF2668:1:3C48E6A8B15D, -1) 71CF830A252C @ 2026-05-27T15:16:13.567090264Z}" proposer=39AAFA02D13D7F71A7C78B92F6CECE4B94CFE154 server=node
3:16PM INF received complete proposal block hash=5F5AF0C79B65305C1C88EE4D9AE176906C9D0B095FB3A3B7F5364C8529AF2668 height=109 module=consensus server=node
3:16PM INF finalizing commit of block hash=5F5AF0C79B65305C1C88EE4D9AE176906C9D0B095FB3A3B7F5364C8529AF2668 height=109 module=consensus num_txs=0 root=392A89DD8718E90856E78C72DAE44CA37D3B4DD40C7B07689ACBEDB4B34864EE server=node
3:16PM ERR error in proxyAppConn.FinalizeBlock err="spendable balance 3437akii is smaller than 4501akii: insufficient funds" module=state server=node
3:16PM ERR CONSENSUS FAILURE!!! err="failed to apply block; error spendable balance 3437akii is smaller than 4501akii: insufficient funds [cosmos/cosmos-sdk@v0.53.6/x/bank/keeper/send.go:298]" module=consensus server=node stack="goroutine 367 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:26 +0x64\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine.func2()\n\tgithub.com/cometbft/cometbft@v0.38.21/consensus/state.go:801 +0x44\npanic({0x3d72ae0?, 0x4001f23ba0?})\n\truntime/panic.go:792 +0x124\ngithub.com/cometbft/cometbft/consensus.(*State).finalizeCommit(0x40027eb508, 0x6d)\n\tgithub.com/cometbft/cometbft@v0.38.21/consensus/state.go:1784 +0xae0\ngithub.com/cometbft/cometbft/consensus.(*State).tryFinalizeCommit(0x40027eb508, 0x6d)\n\tgithub.com/cometbft/cometbft@v0.38.21/consensus/state.go:1683 +0x268\ngithub.com/cometbft/cometbft/consensus.(*State).enterCommit.func1()\n\tgithub.com/cometbft/cometbft@v0.38.21/consensus/state.go:1618 +0x8c\ngithub.com/cometbft/cometbft/consensus.(*State).enterCommit(0x40027eb508, 0x6d, 0x0)\n\tgithub.com/cometbft/cometbft@v0.38.21/consensus/state.go:1656 +0xa90\ngithub.com/cometbft/cometbft/consensus.(*State).addVote(0x40027eb508, 0x4000b1e820, {0x0, 0x0})\n\tgithub.com/cometbft/cometbft@v0.38.21/consensus/state.go:2346 +0x19c4\ngithub.com/cometbft/cometbft/consensus.(*State).tryAddVote(0x40027eb508, 0x4000b1e820, {0x0?, 0x4fc3f0?})\n\tgithub.com/cometbft/cometbft@v0.38.21/consensus/state.go:2070 +0x28\ngithub.com/cometbft/cometbft/consensus.(*State).handleMsg(0x40027eb508, {{0x66b0d20, 0x40033b4180}, {0x0, 0x0}})\n\tgithub.com/cometbft/cometbft@v0.38.21/consensus/state.go:929 +0x334\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine(0x40027eb508, 0x0)\n\tgithub.com/cometbft/cometbft@v0.38.21/consensus/state.go:856 +0x35c\ncreated by github.com/cometbft/cometbft/consensus.(*State).OnStart in goroutine 1\n\tgithub.com/cometbft/cometbft@v0.38.21/consensus/state.go:398 +0xe8\n"
3:16PM INF service stop impl=baseWAL module=consensus msg="Stopping baseWAL service" server=node wal=/home/nonroot/.kiichain/data/cs.wal/wal
3:16PM INF service stop impl=Group module=consensus msg="Stopping Group service" server=node wal=/home/nonroot/.kiichain/data/cs.wal/wal
3:16PM INF Timed out dur=500 height=109 module=consensus round=0 server=node step=RoundStepPropose
3:16PM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10 server=node
3:16PM INF No addresses to dial. Falling back to seeds module=pex server=node
The chain halts after a few
Expected Behavior
The chain should work if the ledger balance are out of sync or should just return haltSchedule
Actual Behavior
While bank still has coins, blocks look healthy. Once bank is empty but CommunityPool still shows funds, the pre-check passes, the bank send fails, and every validator panics on the same block.
Impact Assessment
Chain halt after a faulty upgrade where the communityPool and Bank balance differs and gets out of sync
Reporter Declaration
By submitting this issue, you confirm that:
Bug Description
When
CommunityPoolclaims more funds than the bank account actually holds,BeginBlocker: Passes the pre-check (readsCommunityPool)haltSchedule)FinalizeBlock→ CometBFT logs CONSENSUS FAILURE and panicsthe function
InitGenesislogs the error whenbankBalance == poolAmountand does nothing :So here the code runs as usual with just logging the error and then if :
The precheck in
BeginBlockerBeginBlocker line 66 — pre-check uses CommunityPool:
x/rewards/keeper/abci.go lines 65-68
poolAmount = 500,000. amountToDistribute = 5,000. 5,000 > 500,000 → false → pre-check PASSES.
BeginBlocker line 73 — bank send uses actual balance:
x/rewards/keeper/abci.go lines 72-74
x/bank checks module account: 0 ukii. Needs 5,000 ukii. Returns ErrInsufficientFunds.
return err → BeginBlocker fails → FinalizeBlock returns error → CometBFT panics → all validators halt
simultaneously.
Steps to Reproduce
Provide deterministic steps to reproduce the issue.
Add a file
e2e_rewards_chain_halt_test.goin the test folder and then run the test with following command to see the chain halt :go test -mod=readonly -timeout=20m -v ./tests/e2e -tags=test \ -run TestRewardsDualLedgerChainHaltE2EThe chain halts after a few
Expected Behavior
The chain should work if the ledger balance are out of sync or should just return
haltScheduleActual Behavior
While bank still has coins, blocks look healthy. Once bank is empty but CommunityPool still shows funds, the pre-check passes, the bank send fails, and every validator panics on the same block.
Impact Assessment
Chain halt after a faulty upgrade where the communityPool and Bank balance differs and gets out of sync
Reporter Declaration
By submitting this issue, you confirm that: