cosmos-sdk: Consensus fails when using statesync mode to synchronize the application state

Summary of Bug

Consensus fails when using statesync mode to synchronize the application state and then execute the ibc-transfer transaction.

Description

When the cosmos-sdk-based chain is started, the capability/keeper/keeper.go#L177:InitializeCapability(…) method will be called to initialize the memStore from the application store. However, if the node is started using statesync mode, the application store will not be loaded until the node is switched to fastsync mode. But in this case, the method InitializeCapability will not be called again to initialize memStore. Therefore, when calling the method capability/keeper/keeper.go#L344:GetCapability(…), the node started using statesync mode cannot get the same result as other node.

Steps to Reproduce

The GetCapability(…) mothod used in IBC module, so it can be reproduced through ibc-transfer:

1. Start two testnets via gaia and create relayer for them, then create clients and channels. Refer: https://github.com/cosmos/relayer#demo

2. Create node1, node2 to join testnet ibc-0:

   gaiad init node1 --home node1
   cp data/ibc-0/config/genesis.json node1/config/genesis.json
   
   gaiad init node2 --home node2
   cp data/ibc-0/config/genesis.json node2/config/genesis.json
   

   then update state-sync config in node1/config/app.toml and node2/config/app.toml:

   [state-sync]
   snapshot-interval = 100
   snapshot-keep-recent = 4
   

   Start node1 and node2:

   # NOTE: modify ports and add ibc-0 peer
   gaiad start --home node1
   
   # NOTE: modify ports and add ibc-0 peer
   gaiad start --home node2
   

3. Create node3 to join testnet ibc-0.

   gaiad init node3 --home node3
   cp data/ibc-0/config/genesis.json node3/config/genesis.json
   

   Update config:

   # config.toml
   [statesync]
   enable = true
   
   rpc_servers = "ibc-0 node rpc"
   trust_height = 1
   trust_hash = "block 1 hash"
   trust_period = "168h0m0s"
   

4. Send ibc-transfer

   rly tx transfer ibc-0 ibc-1 1000000samoleans $(rly chains address ibc-1)
   rly tx relay-packets demo -d
   

5. Start node3 using statesync mode

   # NOTE: modify ports and add ibc-0 peer
   gaiad start --home node3
   

   Get consensus failure error on executing the ibc-transfer transaction:

   NOTE: if the latest block height is greater than the ibc-transfer tranaction exexuted height, no error is returned, you can unsafe-reset-all node3 and repeat steps 4-5.

   4:49PM INF committed state app_hash=0475A43BE9A8BD240551895B01A31C5B1ABACD710DC273D819B863C9F355804C height=34721 module=state num_txs=1
   4:49PM INF indexed block height=34721 module=txindex
   panic: Failed to process committed block (34722:51062BEB78119D7A5CF971B7FEC787C428E8347FE20C308BF98311C2F95BFA1B): wrong Block.Header.AppHash.  Expected 0475A43BE9A8BD240551895B01A31C5B1ABACD710DC273D819B863C9F355804C, got 2443E8D78F4B2025252055EDD384DBF80839893092110C0A5D072DCABED9FB17
   
   goroutine 135 [running]:
   github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc000548a80, 0xc0032dac01)
     github.com/tendermint/tendermint@v0.34.9/blockchain/v0/reactor.go:401 +0x15bf
   created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).SwitchToFastSync
     github.com/tendermint/tendermint@v0.34.9/blockchain/v0/reactor.go:125 +0xd8
   

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned