cosmos-sdk: gRPC Tx simulation sometimes fails with account sequence mismatch
Summary of Bug
gRPC Tx simulation sometimes fails with account sequence mismatch. This is seen with a higher number of Tx-es spawning multiple blocks.
Version
v0.45.1 (running gaia v7.0.0)
Steps to Reproduce
To reproduce I run hermes relayer with a high number of transactions. In the log below there are 526 IBC messages generated. Hermes bundles 15 per transaction and performs a tx simulate followed by broadcast_tx_sync. Normally if simulation fails hermes breaks the loop but I modified it here to continue. The error in this run is:
"account sequence mismatch, expected 1509, got 1512: incorrect account sequence", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "5481"} }
But then the next simulate seems perfectly fine, the simulate for seg 1513 works fine.
The symptom is that the simulation context sometimes “forgets” previous calls but recovers quickly after. Not sure if there are some caching glitches (maybe around block boundary) or we are using the gRPC wrongly. If the former I expect this to be reproducible with any type of transactions.
In this run:
- first Tx (seq 1508) was eventually included in block 5481
- the next (1508-1538), including the one that failed simulation (seq 1512) were included in 5482:
- 1508-1511 simulation works fine
- 1512 simulation fails
- 1513-… simulation works fine
- the last (1539-1543) were included in 5483
Below are more logs up acc seq 1513 (all the rest after were fine)
2022-04-11T14:30:58.640661Z INFO ThreadId(59) packet_cmd{src_chain=ibc-1 src_port=transfer src_channel=channel-0 dst_chain=ibc-0}:relay{odata=PBE8O38QpJ ->Destination @1-5376; len=525}: assembled batch of 526 message(s)
2022-04-11T14:30:58.641176Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: sending 15 messages using account sequence AccountSequence(1508)
2022-04-11T14:30:58.642412Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: max fee, for use in tx simulation: Fee { amount: "4001stake", gas_limit: 4000000 }
2022-04-11T14:30:58.659263Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}:estimate_gas: tx simulation successful, gas amount used: 1138114
2022-04-11T14:30:58.659650Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: send_tx: using 1138114 gas, fee Fee { amount: "2277stake", gas_limit: 2276228 } id=ibc-0
2022-04-11T14:30:58.674143Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: broadcast_tx_sync: Response { code: Ok, data: Data([]), log: Log("[]"), hash: transaction::Hash(5BD46F5034BBCA706ABC2FEA88A472F68A833D8856D0D888A0BA3202EF9610EE) }
2022-04-11T14:30:58.674424Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: sending 15 messages using account sequence AccountSequence(1509)
2022-04-11T14:30:58.675625Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: max fee, for use in tx simulation: Fee { amount: "4001stake", gas_limit: 4000000 }
2022-04-11T14:30:58.694100Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}:estimate_gas: tx simulation successful, gas amount used: 1162682
2022-04-11T14:30:58.694639Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: send_tx: using 1162682 gas, fee Fee { amount: "2326stake", gas_limit: 2325364 } id=ibc-0
2022-04-11T14:30:58.708255Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: broadcast_tx_sync: Response { code: Ok, data: Data([]), log: Log("[]"), hash: transaction::Hash(2ADF3899564CB78F34927ACB8CBF6199CA18F237C3716AA599F7EE7D5608FD3C) }
2022-04-11T14:30:58.708523Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: sending 15 messages using account sequence AccountSequence(1510)
2022-04-11T14:30:58.710246Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: max fee, for use in tx simulation: Fee { amount: "4001stake", gas_limit: 4000000 }
2022-04-11T14:30:58.729467Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}:estimate_gas: tx simulation successful, gas amount used: 1162665
2022-04-11T14:30:58.729750Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: send_tx: using 1162665 gas, fee Fee { amount: "2326stake", gas_limit: 2325330 } id=ibc-0
2022-04-11T14:30:58.743948Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: broadcast_tx_sync: Response { code: Ok, data: Data([]), log: Log("[]"), hash: transaction::Hash(CB837A33A58ADE26B9318C9563E4CFF8E670E089DF3A701F2AB324AE6C2B3F28) }
2022-04-11T14:30:58.744173Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: sending 15 messages using account sequence AccountSequence(1511)
2022-04-11T14:30:58.745432Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: max fee, for use in tx simulation: Fee { amount: "4001stake", gas_limit: 4000000 }
2022-04-11T14:30:58.762630Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}:estimate_gas: tx simulation successful, gas amount used: 1162365
2022-04-11T14:30:58.762906Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: send_tx: using 1162365 gas, fee Fee { amount: "2325stake", gas_limit: 2324730 } id=ibc-0
2022-04-11T14:30:58.776655Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: broadcast_tx_sync: Response { code: Ok, data: Data([]), log: Log("[]"), hash: transaction::Hash(16A9D37AC2EFC5460F4C1A438F0E92F8A0616D4086DAEF968F9DFF549106774A) }
2022-04-11T14:30:58.776880Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: sending 15 messages using account sequence AccountSequence(1512)
2022-04-11T14:30:58.778153Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: max fee, for use in tx simulation: Fee { amount: "4001stake", gas_limit: 4000000 }
2022-04-11T14:30:58.786158Z WARN ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}:estimate_gas: failed to simulate tx, falling back on default gas because the error is potentially recoverable: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 1509, got 1512: incorrect account sequence", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "5481"} }
2022-04-11T14:30:58.786400Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: send_tx: using 4000000 gas, fee Fee { amount: "4001stake", gas_limit: 4000000 } id=ibc-0
2022-04-11T14:30:58.801198Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: broadcast_tx_sync: Response { code: Ok, data: Data([]), log: Log("[]"), hash: transaction::Hash(39C606B3A10D9F9175B2E2AC17108C4F42ABF97A7C3911AA41F95936180C9E4D) }
2022-04-11T14:30:58.801412Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: sending 15 messages using account sequence AccountSequence(1513)
2022-04-11T14:30:58.802668Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: max fee, for use in tx simulation: Fee { amount: "4001stake", gas_limit: 4000000 }
2022-04-11T14:30:58.894121Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}:estimate_gas: tx simulation successful, gas amount used: 1162745
2022-04-11T14:30:58.894369Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: send_tx: using 1162745 gas, fee Fee { amount: "2326stake", gas_limit: 2325490 } id=ibc-0
2022-04-11T14:30:58.907547Z DEBUG ThreadId(40) send_tx_check{id=PBE8O38QpJ}:send_tx{id=ibc-0}: broadcast_tx_sync: Response { code: Ok, data: Data([]), log: Log("[]"), hash: transaction::Hash(26C864956FC26924E8B2F2941EB0B7A7F9276676F4660B6BDEDB583A54D5D8CD) }
...
For Admin Use
- Not duplicate issue
- Appropriate labels applied
- Appropriate contributors tagged
- Contributor assigned/self-assigned
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 16 (10 by maintainers)
Great work SDK team!
We’ll be tracking integration in Hermes against this feature in https://github.com/informalsystems/hermes/issues/3763.
You guys will love this feature I think. It should make relayers drastically simpler IMO
This is extremely trivial and easy:
sorry for the late update. After testing more, experimenting with different changes in hermes, talking to @marbar3778, opened https://github.com/informalsystems/ibc-rs/issues/2249 with my understanding and a proposed workaround that seems to work for hermes. We also discussed with Marko if it would be possible to have a
simulateTxflag to ignore the account sequence check. Not sure how feasible, Marko mentioned that he will discuss with you/ SDK team. thanks for the help with this!