tendermint: ABCI: RequestBeginBlock.Header.Time is sometimes far in the future

Tendermint version: 0.31.5 (d2eab536)
ABCI app: self-written, not currently publicly available
OS: Ubuntu 16.04.5 LTS x86_64

What happened: The block header time starts drifting into the future. The below graph shows now - BlockHeaderTime, so if the value is negative, Block.Header.Time is ahead of real time.

timediff

What was expected to happen: The block header time should match real time, to within a couple of seconds.

Tried the latest version: no

How to reproduce it: I have yet to create a minimal ABCI app that reproduces this problem.

Logs: At the default log levels (main:info,state:info,*:error) there are no useful errors. I have changed the config to *:debug and am waiting for the next occurrence of time drift.

Config: Default config, as generated by tendermint testnet

node command runtime flags: Just tendermint node

Our tendermint cluster consists of 3 nodes, in GB, SG and US. Transactions are submitted at a rate of under 10 per second. If under heavy load, it would be reasonable to expect the block header time to lag real time (ie drift into the past), but this issue is drift in the opposite direction.

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 19 (14 by maintainers)

Most upvoted comments

Thanks for the details. Any chance you can try with a smaller setting of the time_iota_ms param? Say 1ms instead of 1000ms ? This setting ensures that precommits are at least time_iota_ms ahead of the last block time, which could cause the time to move into the future incorrectly if the blocks are fast. 1s blocks seems like it should be slow enough that this isn’t a big issue, but it could also be right on the cusp.

ebuchman on Jun 19, 2019

@aleksraiden Have you tried adding time_iota_ms=1 as @ebuchman suggests?

Yes, added. After 2day of running all OK.

aleksraiden on Jun 24, 2019