tendermint-rs: light-node: run against a Tendermint image fails with clock drift

After spawning a light node in a Docker container with:

$ docker run \
    -p 26656:26656 \
    -p 26657:26657 \
    -p 26660:26660 \
    tendermint/tendermint:v0.33.6

If we fetch the latest block height, hash and time via RPC, we get:

$ curl -X GET "http://localhost:26657/status" -H  "accept: application/json" | jq '.result.sync_info | {latest_block_hash,latest_block_height,latest_block_time}'

{
  "latest_block_hash": "CCFF1A6907DB9DC08E7C58D9C590088D1522069FFDFF7ABD3E6D62DCD083DBC4",
  "latest_block_height": "2469",
  "latest_block_time": "2020-08-11T10:30:13.352595649Z"
}

Here we see that the block was committed at 10:30 UTC, whereas both on my machine and within the Docker container, the time is reported as 10:20 UTC, meaning that the block has been committed 10 minutes “in the future”.

So if we initialize the light-node with the block above, by running:

$ cargo run -- initialize 2469 CCFF1A6907DB9DC08E7C58D9C590088D1522069FFDFF7ABD3E6D62DCD083DBC4

and then start the light node with:

$ cargo run -- start

we get:

started RPC server: 127.0.0.1:8888
error: sync failed: no witness left: no initial trusted state
error: sync failed: no witness left: no initial trusted state

etc.

After sprinkling some dbg! statements in the supervisor, we indeed see that verification fails because the headers pulled from the node are “from the future”:

[light-client/src/supervisor.rs:221] &verdict = Err(
    Error(
        Context {
            kind: InvalidLightBlock(
                HeaderFromTheFuture {
                    header_time: Time(
                        2020-08-11T10:32:44.352595649Z,
                    ),
                    now: Time(
                        2020-08-11T10:22:58.589649Z,
                    ),
                },
            ),

Again, a 10 minutes clock drift.

What is weird here, is that the reported time on both my machine and within the Docker container actually do match, which may point to a configuration issue in the Docker image.

The very same behavior has been observed by @brapse on his machine.

---

If we do the same operations above against a full node started with tendermint node --proxyapp=kvstore, things work out just fine and the light node continuously syncs to the latest block without issues:

started RPC server: 127.0.0.1:8888
synced to block: 5006
synced to block: 5007
synced to block: 5008

etc.