tendermint: Consensus stuck for 50 minute trying to finish a block

Tendermint version (use tendermint version or git rev-parse --verify HEAD if installed from source): 0.26.4

ABCI app (name for built-in, URL for self-written if it’s publicly available): Loom SDK

Environment:

• OS (e.g. from /etc/os-release): Ubuntu 16.04
• Install tools:
• Others:

What happened: Our 4 node staging tendermint cluster in a single geographic region will get stuck trying to finish a round, something for 50 minutes straight, it accepts tx in the mempool but never completes a block. Any ideas how to debug this more or allow the cluster to just cancel blocks when it can’t come to consensus.

See the attached logs from all 4 nodes. At the end of the 50 minutes it finally commits and 5000 txs stuck in the mempool clear

What you expected to happen:

Would expect the round to close within 5-10seconds, if they can’t get agreement from one node

Have you tried the latest version: yes/no no, the version is only a few days old

How to reproduce it (as minimally and precisely as possible): no consistent way, it seems to do with having dozens of users connected with a few transactions a second

Logs (paste a small part showing an error (< 10 lines) or link a pastebin, gist, etc. containing more of the log file): Archive.zip Attached log from 4 nodes

Config (you can paste only the changes you’ve made):

replay = false
broadcast = true
create_empty_blocks = false

node command runtime flags:

/dump_consensus_state output for consensus bugs This is an hour after the event, the cluster came back to life with no outside intervention. So I’m not sure if this is super relevant

consensus_dump.log

Anything else we need to know:

A few hundred connected clients to the cluster but still low transaction volume, maybe a few transactions a second