jormungandr: The Node stops creating blocks (BFT and Genesis)

Describe the bug After 15 mins (tens of transactions), the Node stops creating blocks (BFT and Genesis)

To Reproduce Even simpler:

  1. start node
  2. create 1 transaction and wait 15 mins while blocks are created
  3. after 15 mins, blocks are no longer created

Steps to reproduce the behavior:

  1. generate files using bootstrap script: bash bootstrap -p 8607 -b | tee initialValues.txt
  2. start node: jormungandr --genesis-block ./block-0.bin --config ./config.yaml --secret ./pool-secret1.yaml
  3. run attached script (one-transaction-per-slot.sh) - after some time, the node stops creating new blocks; In this state, no new transactions are accepted and the script will fail;

Expected behavior Especially because we are in BFT mode, as long as the node is ON, the leader should create blocks.

Additional context Machine used: Win10 I ran the script in Genesis mode too and the behavior was the same (there still was stake in the system - on Faucet account)

As @rinor observed, if you don’t make any transaction, the blocks will continue to be created even after 15 mins.

Attached:

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 1
  • Comments: 28 (20 by maintainers)

Commits related to this issue

Most upvoted comments

Everyone gives @rinor a big virtual hug. The link in his last comment lead me to find the real issue. Fix is in #703 .

Some more info that may help. Based on my tests the issue is caused by MemPool/Logs garbage collector. https://github.com/input-output-hk/jormungandr/blob/d239f842b15a322e5186372ba847e9268f15a41a/jormungandr/src/main.rs#L128-L129 Block production stops based on that timing. In local test I change that value to smth like 3600 secods (1 hour) which gives a decent time to test before crashing.

I’m trying to reproduce it with automation test. Will post results soon

Any news on that bug ?

Ok. I got the latest master again (there were some more commits) and now the issue is not reproducible neither on my machine.

Git revision: 17634fc83e578f29ab25ac78c65bbfeb60b67c05

It seems I was too eager to see this working and I needed to wait a little more for all the fixes 😃. Good job guys!

if you want to see more logs, try --log-level=trace we have reduced the logging verbosity of debug to not display non dev thing

@rinor - I just saw #705 and the commit for it. I am giving it 1 more try after this commit/pr and will update here.

@vincenthz - this bug is still reproducible. The node stops creating blocks after 15 mins so can you please reopen it as it already has a lot of logs from different users inside it?