jormungandr: The Node stops creating blocks (BFT and Genesis)
Describe the bug After 15 mins (tens of transactions), the Node stops creating blocks (BFT and Genesis)
To Reproduce Even simpler:
- start node
- create 1 transaction and wait 15 mins while blocks are created
- after 15 mins, blocks are no longer created
Steps to reproduce the behavior:
- generate files using bootstrap script:
bash bootstrap -p 8607 -b | tee initialValues.txt - start node:
jormungandr --genesis-block ./block-0.bin --config ./config.yaml --secret ./pool-secret1.yaml - run attached script (
one-transaction-per-slot.sh) - after some time, the node stops creating new blocks; In this state, no new transactions are accepted and the script will fail;
Expected behavior Especially because we are in BFT mode, as long as the node is ON, the leader should create blocks.
Additional context Machine used: Win10 I ran the script in Genesis mode too and the behavior was the same (there still was stake in the system - on Faucet account)
As @rinor observed, if you don’t make any transaction, the blocks will continue to be created even after 15 mins.
Attached:
-
print screen -

-
node logs (logger: verbosity: 4) - nodeLogs.txt
-
storage file - blocks.sqlite.txt
-
one-transaction-per-slot.sh script - one-transaction-per-slot.sh.txt
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 1
- Comments: 28 (20 by maintainers)
Everyone gives @rinor a big virtual hug. The link in his last comment lead me to find the real issue. Fix is in #703 .
Some more info that may help. Based on my tests the issue is caused by MemPool/Logs garbage collector. https://github.com/input-output-hk/jormungandr/blob/d239f842b15a322e5186372ba847e9268f15a41a/jormungandr/src/main.rs#L128-L129 Block production stops based on that timing. In local test I change that value to smth like 3600 secods (1 hour) which gives a decent time to test before crashing.
I’m trying to reproduce it with automation test. Will post results soon
Any news on that bug ?
Ok. I got the latest master again (there were some more commits) and now the issue is not reproducible neither on my machine.
Git revision: 17634fc83e578f29ab25ac78c65bbfeb60b67c05
It seems I was too eager to see this working and I needed to wait a little more for all the fixes 😃. Good job guys!
if you want to see more logs, try
--log-level=tracewe have reduced the logging verbosity of debug to not display non dev thing@rinor - I just saw #705 and the commit for it. I am giving it 1 more try after this commit/pr and will update here.
@vincenthz - this bug is still reproducible. The node stops creating blocks after 15 mins so can you please reopen it as it already has a lot of logs from different users inside it?