jormungandr: Leadership schedule may miss tip branch switch and create blocks in dimissied branch
I have been receiving comments regarding some issues stake pools are experiencing when following the creation of consecutive blocks from different stake pools.
Description of the problem
What is being observed
In the current implementation of the leadership schedule it is possible a node creates a block on the parent of the tip instead of the tip itself.
let’s say we have a 2 nodes, scheduled to respectively create a Block A and a Block B on 2 consecutive slots.
What we would expect is:
Parent Block <------ Block A <--------- Block B
However from the PoV of the node creating the block B it may be possible that we have the following:
Parent Block <--------- Block A
\
`------------------------------------- Block B
- the node creating block B sees the
Parent Blockas the current tip; - the node start creating block B
- the node create block B with
Parent Blockas its block ancestor - the node receives Block A whilst still creating block B and switch its tip to block A.
- block B was created still with Block A using
Parent Blockas parent
what are the consequences
For the blockchain in general, the consequence is that a fork has been created where it could have been avoided. For the stake pools involved, it means one of the 2 blocks won’t be created and they will miss on rewards.
Is this a problem? can we mitigate this?
Under discussion
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 19
- Comments: 19 (1 by maintainers)
Seeing the same in v8.6.0.
Three examples:
Take the last missed block as an example. My node considered
b416ba88ccda8c95145a200b6fb99f75d835f05201ccdf678ecdf3e4542b89bf(36.3451) as the parent block, but it did received the correct block from the next slot (36.3452), as I can see in the log:However, my node still took
b416ba88ccda8c95145a200b6fb99f75d835f05201ccdf678ecdf3e4542b89bfas the parent block. My case is slightly different from the issue in the way that the node is aware of the latest tip (because we can see it received block announcement), but it still chose to use previous block as parent. Is this considered as a bug?Similar problem here - announcement received about a new block with parent X, then two seconds later my own leader event was actioned and created a block with parent X (i.e. fork):
Jan 07 11:57:02.813 WARN received block announcement from network, from_node_id: 1a866537825726db248b5f596fd6e0cd181aaa636f1e389a, date: 24.28301, parent: 3de39c7d43478a3649b24d9bf221cfeb816fe3fd38b7e198d98928972a114695, hash: 992f626af4077cb231972317e262e7ca12ee57e908cc29cd9a6267cb5111d0cc, task: block
Jan 07 11:57:05.008 WARN Leader event started, event_remaining_time: 1s 997ms 674us 404ns, event_end: 2020-01-07T10:57:07+00:00, event_start: 2020-01-07T10:57:05+00:00, event_date: 24.28304, leader_id: 1, task: leadership
Jan 07 11:57:05.015 WARN processing block from leader event, date: 24.28304, parent: 3de39c7d43478a3649b24d9bf221cfeb816fe3fd38b7e198d98928972a114695, hash: 2b2025c798c4e323b8165699a8ec5d48bb6d7a25016da1f19927b98c9d806ad1, task: block
I observed similar issues, at least two times when I monitored my nodes. which seems to me a bug in jormungandr. Using the similar tree:
Despite I received at least 8 25383 announcement till 45sec, my node created a block /w 25382 as its parent instead of 25383.
Probably seeing the same. I have a block scheduled at date 40.21999:
From my logs:
The parent does exist in the chain. Epoch 40, slot 21898. Created at 07:23:33. So, I create my block (with chain length 123476 ) 3 minutes and 22 seconds later. My block is not accepted and there appears to be a block
53d41be708cc8a1eb7544a46cfb642877831a757936597bea04628df752cca41with the same chain length (123476) in slot 21909. This slot has the same parent as my node received at 07:26:55.The next block announcement my node gets is: also exists in the chain at slot 22020:
Scheduled slot 52.8624
More than 1 minute later:
This later block is accepted in slot 8655, 31 slots later than my scheduled slot!
I have the feeling this issue is increasing in frequency. When it started around halve of my blocks were vaporized this way. Currently, 75% of my blocks vaporize this way. And it is always taken by one of the big pools like the 1PCT cluster.
Looking at the competitive fork page on Pooltools.io it seems to intensify too: competitive blockheights: 162101, 162103, 162110, 162134, 162146, 162148, 162155, 162162, 162163, 162164 and 162172.
Yesterday, two of my competitive blockheights weren’t even listed on Pooltool. So, there is even more.
In conclusion: currently, I only make 25% of assigned slots, the other 75% is lost due to slot or block height competition. This remains a serious problem! Is this still a Proof of Stake protocol? I have the stake so my blocks should be accepted if created correctly (which the log shows). It shouldn’t be a battle of resources making it a kind of Proof of Work. If smaller pools keep losing these battles they will drop out and only a few large clusters of stake pools will remain. That’s not decentralisation.
Up till now there has been many reports but no feedback from devs. Why? Do they think this is irrelevant? This issue (1513) was opened by a dev 28 days ago. Nothing after that.
Well, this is a huge issue for pools with 2-6m stake, bc. if we miss our 1-3 blocks per epoch the ROI and performance metric is destroyedin no minute and the small amount of stakers who gave us there trust is gone. I havnt had any issue before but since 3Days no block was accepted by the network and I created 3 of them per Day.
There was no downtime issue nore a fork issue, I have a completely fine tuned setup with multiple nodes and I am always on top of the chain tip.
blockRecvCnt: 5089 lastBlockContentSize: 0 lastBlockDate: "45.1724" lastBlockFees: 0 lastBlockHash: a906453a9ba0740498f24a523c77d14168b606b6c4dd6b3f8b45fd1ec7db8afe lastBlockHeight: "137804" lastBlockSum: 0 lastBlockTime: "2020-01-27T20:11:05+00:00" lastBlockTx: 0 lastReceivedBlockTime: "2020-01-27T20:11:09+00:00" peerAvailableCnt: "96363" peerQuarantinedCnt: "31559" peerUnreachableCnt: "4448" state: Running txRecvCnt: 5262 uptime: 87299 version: jormungandr 0.8.7-364cd84Edit:
still having issues in 0.8.9, this epoch, 7 out of 10 blocks disappeared in the air, it all started this week and I cant find anything what could cause the issue its like I am connected to bad peers that do not forward the blocks, my blocks are definitely sent out
`
created_at_time: “2020-01-30T20:01:23.829574394+00:00” enclave_leader_id: 1 finished_at_time: “2020-01-30T23:15:39.006257098+00:00” scheduled_at_date: “48.7261” scheduled_at_time: “2020-01-30T23:15:39+00:00” status: Block: block: 88f7244ce1c27954dbc6ee4affa94e85a8d687b92d1d86d75bcc79db31ca1ccd chain_length: 147632 wake_at_time: “2020-01-30T23:15:39.001649377+00:00”
created_at_time: “2020-01-30T20:01:23.829590383+00:00” enclave_leader_id: 1 finished_at_time: “2020-01-30T23:33:51.003470924+00:00” scheduled_at_date: “48.7807” scheduled_at_time: “2020-01-30T23:33:51+00:00” status: Block: block: cc83eecf11a31cba92d27caeb775834d15db288f14637b9e32157ceca5417eb2 chain_length: 147679
created_at_time: “2020-01-30T20:01:23.829597016+00:00” enclave_leader_id: 1 finished_at_time: “2020-01-31T07:41:53.210604750+00:00” scheduled_at_date: “48.22448” scheduled_at_time: “2020-01-31T07:41:53+00:00” status: Block: block: c47e13afa29a0b4b693a7fc0d414954f88d6cb40982bf56acb397c906dfe3075 chain_length: 148838 wake_at_time: “2020-01-31T07:41:53.208791615+00:00”
created_at_time: “2020-01-30T20:01:23.829599043+00:00” enclave_leader_id: 1 finished_at_time: “2020-01-31T14:19:45.195568536+00:00” scheduled_at_date: “48.34384” scheduled_at_time: “2020-01-31T14:19:45+00:00” status: Block: block: d4ec7c943418daa14fb20792953c7fd737dfb90b498c236f6835753ad9a4858c chain_length: 149760
created_at_time: “2020-01-30T20:01:23.829600603+00:00” enclave_leader_id: 1 finished_at_time: “2020-01-31T15:20:27.623061215+00:00” scheduled_at_date: “48.36205” scheduled_at_time: “2020-01-31T15:20:27+00:00” status: Block: block: f96479f5fc97f727c4628b9bf2534856c1ea62406adfdc5f064b1f43893f3064 chain_length: 149908
created_at_time: “2020-01-31T16:37:17.497276096+00:00” enclave_leader_id: 1 finished_at_time: “2020-01-31T17:09:49.019645563+00:00” scheduled_at_date: “48.39486” scheduled_at_time: “2020-01-31T17:09:49+00:00” status: Block: block: e4eb24c2a06e25a9565070ab549e2933dcced856555ef878b54ac63cc1f6e548 chain_length: 150170 wake_at_time: “2020-01-31T17:09:49.001249306+00:00”
created_at_time: “2020-01-31T16:37:17.497342386+00:00” enclave_leader_id: 1 finished_at_time: “2020-01-31T17:17:59.002552026+00:00” scheduled_at_date: “48.39731” scheduled_at_time: “2020-01-31T17:17:59+00:00” status: Block: block: 68d8a0a4cb58ca459d0dc1253d170a725055b57ca5c26467101d7ebe99c54b94 chain_length: 150187 wake_at_time: “2020-01-31T17:17:59.001419411+00:00”`
`