chia-blockchain: [Bug] uncaught KeyError exception in block_record

What happened?

I am not sure if this would be possible to reproduce, but I was running chia on raspberry (4GB RAM) for a like half a year, when in October the entire FS crashed, and I had to reinstall and unfortunately sync from scratch. After like 2 months of checking and regular restarts of chia daemon it got into near sync state of blockchain (wallet was full sync, but chain last day missing, this was somewhere between 21-22nd December). But when I’ve checked it after Christmas, the log is full of this:

2021-12-27T13:59:13.294 full_node full_node_server        : WARNING  Banning 99.152.32.71 for 10 seconds
2021-12-27T13:59:13.296 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '99.152.32.71', 'port': 8444}
2021-12-27T13:59:13.298 full_node full_node_server        : ERROR    Exception b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2', exception Stack: Traceback (most recent call last):
  File "chia/server/server.py", line 437, in start_client
  File "chia/server/server.py", line 335, in connection_added
  File "chia/full_node/full_node.py", line 619, in on_connect
  File "chia/full_node/full_node.py", line 585, in synced
  File "chia/consensus/blockchain.py", line 157, in get_peak
  File "chia/consensus/blockchain.py", line 674, in height_to_block_record
  File "chia/consensus/blockchain.py", line 670, in block_record
KeyError: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2'

2021-12-27T13:59:13.301 full_node asyncio                 : ERROR    Task exception was never retrieved
future: <Task finished name='Task-291378' coro=<RpcServer._state_changed() done, defined at chia/rpc/rpc_server.py:48> exception=KeyError(b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2')>
Traceback (most recent call last):
  File "chia/rpc/rpc_server.py", line 51, in _state_changed
  File "chia/rpc/full_node_rpc_api.py", line 61, in _state_changed
  File "chia/rpc/full_node_rpc_api.py", line 102, in get_blockchain_state
  File "chia/consensus/blockchain.py", line 157, in get_peak
  File "chia/consensus/blockchain.py", line 674, in height_to_block_record
  File "chia/consensus/blockchain.py", line 670, in block_record
KeyError: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2'
2021-12-27T13:59:14.603 full_node full_node_server        : INFO     Cannot connect to host 112.25.210.130:8444 ssl:<ssl.SSLContext object at 0xffff377097c0> [Connect call failed ('112.25.210.130', 8444)]

running eg. chia show -s results in similar message:

Exception from 'show' Traceback (most recent call last):
  File "chia/cmds/show.py", line 41, in show_async
  File "chia/rpc/full_node_rpc_client.py", line 27, in get_blockchain_state
  File "chia/rpc/rpc_client.py", line 49, in fetch
ValueError: {'error': "b'\\xd7\\x80\\xd2,z\\x87\\xc9\\xe0\\x1d\\x98\\xb4\\x9a\\t\\x10\\xf6p\\x1c;\\x95\\x01WA1k?\\xda\\x04.]{\\x81\\xd2'", 'success': False}

but chia wallet show is ok, so seems only main daemon is somehow stuck:

chia wallet show
Wallet height: 1338206
Sync status: Synced
...

Version

1.2.11

What platform are you using?

Pi4

What ui mode are you using?

CLI

Relevant log output

No response

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 5
Comments: 88 (28 by maintainers)

Most upvoted comments

Same problem, same header hash. Looking at the db I found this: … EDIT: With that change in the database the sync continued to work, the node is fully synced now

I’ve just checked my node, and you are right, the block 0 is actually set as peak:

sqlite> select header_hash, prev_hash, height, is_peak, is_block from block_records where is_peak == 1 and is_block == 1;

header_hash|prev_hash|height|is_peak|is_block
d780d22c7a87c9e01d98b49a0910f6701c3b95015741316b3fda042e5d7b81d2|ccd5bb71183532bff220ba46c268991a3ff07eb358e8255a65c30a2dce0e5fbb|0|1|1

sqlite> select header_hash, prev_hash, height, is_peak, is_block from block_records where height == (select max(height) from block_records);

header_hash|prev_hash|height|is_peak|is_block
3c61c9e9730722c45f738cd5cb3e3203428683679e966cc52dbfc78a623e69d5|356586bc3f5fa2e3671a1498e8d875e7892a59dccb40cd5cfdd369329132d561|1313895|0|1

So I’ve tried swapping peak on block 0 vs last block as suggested:

sqlite> update block_records set is_peak = 1 where height == (select max(height) from block_records);
sqlite> update block_records set is_peak = 0 where height == 0;
sqlite> select header_hash, prev_hash, height, is_peak, is_block from block_records where is_peak == 1 and is_block == 1;

header_hash|prev_hash|height|is_peak|is_block
3c61c9e9730722c45f738cd5cb3e3203428683679e966cc52dbfc78a623e69d5|356586bc3f5fa2e3671a1498e8d875e7892a59dccb40cd5cfdd369329132d561|1313895|1|1

and yes, it solves the problem, chia is again syncing. Question still remains how this could happen? Some remote trigger? ( / attacker?) Client bug? Protocol bug?

+10

durdin85 on Dec 29, 2021

We will shortly release 1.3 beta which fixes this issue.

mariano54 on Feb 10, 2022

Please see my message above. The current fix is:

Delete full node blockchain file
Install and run 1.3 beta and wait for resync

mariano54 on Feb 22, 2022