reth: TransactionLookup stage hanging

Describe the bug

Database sync is stalled at progress=none on TransactionLookup. ForkChoice messages are stopped and the validator reports the EL is offline (reth is paired with my backup beacon node).

Steps to reproduce

This node performed sync to stage 4 on the beta1 and hung. I upgraded to beta3, the sync started progressing again, with ForkChoice messages, until it hung here on TransactionLookup. The message “local database is uninitialized” is there when I run reth db version smaller_truncated_reth_upload.log

Running: Fedora Server 38 Lighthouse v5.1.0 reth 0.2.0-beta.3 mev-boost 1.7

Node logs

I am attaching my log

Platform(s)

Linux (x86)

What version/commit are you on?

0.2.0-beta.3

What database version are you on?

Current database version: 2 Local database is uninitialized

What type of node are you running?

Full via --full flag

What prune config do you use, if any?

No response

If you’ve built Reth from source, provide the full command you used

No response

Code of Conduct

I agree to follow the Code of Conduct

About this issue

Original URL
State: closed
Created 3 months ago
Reactions: 1
Comments: 18 (3 by maintainers)

Most upvoted comments

I have the same problem, I also updated from v0.2.0-beta.1 to Beta.2 and Beta.3.

It takes just over an hour and does not increase progress, it show “stage_progress=None”

execution-1  | 2024-03-23T12:54:20.634938Z  INFO Stage committed progress pipeline_stages=8/12 stage=MerkleExecute checkpoint=0 target=19420672 stage_progress=99.97% stage_eta=1s
execution-1  | 2024-03-23T12:54:20.649068Z  INFO Preparing stage pipeline_stages=8/12 stage=MerkleExecute checkpoint=0 target=19420672
execution-1  | 2024-03-23T12:54:20.649148Z  INFO Executing stage pipeline_stages=8/12 stage=MerkleExecute checkpoint=0 target=19420672 stage_eta=1s
execution-1  | 2024-03-23T12:54:21.886772Z  INFO Status connected_peers=100 freelist=16 stage=MerkleExecute checkpoint=0 target=19420672 stage_progress=99.97% stage_eta=unknown
execution-1  | 2024-03-23T12:54:22.141560Z  INFO Stage finished executing pipeline_stages=8/12 stage=MerkleExecute checkpoint=19420672 target=19420672 stage_progress=100.00% stage_eta=unknown
execution-1  | 2024-03-23T12:54:22.147073Z  INFO Preparing stage pipeline_stages=9/12 stage=TransactionLookup checkpoint=0 target=19420672
execution-1  | 2024-03-23T12:54:22.147098Z  INFO Executing stage pipeline_stages=9/12 stage=TransactionLookup checkpoint=0 target=19420672
execution-1  | 2024-03-23T12:54:46.894244Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T12:55:11.886978Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T12:55:36.887023Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T12:56:01.887459Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T12:56:26.890707Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T12:56:51.893334Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T12:57:16.893971Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T12:57:41.894015Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T12:58:06.897571Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T12:58:31.894197Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T12:58:56.894396Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T12:59:21.894734Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T12:59:46.893849Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T13:00:11.894551Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T13:00:36.893888Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T13:01:01.895248Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T13:01:26.893887Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T13:01:51.896120Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T13:02:16.893682Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T13:02:41.894532Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None

.
.
.
.
.
execution-1  | 2024-03-23T14:13:07.866548Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T14:13:32.867939Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T14:13:57.870988Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None
execution-1  | 2024-03-23T14:14:22.871637Z  INFO Status connected_peers=100 freelist=20 stage=TransactionLookup checkpoint=0 target=19420672 stage_progress=None

Thanks

victorelec14 on Mar 23, 2024

Confirmed that with the v0.2.0-beta.4 , the synchronization completes correctly. There was no need to update to beta5.

Thanks!

victorelec14 on Apr 10, 2024

~hung again on TransactionLookup, checkpoint = 0 :(~

edit: Well, it jumped from checkpoint=0 to 100% complete and finished the pipeline!

Scott-Sanderson on Apr 2, 2024

hey guys, wanted to report that for my beta 4 syncing reth node, the TransactionLookup stage stopped hanging and I am past this stage now after I realize I haven’t upgraded my lighthouse to a 4844 compliant version (pretty dumb i know…)

note that, my backfill sync target block is 19426523, which is a bit before 4844 goes live.

so what i would suggest to people having the same problem is to check if your consensus client is the latest version, if not upgrade it and restart reth to see if it syncs through the tx lookup stage now.

now the question is, does during TransactionLookup stage execution client must need to communicate consensus client in this stage, and if the consensus client is stalled due to it being a pre-4844 version it cause reth sync also hangs?

jonathanykh on Mar 31, 2024

I too have this issue. The lovely folk in Telegram asked to post trace logs from net::session, so here they are. Reth Version: 0.2.0-beta.4 Lighthouse version: v5.1.2-5ce1619

reth.log

RomulusERC on Mar 27, 2024

Hey, I have the same issue at 0.2.0-beta 3, it seems that the stage is stuck and nothing gets done after calculating transaction hashes is completed (I do not know how many transactions in total in Ethereum mainnet, therefore actually not sure if all transaction hashes have been calculated).

Here are my logs captured at the end of calculating transaction hashes:

2024-03-27T03:46:28.448535Z DEBUG sync::stages::transaction_lookup: Calculating transaction hashes tx_range=1765037468..1770037571
2024-03-27T03:46:39.257282Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0xcfbb1b82c46ea98a6fd96b33bdf1038ba72b8e891fc2a2ce818110805925130db13471b0f3207a8ec36a4b853bec98ac77eac0baa955f030934fb2a45e76f853
2024-03-27T03:46:43.821987Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0xdf7ac5ee488b74924bdb861350f174764014ded895332e61d7d8f7672d5b8421e73549bb9b71f0ec2509d179eceac9f5673cb3f69bc5ae2008da3633b17b22bd
2024-03-27T03:46:45.351189Z DEBUG sync::stages::transaction_lookup: Calculating transaction hashes tx_range=1770037571..1775037633
2024-03-27T03:46:48.324056Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x25b663f4199eee2490ff82110fce5fc2051d3808a18bb79e7ee410bdddca18e67bfb5710ddebb8469a36fa6867d374c0c0f89aa40ba2ff311a3e87ee9fe6b827
2024-03-27T03:46:50.755818Z  INFO reth::cli: Status connected_peers=30 freelist=6202 stage=TransactionLookup checkpoint=0 target=19426523 stage_progress=None
2024-03-27T03:46:54.325325Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0xc77217ce8abd85ae54e778835cf68cd0289bbeee30cdb32d516be5fd7571adbc72503eec06fc7e555e0929ddf0d0d06772bf9ff0abc326cafdc1e6a0945e900c
2024-03-27T03:46:54.813906Z DEBUG sync::stages::transaction_lookup: Calculating transaction hashes tx_range=1775037633..1780037648
2024-03-27T03:47:04.108371Z DEBUG sync::stages::transaction_lookup: Calculating transaction hashes tx_range=1780037648..1785037666
2024-03-27T03:47:09.149491Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x1cf7db662897dcd2af946fdd47e5b6d8273846e6a6bef4f1f3ec1f2b38c77c6a2382fb9409e43681c15cc043298d9476f2faa14c1adb805c15559e616ff4c361
2024-03-27T03:47:15.756289Z  INFO reth::cli: Status connected_peers=29 freelist=6202 stage=TransactionLookup checkpoint=0 target=19426523 stage_progress=None
2024-03-27T03:47:16.839189Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0xfea359e2843059d4f99bb09da45400a6b3f145e80218a1769b86c31362269795cccb1981e87733e45ce4300af01ec524ba28344df293823a262f6add3cfa7af7
2024-03-27T03:47:22.414535Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x02f7dc8b8b5b49fd56e6100eeb10391d58e405639e38194d199b77a5f70b7bf5c3ddfd76af6dfa4dc7dfce3fa21d96f631199ee70795aa3113d5c349de68335d
2024-03-27T03:47:31.688840Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0xdf7ac5ee488b74924bdb861350f174764014ded895332e61d7d8f7672d5b8421e73549bb9b71f0ec2509d179eceac9f5673cb3f69bc5ae2008da3633b17b22bd
2024-03-27T03:47:40.762456Z  INFO reth::cli: Status connected_peers=28 freelist=6202 stage=TransactionLookup checkpoint=0 target=19426523 stage_progress=None
2024-03-27T03:47:58.556377Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x1cf7db662897dcd2af946fdd47e5b6d8273846e6a6bef4f1f3ec1f2b38c77c6a2382fb9409e43681c15cc043298d9476f2faa14c1adb805c15559e616ff4c361
2024-03-27T03:48:02.424149Z DEBUG igd_next::aio::tokio: sending broadcast request to: 239.255.255.250:1900 on interface: Ok(0.0.0.0:35081)    
2024-03-27T03:48:05.762904Z  INFO reth::cli: Status connected_peers=29 freelist=6202 stage=TransactionLookup checkpoint=0 target=19426523 stage_progress=None
2024-03-27T03:48:12.426832Z DEBUG net::nat: Failed to resolve external IP via UPnP: failed to find gateway err=IO error: search timed out
2024-03-27T03:48:30.329529Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0xdf7ac5ee488b74924bdb861350f174764014ded895332e61d7d8f7672d5b8421e73549bb9b71f0ec2509d179eceac9f5673cb3f69bc5ae2008da3633b17b22bd
2024-03-27T03:48:30.762964Z  INFO reth::cli: Status connected_peers=29 freelist=6202 stage=TransactionLookup checkpoint=0 target=19426523 stage_progress=None
2024-03-27T03:48:44.389554Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x1cf7db662897dcd2af946fdd47e5b6d8273846e6a6bef4f1f3ec1f2b38c77c6a2382fb9409e43681c15cc043298d9476f2faa14c1adb805c15559e616ff4c361
2024-03-27T03:48:46.092315Z DEBUG net::session: eth stream completed remote_peer_id=0x48f462dab313723e05fee35df8172a7d369c42a42d1113a31f880c478884e8ce3f3173818868cdd9398c6e5bbf2c1d0ccb969a6946275905372c18caf6db0599
2024-03-27T03:48:55.762794Z  INFO reth::cli: Status connected_peers=29 freelist=6202 stage=TransactionLookup checkpoint=0 target=19426523 stage_progress=None
2024-03-27T03:49:13.058189Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x91286039f8810f34cabacad863fdd978b8e71adfc4de98a255246b05ef4584495d39541bc83d62296bd355970145596b38927657a9e3b1ce2b1d17e0b8e6ccd9
2024-03-27T03:49:13.542091Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0xdf7ac5ee488b74924bdb861350f174764014ded895332e61d7d8f7672d5b8421e73549bb9b71f0ec2509d179eceac9f5673cb3f69bc5ae2008da3633b17b22bd
2024-03-27T03:49:16.096160Z DEBUG net::session: eth stream completed remote_peer_id=0x48f462dab313723e05fee35df8172a7d369c42a42d1113a31f880c478884e8ce3f3173818868cdd9398c6e5bbf2c1d0ccb969a6946275905372c18caf6db0599
2024-03-27T03:49:20.762776Z  INFO reth::cli: Status connected_peers=29 freelist=6202 stage=TransactionLookup checkpoint=0 target=19426523 stage_progress=None
2024-03-27T03:49:23.156096Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x1cf7db662897dcd2af946fdd47e5b6d8273846e6a6bef4f1f3ec1f2b38c77c6a2382fb9409e43681c15cc043298d9476f2faa14c1adb805c15559e616ff4c361
2024-03-27T03:49:39.511660Z DEBUG net::session: eth stream completed remote_peer_id=0x812fc1180f5532fb0550ef307a1a902dd24b3674f00bd8d06bfa2ec8c4348da220ab3b44f6432ffa7b08d16888672cdabce70a835df67b5bf04715ea3c2b4089
2024-03-27T03:49:45.048798Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0xfea359e2843059d4f99bb09da45400a6b3f145e80218a1769b86c31362269795cccb1981e87733e45ce4300af01ec524ba28344df293823a262f6add3cfa7af7
2024-03-27T03:49:45.763656Z  INFO reth::cli: Status connected_peers=29 freelist=6202 stage=TransactionLookup checkpoint=0 target=19426523 stage_progress=None

Regarding reth db version It shows that:

Current database version: 2
Local database version: 2

I also tried reth stage drop tx-lookup and reth stage drop merkle, which it syncs the merkle stage again but still stuck at TransactionsLookup.

I am now upgrading to 0.2.0-beta 4 to see if it still happens, will provide update later.

update: the issue is still here despite upgrading to beta 4

jonathanykh on Mar 31, 2024