reth: Database Commit error code: 998

Describe the bug

Ran into an issue syncing reth + lighthouse on goerli, each being cloned from latest (and stable on lighthouse), on Windows.

Steps to reproduce

1. lighthouse bn --network goerli --checkpoint-sync-url https://goerli.checkpoint-sync.ethpandaops.io --execution-endpoint http://localhost:8551 --execution-jwt .\secrets\jwt.hex
2. reth node --chain goerli --authrpc.jwtsecret .\secrets\jwt.hex --authrpc.addr 127.0.0.1 --authrpc.port 8551

after some time ran into issues, *note, I left my node running last night and when I woke up terminal seemed to have crashed, had used same commands but could’ve been some kind of corrupted db

Node logs

2023-07-15T15:24:31.440934Z  INFO reth::node::events: Executing stage pipeline_stages=2/13 stage=TotalDifficulty from=9347975 checkpoint=100.0% eta=unknown
2023-07-15T15:24:31.441091Z  INFO reth::node::events: Stage finished executing pipeline_stages=2/13 stage=TotalDifficulty block=9347975 checkpoint=100.0% eta=unknown
2023-07-15T15:24:31.441190Z  INFO reth::node::events: Executing stage pipeline_stages=3/13 stage=Bodies from=3450000 checkpoint=36.9% eta=unknown
2023-07-15T15:24:31.441233Z  INFO downloaders::bodies: Downloading bodies 3450001..=9347975 distance=5897974
2023-07-15T15:24:33.090370Z  INFO net: Session established remote_addr=13.42.4.128:30303 client_version=Geth/v1.11.6-stable/linux-amd64/go1.20.3 peer_id=0x592916e9808f0f558a93518fc6d18e97f83bc79b95a4075695b4a2d78189219f6484acdefcb20cdfa04db6dbfcb513b928e18e9739afe9f738908bcf2190ee11 total_active=1
2023-07-15T15:24:34.264201Z  INFO net: Session established remote_addr=65.21.196.48:30305 client_version=Geth/v1.11.6-stable-ea9e62ca/linux-amd64/go1.20.3 peer_id=0x4abf75fb612753b1e4e1b1ffec6dc9af0b8ffda00758e1c2ddf7e79830eaeb0eae933ed89f067f7671ce8cafbfd2d18623b7b727c2d08af513c64ddf4e7bf798 total_active=2
2023-07-15T15:24:34.970126Z  INFO net: Session established remote_addr=51.15.142.158:30303 client_version=besu/v23.4.1/linux-x86_64/openjdk-java-17 peer_id=0xb860febca5939733d15b699e61040da398b5f907671b8339eaac325cbe375cf9103fa9ee8d6bf385e48a3e70f1e99f55d5a82807438e179dcebc05ef715e756f total_active=2
2023-07-15T15:24:35.567492Z  INFO net: Session established remote_addr=54.180.2.59:30303 client_version=Geth/v1.11.6-stable/linux-amd64/go1.20.3 peer_id=0xfc35f8a83512e25b01c76872061689cf6fa86bc3c8d564b1422721d9c3cb2c8de4bb9fc691879d9dae650931e97b057b1f5dce505ec3849d056a5027d27eb33d total_active=2
2023-07-15T15:24:36.720003Z  INFO net: Session established remote_addr=35.207.134.193:30303 client_version=Geth/v1.10.23-stable-d901d853/linux-amd64/go1.17.12 peer_id=0x2beadd3fa07e5db27c8b8a23136bc46ee2e0b5e770b5edcc53635bd55736cdf8b540581915fe51259d8bb11855aa100c966b71aad5b8f5e7a12bcea2b768b1cd total_active=3
2023-07-15T15:25:01.437888Z  INFO reth::cli: Status connected_peers=2 stage=Bodies checkpoint=36.9% eta=unknown
2023-07-15T15:25:16.389011Z  INFO reth::node::events: Stage committed progress pipeline_stages=3/13 stage=Bodies block=3460000 checkpoint=37.0% eta=unknown
Error: An interface error occurred.

Caused by:
    Database commit error code: 998

Location:
    bin\reth\src\node\mod.rs:439:9

Error from lighthouse was

Jul 15 15:25:46.498 WARN Execution engine call failed            error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:25:46.500 ERRO Error during execution engine upcheck   error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:25:46.501 WARN Execution engine call failed            error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:25:46.503 ERRO Unable to get transition config         error: Api { error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out) }, service: exec
Jul 15 15:25:46.504 WARN Error whilst processing payload status  error: Api { error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out) }, service: exec
Jul 15 15:25:46.505 ERRO Failed to check transition config       error: EngineError(Api { error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out) }), service: exec
Jul 15 15:25:46.506 WARN Failed to update head on execution engines, error: ExecutionForkChoiceUpdateFailed(EngineError(Api { error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out) }))
Jul 15 15:25:47.503 ERRO Error during execution engine upcheck   error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:25:47.504 ERRO Error during execution engine upcheck   error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:25:47.506 ERRO Error during execution engine upcheck   error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:25:54.007 INFO Syncing                                 est_time: --, distance: 77 slots (15 mins), peers: 3, service: slot_notifier
Jul 15 15:25:54.010 WARN Syncing deposit contract block cache    est_blocks_remaining: initializing deposits, service: slot_notifier
Jul 15 15:25:55.454 INFO UPnP not available                      error: IO error: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (os error 10060), service: UPnP
Jul 15 15:25:59.515 ERRO Error during execution engine upcheck   error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:26:06.006 INFO Syncing                                 est_time: --, distance: 78 slots (15 mins), peers: 9, service: slot_notifier
Jul 15 15:26:06.008 WARN Syncing deposit contract block cache    est_blocks_remaining: initializing deposits, service: slot_notifier
Jul 15 15:26:10.438 WARN Error connecting to eth1 node endpoint  endpoint: http://localhost:8551/, auth=true, service: deposit_contract_rpc
Jul 15 15:26:10.439 ERRO Error updating deposit contract cache   error: Invalid endpoint state: RequestFailed("eth_chainId call failed HttpClient(url: http://localhost:8551/, kind: request, detail: error trying to connect: tcp connect error: No connection could be made because the target machine actively refused it. (os error 10061))"), retry_millis: 60000, service: deposit_contract_rpc
Jul 15 15:26:11.518 ERRO Error during execution engine upcheck   error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:26:18.000 INFO Syncing                                 est_time: --, distance: 79 slots (15 mins), peers: 11, service: slot_notifier
Jul 15 15:26:18.002 WARN Syncing deposit contract block cache    est_blocks_remaining: initializing deposits, service: slot_notifier
Jul 15 15:26:23.525 ERRO Error during execution engine upcheck   error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:26:30.001 INFO Syncing                                 est_time: --, distance: 80 slots (16 mins), peers: 11, service: slot_notifier
Jul 15 15:26:30.003 WARN Syncing deposit contract block cache    est_blocks_remaining: initializing deposits, service: slot_notifier
Jul 15 15:26:35.534 ERRO Error during execution engine upcheck   error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:26:42.003 INFO Syncing                                 est_time: --, distance: 81 slots (16 mins), peers: 11, service: slot_notifier
Jul 15 15:26:42.006 WARN Syncing deposit contract block cache    est_blocks_remaining: initializing deposits, service: slot_notifier
Jul 15 15:26:47.507 WARN Execution engine call failed            error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:26:47.508 ERRO Unable to get transition config         error: Api { error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out) }, service: exec
Jul 15 15:26:47.510 ERRO Failed to check transition config       error: EngineError(Api { error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out) }), service: exec
Jul 15 15:26:47.533 ERRO Error during execution engine upcheck   error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:26:48.513 ERRO Error during execution engine upcheck   error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:26:54.000 INFO Syncing                                 est_time: --, distance: 82 slots (16 mins), peers: 18, service: slot_notifier
Jul 15 15:26:54.003 WARN Syncing deposit contract block cache    est_blocks_remaining: initializing deposits, service: slot_notifier
Jul 15 15:26:59.540 ERRO Error during execution engine upcheck   error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:27:06.004 INFO Syncing                                 est_time: --, distance: 83 slots (16 mins), peers: 17, service: slot_notifier
Jul 15 15:27:06.006 WARN Syncing deposit contract block cache    est_blocks_remaining: initializing deposits, service: slot_notifier
Jul 15 15:27:10.420 WARN Error connecting to eth1 node endpoint  endpoint: http://localhost:8551/, auth=true, service: deposit_contract_rpc
Jul 15 15:27:10.421 ERRO Error updating deposit contract cache   error: Invalid endpoint state: RequestFailed("eth_chainId call failed HttpClient(url: http://localhost:8551/, kind: request, detail: error trying to connect: tcp connect error: No connection could be made because the target machine actively refused it. (os error 10061))"), retry_millis: 60000, service: deposit_contract_rpc
Jul 15 15:27:11.544 ERRO Error during execution engine upcheck   error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:27:18.006 INFO Syncing                                 est_time: --, distance: 84 slots (16 mins), peers: 18, service: slot_notifier
Jul 15 15:27:18.008 WARN Syncing deposit contract block cache    est_blocks_remaining: initializing deposits, service: slot_notifier
Jul 15 15:27:23.546 ERRO Error during execution engine upcheck   error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:27:30.000 INFO Syncing                                 est_time: --, distance: 85 slots (17 mins), peers: 18, service: slot_notifier
Jul 15 15:27:30.002 WARN Syncing deposit contract block cache    est_blocks_remaining: initializing deposits, service: slot_notifier
Jul 15 15:27:35.549 ERRO Error during execution engine upcheck   error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:27:42.000 INFO Syncing                                 est_time: --, distance: 86 slots (17 mins), peers: 18, service: slot_notifier
Jul 15 15:27:42.003 WARN Syncing deposit contract block cache    est_blocks_remaining: initializing deposits, service: slot_notifier
Jul 15 15:27:47.553 ERRO Error during execution engine upcheck   error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:27:48.528 WARN Execution engine call failed            error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:27:48.530 ERRO Unable to get transition config         error: Api { error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out) }, service: exec
Jul 15 15:27:48.531 ERRO Failed to check transition config       error: EngineError(Api { error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out) }), service: exec
Jul 15 15:27:49.536 ERRO Error during execution engine upcheck   error: HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out), service: exec
Jul 15 15:27:54.000 INFO Syncing                                 est_time: --, distance: 87 slots (17 mins), peers: 23, service: slot_notifier
Jul 15 15:27:54.002 WARN Syncing deposit contract block cache    est_blocks_remaining: initializing deposits, service: slot_notifier

Platform(s)

Windows (x86)

What version/commit are you on?

reth Version: 0.1.0-alpha.3 Commit SHA: c1b1eac5 Build Timestamp: 2023-07-15T03:22:44.069834700Z Build Features: default,jemalloc Build Profile: release

What database version are you on?

PS C:\Users\ControlCplusControlV\Code\node> reth db version
Current database version: 1
Local database version: 1

If you’ve built Reth from source, provide the full command you used

cargo install --locked --path bin/reth --bin reth

Code of Conduct

• I agree to follow the Code of Conduct