electrs: Bug: tip_loop thread failed: JSON-RPC error: transport error: Didn't receive response data in time, timed out.
Reproduced on both arm64 RaspberryOS (Debian base) and amd64 Ubuntu 20.04 VM both using RaspiBlitz scripts. The install script is developed and discussed here: https://github.com/rootzoll/raspiblitz/issues/2548 Running 0.9.0-rc1 and same on master.
indexed and compacted successfully, but since then Electrs keeps restarting with the error:
WARN electrs::thread] tip_loop thread failed: JSON-RPC error: transport error: Didn't receive response data in time, timed out.
Bitcoind is running without errors.
More logs:
systemd[1]: Stopped Electrs.
systemd[1]: Started Electrs.
electrs[1167816]: Config { network: Bitcoin, db_path: "/mnt/hdd/app-storage/electrs/db/bitcoin", daemon_dir: "/home/electrs/.bitcoin", daemon_auth: UserPass("raspibolt", "<sensitive>"), daemon_rpc_addr: 127.0.0.1:8332, daemon_p2p_addr: 127.0.0.1:8333, electrum_rpc_addr: 127.0.0.1:50001, monitoring_addr: 127.0.0.1:4224, wait_duration: 20s, index_batch_size: 10, index_lookup_limit: Some(200), auto_reindex: true, ignore_mempool: false, sync_once: false, server_banner: "Welcome to electrs 3041e89cd2fb377541b929d852ef6298c2d4e60a - the Electrum Rust Server on your RaspiBlitz", args: [] }
electrs[1167816]: [2021-09-21T08:43:40.686Z INFO electrs::metrics::metrics_impl] serving Prometheus metrics on 127.0.0.1:4224
electrs[1167816]: [2021-09-21T08:43:40.984Z INFO electrs::db] "/mnt/hdd/app-storage/electrs/db/bitcoin": 129 SST files, 31.649062994 GB, 3.94259865 Grows
electrs[1167816]: [2021-09-21T08:43:43.019Z INFO electrs::chain] loading 701538 headers, tip=0000000000000000000be09a4d70c768038a3b7e153517276e9d04e1fb615c48
electrs[1167816]: [2021-09-21T08:43:44.904Z INFO electrs::chain] chain updated: tip=0000000000000000000be09a4d70c768038a3b7e153517276e9d04e1fb615c48, height=701538
electrs[1167816]: [2021-09-21T08:43:47.712Z INFO electrs::server] serving Electrum RPC on 127.0.0.1:50001
electrs[1167816]: [2021-09-21T08:44:07.716Z WARN electrs::thread] tip_loop thread failed: JSON-RPC error: transport error: Didn't receive response data in time, timed out.
electrs[1167816]: [2021-09-21T08:44:07.716Z INFO electrs::server] stopping Electrum RPC server
electrs[1167816]: [2021-09-21T08:44:07.716Z INFO electrs::db] closing DB at /mnt/hdd/app-storage/electrs/db/bitcoin
systemd[1]: electrs.service: Succeeded.
systemd[1]: electrs.service: Scheduled restart job, restart counter is at 2.
systemd[1]: Stopped Electrs.
Installed with: https://github.com/rootzoll/raspiblitz/blob/dev/home.admin/config.scripts/bonus.electrs.sh Indexed successfully then keeps restarting as above.
Configuration
sudo cat /home/electrs/.electrs/config.toml
verbose = 2
timestamp = true
jsonrpc_import = true
index-batch-size = 10
db_dir = "/mnt/hdd/app-storage/electrs/db"
auth = "raspibolt:XXXXX"
# allow BTC-RPC-explorer show tx-s for addresses with a history of more than 100
txid_limit = 1000
# https://github.com/Stadicus/RaspiBolt/issues/646
wait_duration_secs = 20
server_banner = "Welcome to electrs 3041e89cd2fb377541b929d852ef6298c2d4e60a - the Electrum Rust Server on your RaspiBlitz"
$ cat /etc/systemd/system/electrs.service
[Unit]
Description=Electrs
After=bitcoind.service
[Service]
WorkingDirectory=/home/electrs/electrs
ExecStart=/home/electrs/electrs/target/release/electrs --electrum-rpc-addr="0.0.0.0:50001"
User=electrs
Group=electrs
Type=simple
TimeoutSec=60
Restart=always
RestartSec=60
# Hardening measures
PrivateTmp=true
ProtectSystem=full
NoNewPrivileges=true
PrivateDevices=true
[Install]
WantedBy=multi-user.target
(same when removing the hardening measures from systemd)
System running electrs
Linux 5.11.0-34-generic #36~20.04.1-Ubuntu x86_64 GNU/Linux
and
Linux 5.10.60-v8+ #1449 SMP PREEMPT aarch64 GNU/Linux
Both systems have 4GB RAM available with 1.4GB and 900MB empty. Swapfile at 1GB usage out of 2 and 4GB.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 30 (19 by maintainers)
Commits related to this issue
- Don't fail if `wait_for_new_block` RPC timeouts Following #495. — committed to romanz/electrs by romanz 3 years ago
- electrs: remove wait_duration_secs setting as discussed in: https://github.com/romanz/electrs/issues/495#issuecomment-923845230 and #2548 — committed to raspiblitz/raspiblitz by openoms 3 years ago
- Allow configuration of JSON-RPC timeout Following https://github.com/romanz/electrs/issues/495. Also, fix https://github.com/romanz/electrs/issues/443. — committed to romanz/electrs by romanz 3 years ago
- Don't fail on HTTP timeouts from bitcoind RPC It should be OK to retry in case of a RPC timeout. Should help with https://github.com/romanz/electrs/issues/495. — committed to romanz/electrs by romanz 3 years ago
- Don't fail on HTTP timeouts from bitcoind RPC It should be OK to retry in case of a RPC timeout. Should help with https://github.com/romanz/electrs/issues/495. — committed to romanz/electrs by romanz 3 years ago
Great! I will leave it debug until tomorrow and grep out any fails/errors if they arise. FYI, that 4gb RPi has nearly every RaspiBlitz service including testnet/signet running simultaneously! My thanks again to all.
Thanks for the debugging! You can also try the latest
master(which should log a warning but not fail on HTTP timeouts).BTW, it’s cool to see that it takes <1 second to index a new block on a Raspberry Pi 😃
RPi 4gb / kernel 5.10.60-v8+ / Core 22.0
I updated electrs with openoms dev script and after 6 mins the same bug in this thread appeared with the default wait_duration_secs = 10
On following auto restart it stopped after 12 mins with “notified via SIG15”
I altered wait_duration_secs = 14 and restarted electrs manually. So far out of a total 60 mins and 7 blocks, I have only gotten the following errors, at the 18th minute: ERROR 67: disconnecting due to failed to send response electrs-check/null ERROR 68: disconnecting due to failed to send response electrs-check/null
SparrowWallet is connecting and “Finding transactions…” no problem within the above 60 mins.
I would copy/paste but it is a bit awkward at the moment!
We should use
jsonrpc::simple_http::Builder::timeout: https://github.com/apoelstra/rust-jsonrpc/blob/64005f2896b98934e58c3a22f7d1430b110cb01a/src/simple_http.rs#L231Ah, I see now. I think apart from warning about error it’d be great to modify jsonrpc to support larger timeout for specific request (and add a buffer of few seconds to it).
on the RPi4 4GB there was not a perceivable difference between using:
wait_duration_secs = 1or just commenting out the line falling back to the default 10. Synced a larger wallet with 400 addresses in less than a minute with both settings over Tor.I prefer leaving this at the default value especially if it wouldn’t fail later on.
From my point of view the issue is solved, thank you for the quick help!
I have created https://github.com/romanz/electrs/compare/wait-timeout, which should warn in case of RPC timeouts (when waiting for new blocks) instead of failing.
@romanz great news, lowering the
wait_duration_secsto 1 results in not having these restarts any more. Now the question is what would be the best value to use? A too low value would cause failures too I presume. The default 10 sounds to be a good middle-ground, testing it next.