electrs: Config: electrum on a client computer cannot reach electrs on the server, electrs itself seems to be running fine

Have you read the documentation? YES

How did you configure electrs?

electrs@mlbb2:~/.electrs$ cat electrs.conf network = “bitcoin” daemon_dir= “/home/bitcoin/.bitcoin” daemon_rpc_addr = “127.0.0.1:8332” daemon_p2p_addr = “127.0.0.1:8333”

electrum_rpc_addr = “127.0.0.1:50001”

db_dir = “/home/electrs/.electrs/db” index_lookup_limit = 1000

log_filters = “INFO” timestamp = true

The issue is that electrum running on another computer cannot reach electrs. I use nginx as a reverse proxy. The log below shows (I believe) that electrum can reach the server OK, the request seems to be passed on to electrs but no response is received.

electrs itself is running fine.

Environment variables: ELECTRS_X=Y;... Arguments: --foo

Debug output of configuration

Below I describe my nginx configuration. My actual domain has been replaced by MY_SITE.COM and my true IP by 99.99.99.99. I also describe how nginx was configured and how I have tried to trace network traffic using lsof, curl, and tcpdump.

I have compiled nginx with these arguments:

marius@mlbb2:~$ nginx -V
nginx version: nginx/1.25.3
built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04) 
built with OpenSSL 3.0.2 15 Mar 2022
TLS SNI support enabled
configure arguments: --sbin-path=/usr/local/bin/nginx --conf-path=/etc/nginx/nginx.conf --modules-path=/etc/nginx/modules --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --with-pcre --with-http_ssl_module --with-http_gzip_static_module --with-http_gunzip_module --with-http_v2_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-stream=dynamic --with-stream_ssl_preread_module --with-stream_ssl_module --with-stream_geoip_module=dynamic

nginx.conf and the included electrs.conf:

marius@mlbb2:~$ cat /etc/nginx/nginx.conf
user www-data;
worker_processes auto;
pid /run/nginx.pid;

events {
    worker_connections 1024;
}

http { ... details removed ... }

stream {
    ssl_certificate /etc/letsencrypt/live/MY_SITE.COM/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/MY_SITE.COM/privkey.pem; # managed by Certbot
    ssl_session_timeout 4h;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;

    include streams-enabled/*.conf;
}

marius@mlbb2:~$ cat /etc/nginx/streams-enabled/electrs.conf
upstream electrs {
    server 127.0.0.1:50001;
}

server {
    listen 50002 ssl;
    proxy_pass electrs;
}

below we see that electrs is listening on ports 50001 and nginx is listening on port 50002:

marius@mlbb2:~$ sudo lsof -i -n -P | grep LISTEN
[sudo] password for marius: 
systemd        1            root  147u  IPv4    20186      0t0  TCP *:111 (LISTEN)
systemd        1            root  149u  IPv6    20187      0t0  TCP *:111 (LISTEN)
rpcbind      539            _rpc    4u  IPv4    20186      0t0  TCP *:111 (LISTEN)
rpcbind      539            _rpc    6u  IPv6    20187      0t0  TCP *:111 (LISTEN)
systemd-r    661 systemd-resolve   14u  IPv4    21687      0t0  TCP 127.0.0.53:53 (LISTEN)
node         676             rtl   25u  IPv6    21166      0t0  TCP *:7000 (LISTEN)
sshd         716            root    3u  IPv4    23719      0t0  TCP *:22 (LISTEN)
sshd         716            root    4u  IPv6    23730      0t0  TCP *:22 (LISTEN)
rpc.statd    771           statd    9u  IPv4    25254      0t0  TCP *:41613 (LISTEN)
rpc.statd    771           statd   11u  IPv6    25260      0t0  TCP *:38191 (LISTEN)
bitcoind     867         bitcoin   11u  IPv6    21150      0t0  TCP [::1]:8332 (LISTEN)
bitcoind     867         bitcoin   12u  IPv4    21152      0t0  TCP 127.0.0.1:8332 (LISTEN)
bitcoind     867         bitcoin   28u  IPv4    21341      0t0  TCP 127.0.0.1:8334 (LISTEN)
bitcoind     867         bitcoin   29u  IPv6    21342      0t0  TCP *:8333 (LISTEN)
bitcoind     867         bitcoin   31u  IPv4    21343      0t0  TCP *:8333 (LISTEN)
tor          879      debian-tor    6u  IPv4    24237      0t0  TCP 127.0.0.1:9050 (LISTEN)
tor          879      debian-tor    7u  IPv4    24238      0t0  TCP 127.0.0.1:9051 (LISTEN)
electrs     1158         electrs    3u  IPv4    25702      0t0  TCP 127.0.0.1:4224 (LISTEN)
electrs     1158         electrs    4u  IPv4    25703      0t0  TCP 127.0.0.1:50001 (LISTEN)
python      1747          lnbits   19u  IPv4    30823      0t0  TCP 127.0.0.1:5000 (LISTEN)
lightning  56323       lightning    5u  IPv4   624561      0t0  TCP 127.0.0.1:9736 (LISTEN)
lightning  56323       lightning    6u  IPv4   624562      0t0  TCP *:9735 (LISTEN)
node       56610       lightning   26u  IPv6   628171      0t0  TCP *:3001 (LISTEN)
node       56610       lightning   27u  IPv6   628172      0t0  TCP *:4001 (LISTEN)
nginx     975705            root    6u  IPv4 11787171      0t0  TCP *:443 (LISTEN)
nginx     975705            root    7u  IPv4 11787172      0t0  TCP *:80 (LISTEN)
nginx     975705            root    8u  IPv4 11787173      0t0  TCP *:50002 (LISTEN)
(the last three lines repeated 3 times as I have 4 workers)

I have compiled nginx with these arguments:

marius@mlbb2:~$ nginx -V
nginx version: nginx/1.25.3
built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04) 
built with OpenSSL 3.0.2 15 Mar 2022
TLS SNI support enabled
configure arguments: --sbin-path=/usr/local/bin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --modules-path=/etc/nginx/modules --with-pcre --with-http_ssl_module --with-stream --with-stream_ssl_preread_module --with-stream_ssl_module --with-http_gzip_static_module --with-http_gunzip_module

Electrum client: (getinfo(): “auto_connect”: false, “blockchain_height”: 818697,“connected”: false,“default_wallet”: “/Users/marius/.electrum/wallets/default_wallet”, “fee_per_kb”: null,“network”: “mainnet”,“path”: “/Users/marius/.electrum”,“server”: “MY_SITE.COM”,“server_height”: 0,“spv_nodes”: 9,“version”: “4.4.6”)

as soon as I start the above client I see the following with tcpdump:

marius@mlbb2:~$ sudo tcpdump port 50002 and '(tcp-syn|tcp-ack)!=0'
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens18, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:03:28.828861 IP 99.99.99.99.58811 > mlbb2.fritz.box.50002: Flags [S], seq 1543470894, win 65535, options [mss 1452,nop,wscale 6,nop,nop,TS val 4107815640 ecr 0,sackOK,eol], length 0
12:03:28.828880 IP mlbb2.fritz.box.50002 > 99.99.99.99.58811: Flags [S.], seq 1035971693, ack 1543470895, win 65160, options [mss 1460,sackOK,TS val 4124502244 ecr 4107815640,nop,wscale 7], length 0
12:03:28.832589 IP 99.99.99.99.58811 > mlbb2.fritz.box.50002: Flags [.], ack 1, win 2070, options [nop,nop,TS val 4107815646 ecr 4124502244], length 0
12:03:28.835365 IP 99.99.99.99.58811 > mlbb2.fritz.box.50002: Flags [P.], seq 1:518, ack 1, win 2070, options [nop,nop,TS val 4107815649 ecr 4124502244], length 517
12:03:28.835372 IP mlbb2.fritz.box.50002 > 99.99.99.99.58811: Flags [.], ack 518, win 506, options [nop,nop,TS val 4124502251 ecr 4107815649], length 0
12:03:28.835675 IP mlbb2.fritz.box.50002 > 99.99.99.99.58811: Flags [P.], seq 1:4097, ack 518, win 506, options [nop,nop,TS val 4124502251 ecr 4107815649], length 4096
12:03:28.836354 IP mlbb2.fritz.box.50002 > 99.99.99.99.58811: Flags [P.], seq 4097:4527, ack 518, win 506, options [nop,nop,TS val 4124502252 ecr 4107815649], length 430
12:03:28.838573 IP 99.99.99.99.58811 > mlbb2.fritz.box.50002: Flags [.], ack 2881, win 2025, options [nop,nop,TS val 4107815653 ecr 4124502251], length 0
12:03:28.839609 IP 99.99.99.99.58811 > mlbb2.fritz.box.50002: Flags [.], ack 4527, win 1999, options [nop,nop,TS val 4107815653 ecr 4124502252], length 0
12:03:28.839978 IP 99.99.99.99.58811 > mlbb2.fritz.box.50002: Flags [.], ack 4527, win 2048, options [nop,nop,TS val 4107815653 ecr 4124502252], length 0
12:03:28.841339 IP 99.99.99.99.58811 > mlbb2.fritz.box.50002: Flags [F.], seq 518, ack 4527, win 2048, options [nop,nop,TS val 4107815655 ecr 4124502252], length 0
12:03:28.841371 IP mlbb2.fritz.box.50002 > 99.99.99.99.58811: Flags [F.], seq 4527, ack 519, win 506, options [nop,nop,TS val 4124502257 ecr 4107815655], length 0
12:03:28.846843 IP 99.99.99.99.58811 > mlbb2.fritz.box.50002: Flags [.], ack 4528, win 2048, options [nop,nop,TS val 4107815659 ecr 4124502257], length 0
12:03:30.757673 IP 99.99.99.99.58823 > mlbb2.fritz.box.50002: Flags [S], seq 3684983692, win 65535, options [mss 1452,nop,wscale 6,nop,nop,TS val 1578602325 ecr 0,sackOK,eol], length 0

I assume therefore that the request reaches the server on port 50002.

This is the output of a request to 50002 with the result “* Received HTTP/0.9 when not allowed”.

marius@mlbb2:~$ curl -I --verbose https://MY_SITE.COM:50002
*   Trying 99.99.99.99:50002...
* Connected to MY_SITE.COM (99.99.99.99) port 50002 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=MY_SITE.COM
*  start date: Nov 25 11:03:06 2023 GMT
*  expire date: Feb 23 11:03:05 2024 GMT
*  subjectAltName: host "MY_SITE.COM" matched cert's "MY_SITE.COM"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> HEAD / HTTP/1.1
> Host: MY_SITE.COM:50002
> User-Agent: curl/7.81.0
> Accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Received HTTP/0.9 when not allowed
* Closing connection 0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS alert, close notify (256):
curl: (1) Received HTTP/0.9 when not allowed

Expected behavior the electrs instance running on the server can be reached from the electrum client running on the laptop

Actual behavior No response reaches electrum client on another computer.

About this issue

Original URL
State: closed
Created 7 months ago
Comments: 30 (18 by maintainers)

Most upvoted comments

As far as I am concerned you can close this issue.

I can connect to electrs on LAN using an unencrypted connection. Followed https://bitcointv.com/w/pzyGPRRymuKHwb2YU5Wdkk and https://bitcointv.com/w/gmedeEXmMPidgRTd4uW3hi to the dot; still no luck. I may try to start from scratch. Thanks for the invaluable tips.

marimes on Jan 3, 2024

While unresponsive, the electrs server will still print log lines that it receives incoming JSON-RPC requests.

[2023-12-05T22:19:42.393Z DEBUG electrs::server] 3: recv {"jsonrpc": "2.0", "method": "server.version", "params": ["", "1.4"], "id": 0}

However, these requests are not handled.

All the above is strongly suggesting to me that the rpc.sync() call on this line is blocking for too long, probably because the mempool.sync() call on this line is blocking. This results in JSON-RPC client calls not being handled because the electrs code relies on syncing being paused before handling incoming client requests.

That hypothesis would also explain why i can’t kill the electrum server without using kill -9.

I’m not sure if my case is the same as OP, because OP’s bitcoin node is on the same machine as electrs, and so mempool scanning would take much less time. @marimes do you think this might be your issue? To confirm try launching electrs --ignore-mempool and let me know if this fixes the unresponsiveness

conduition on Dec 5, 2023

In the original issue, you were having trouble with an NGINX TLS proxy for electrs, but if you can connect to electrs over LAN then the issue is definitely not part of electrs.

conduition on Jan 3, 2024

That’s great 😄 @marimes @romanz I suppose we can close this now?

conduition on Jan 3, 2024

@marimes I’d recommend you don’t keep using 0.0.0.0:50001 as a binding address. This makes electrs available publicly on all interfaces which can be dangerous depending on your LAN setup.

To make electrs available ONLY on LAN, find your machine’s LAN IP address and use that as the binding address. Usually this is something like 10.0.0.31 or 192.168.1.31. It’s the same IP that the electrum client would be using to connect to your server. So you’d want something like this in your electrs config file:

electrum_rpc_addr = "192.168.0.82:50001"

Then you’d provide 192.168.0.82:50001:t as the address to the electrum client. (the :t means TCP)

conduition on Dec 28, 2023

This works 🥇 - thank you. I managed to get a connection using 0.0.0.0:50001:t - cool.

Does this suggest that my problem is not related to electrs but rather to the tls and nginx setup?

marimes on Dec 24, 2023

@marimes Unrelated to your problem but the cookie file error indicates that your setup may be sub-optimal. I suggest using After=bitcoind (or whatever the name of the bitcoind service is) And in bitcoind service file use:

Type=notify
ExecStart=/usr/share/bitcoind/bitcoind -startupnotify=systemd-notify --ready
NotifyAccess=all

(Or set startupnotify in config.)

Kixunil on Dec 8, 2023

hi, i have same problem (some times), i use mynodebtc , as i understand electrs is used as electrum server, so it’s happens some time that i cannot reach electrs server in my local network, i tried to restart from mynodebtc dashboard, but this not help, i need to restart entire mynodebtc to connect. when i will have connection stuck, i will post systemctl status, i had problem few minutes ago, and i started to search where to write about this bug, and found oppened topic. best wishes.

onlineapps-cloud on Dec 3, 2023

First of all big thanks for very detailed report! I wish more people were doing it like this!

Could you please check if there’s communication between nginx and electrs? Port 50001 in tcp dump should do it. Also I think @romanz made a mistake not noticing TLS, you need to use openssl s_client -connect MYSITE.COM:50002 instead of nc ....

Possible issues:

electrs may be synchronizing, so it’s not bound
electrs may be broken by some kind of DoS attack - it’s not designed to be publicly accessible and your setup makes it accessible. I’d recommend using a tunnel or check if client certificates can be used (I never tried).
There’s a bug that makes electrs stuck after some time, try restarting it (IIRC I observed this in older versions of electrs but not in new ones. Are you running an up-to-date version?)

Kixunil on Dec 3, 2023