bitcoin: bitcoind hangs waiting for `g_requests.empty()`

Is there an existing issue for this?

  • I have searched the existing issues

Current behaviour

bitcoind hangs

Expected behaviour

bitcoind shouldn’t hang

I was able to dump the running threads and it was stuck waiting for g_requests to be empty. The issue seems to be that this callback may not actually be called in case of error: https://github.com/bitcoin/bitcoin/blob/22139f6e83a2bedd2dad9f280567d2c76c54252f/src/httpserver.cpp#L222-L227

See this issue https://github.com/libevent/libevent/issues/589 for more details. I think what happens is that interrupting bitcoin-cli before it has received the reply calls the error callback and then the success callback is never called.

I think this is also a memory leak since g_requests won’t get cleaned up. Whether it’s severe or not probably depends on the caller and what type of requests they are making. This new code was introduced here https://github.com/bitcoin/bitcoin/pull/26742

Steps to reproduce

I was running a simple shell script in 4 terminal windows for fun:

#!/bin/bash

for i in {1..50000}
do
	./src/bitcoin-cli -rpcuser=admin -rpcpassword=pass getrawtransaction 217819875434324c412d5e938f7f3c1670fb60814974cf7958f1fac90ee5afb7
done

I was messing around with this script and randomly Ctrl-C’d each terminal window in rapid succession and then when I waited up to a minute and Ctrl-C’d bitcoind, it didn’t shutdown. I can reproduce it pretty reliably as well.

Compilation commands:

./autogen.sh; ./configure --without-wallet --with-gui=no --disable-zmq; make -j4

Run-time options (note that I was able to reproduce this with the default number of rpcthreads as well):

./src/bitcoind -txindex -rpcthreads=1 -debug

Output of gcc --version:

Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Output of g++ --version:

Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

My libevent version is 2.1.12

Relevant log output

^C2023-05-22T20:56:10Z [http] Interrupting HTTP server
2023-05-22T20:56:10Z [rpc] Interrupting HTTP RPC server
2023-05-22T20:56:10Z [rpc] Interrupting RPC
2023-05-22T20:56:10Z tor: Thread interrupt
2023-05-22T20:56:10Z Shutdown: In progress...
2023-05-22T20:56:10Z [rpc] Stopping HTTP RPC server
2023-05-22T20:56:10Z torcontrol thread exit
2023-05-22T20:56:10Z [http] Unregistering HTTP handler for / (exactmatch 1)
2023-05-22T20:56:10Z addcon thread exit
2023-05-22T20:56:10Z [http] Unregistering HTTP handler for /wallet/ (exactmatch 0)
2023-05-22T20:56:10Z [rpc] Stopping RPC
2023-05-22T20:56:10Z net thread exit
2023-05-22T20:56:10Z [rpc] RPC stopped.
2023-05-22T20:56:10Z [http] Stopping HTTP server
2023-05-22T20:56:10Z [http] Waiting for HTTP worker threads to exit
2023-05-22T20:56:10Z [http] Exited http event loop
2023-05-22T20:56:10Z [http] Waiting for 1 requests to stop HTTP server

The important bit is the very last log line. bitcoind was still alive and periodically printed log lines about feeding dynamic environment data into the RNG.

How did you obtain Bitcoin Core

Compiled from source

What version of Bitcoin Core are you using?

17acb2782a66f033238340e4fb81009e7f79e97a

Operating system and version

macOS intel

Machine specifications

No response

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 21 (18 by maintainers)

Commits related to this issue

Most upvoted comments

Just upgraded to v25 on my iMac

I would suggest upgrading to 25.1, which has fixed this issue.

As @willcl-ark, tried @theStack’s method with no sucess, my libevent version is 2.1.7, I’ll try to update it and retry again. Thanks.

@Crypt-iQ I did: Ctrl-C the waitforblockheight command before trying to stop bitcoind.

Good to know, I’ll have to test out that version

I was able to fix this with my branch here Crypt-iQ@4fd7adb by adding a evhttp_connection_set_closecb callback. I didn’t need to modify the logic related to EV_READ and no functional test errors are introduced on my machine

Can confirm that this patch fixes the issue on my side.

Thanks for reporting! I observed this issue sporadically, but were never able to reproduce it until now. With the help of your description (“… interrupting bitcoin-cli before it has received the reply …”) I found it’s actually quite trivial at least on my machine:

  1. startup bitcoind -debug=http (and any other desired flags; note that the hangup also happens without the debug flag, but this allows us to see where it’s hanging)
  2. execute bitcoin-cli waitforblockheight 1234567 (or any other RPC call which doesn’t terminate immediately)
  3. terminate bitcoin-cli via CTRL-C
  4. terminate bitcoind via CTRL-C

The shutdown leads to the following messages and then hangs forever:

2023-05-23T13:58:01Z [http] Unregistering HTTP handler for / (exactmatch 1)
2023-05-23T13:58:01Z [http] Unregistering HTTP handler for /wallet/ (exactmatch 0)
2023-05-23T13:58:01Z [http] Stopping HTTP server
2023-05-23T13:58:01Z [http] Waiting for HTTP worker threads to exit
2023-05-23T13:58:01Z [http] Waiting for 1 requests to stop HTTP server
2023-05-23T13:58:01Z [http] Exited http event loop
2023-05-23T13:58:01Z msghand thread exit
2023-05-23T13:58:01Z net thread exit
2023-05-23T13:58:03Z opencon thread exit

Tested on master branch (commit 5ef2c1ee7a7416907e0f9135dc0701a88d92a7f6) with default configuration, running OpenBSD 7.3 with libevent 2.1.12 here.