bitcoin: bitcoind hangs waiting for `g_requests.empty()`
Is there an existing issue for this?
- I have searched the existing issues
Current behaviour
bitcoind hangs
Expected behaviour
bitcoind shouldn’t hang
I was able to dump the running threads and it was stuck waiting for g_requests
to be empty. The issue seems to be that this callback may not actually be called in case of error: https://github.com/bitcoin/bitcoin/blob/22139f6e83a2bedd2dad9f280567d2c76c54252f/src/httpserver.cpp#L222-L227
See this issue https://github.com/libevent/libevent/issues/589 for more details. I think what happens is that interrupting bitcoin-cli
before it has received the reply calls the error callback and then the success callback is never called.
I think this is also a memory leak since g_requests
won’t get cleaned up. Whether it’s severe or not probably depends on the caller and what type of requests they are making. This new code was introduced here https://github.com/bitcoin/bitcoin/pull/26742
Steps to reproduce
I was running a simple shell script in 4 terminal windows for fun:
#!/bin/bash
for i in {1..50000}
do
./src/bitcoin-cli -rpcuser=admin -rpcpassword=pass getrawtransaction 217819875434324c412d5e938f7f3c1670fb60814974cf7958f1fac90ee5afb7
done
I was messing around with this script and randomly Ctrl-C’d each terminal window in rapid succession and then when I waited up to a minute and Ctrl-C’d bitcoind, it didn’t shutdown. I can reproduce it pretty reliably as well.
Compilation commands:
./autogen.sh; ./configure --without-wallet --with-gui=no --disable-zmq; make -j4
Run-time options (note that I was able to reproduce this with the default number of rpcthreads as well):
./src/bitcoind -txindex -rpcthreads=1 -debug
Output of gcc --version
:
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
Output of g++ --version
:
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
My libevent version is 2.1.12
Relevant log output
^C2023-05-22T20:56:10Z [http] Interrupting HTTP server
2023-05-22T20:56:10Z [rpc] Interrupting HTTP RPC server
2023-05-22T20:56:10Z [rpc] Interrupting RPC
2023-05-22T20:56:10Z tor: Thread interrupt
2023-05-22T20:56:10Z Shutdown: In progress...
2023-05-22T20:56:10Z [rpc] Stopping HTTP RPC server
2023-05-22T20:56:10Z torcontrol thread exit
2023-05-22T20:56:10Z [http] Unregistering HTTP handler for / (exactmatch 1)
2023-05-22T20:56:10Z addcon thread exit
2023-05-22T20:56:10Z [http] Unregistering HTTP handler for /wallet/ (exactmatch 0)
2023-05-22T20:56:10Z [rpc] Stopping RPC
2023-05-22T20:56:10Z net thread exit
2023-05-22T20:56:10Z [rpc] RPC stopped.
2023-05-22T20:56:10Z [http] Stopping HTTP server
2023-05-22T20:56:10Z [http] Waiting for HTTP worker threads to exit
2023-05-22T20:56:10Z [http] Exited http event loop
2023-05-22T20:56:10Z [http] Waiting for 1 requests to stop HTTP server
The important bit is the very last log line. bitcoind was still alive and periodically printed log lines about feeding dynamic environment data into the RNG.
How did you obtain Bitcoin Core
Compiled from source
What version of Bitcoin Core are you using?
17acb2782a66f033238340e4fb81009e7f79e97a
Operating system and version
macOS intel
Machine specifications
No response
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 21 (18 by maintainers)
Commits related to this issue
- Merge bitcoin/bitcoin#28551: http: bugfix: allow server shutdown in case of remote client disconnection 68f23f57d77bc172ed39ecdd4d2d5cd5e13cf483 http: bugfix: track closed connection (stickies-v) 084... — committed to bitcoin-core/gui by fanquake 9 months ago
- Merge bitcoin/bitcoin#28551: http: bugfix: allow server shutdown in case of remote client disconnection 68f23f57d77bc172ed39ecdd4d2d5cd5e13cf483 http: bugfix: track closed connection (stickies-v) 084... — committed to syscoin/syscoin by fanquake 9 months ago
I would suggest upgrading to 25.1, which has fixed this issue.
Good to know, I’ll have to test out that version
Can confirm that this patch fixes the issue on my side.
Thanks for reporting! I observed this issue sporadically, but were never able to reproduce it until now. With the help of your description (“… interrupting bitcoin-cli before it has received the reply …”) I found it’s actually quite trivial at least on my machine:
bitcoind -debug=http
(and any other desired flags; note that the hangup also happens without the debug flag, but this allows us to see where it’s hanging)bitcoin-cli waitforblockheight 1234567
(or any other RPC call which doesn’t terminate immediately)The shutdown leads to the following messages and then hangs forever:
Tested on master branch (commit 5ef2c1ee7a7416907e0f9135dc0701a88d92a7f6) with default configuration, running OpenBSD 7.3 with libevent 2.1.12 here.