bitcoin: b-msghand[4988] general protection fault

Is there an existing issue for this?

  • I have searched the existing issues

Current behaviour

Hello dear devs, I caught segfault today, running latest Bitcoin Code in full node mode on AMD64 Linux server.

Expected behaviour

no segfault

Steps to reproduce

I had not seen bitcoind crash for a long time. It’s a rare one I guess.

Relevant log output

dmesg: traps: b-msghand[4988] general protection fault ip:55783548b790 sp:7f57627f3978 error:0 in bitcoind[557834c96000+aee000]

gdb backtrace: debug.txt

How did you obtain Bitcoin Core

Pre-built binaries

What version of Bitcoin Core are you using?

v27.0.0

Operating system and version

Debian GNU/Linux 12 (bookworm)

Machine specifications

Debian Stable server with AMD Ryzen 7 1700 and ECC RAM, everything stable and reliable. Bitcoin Core is using Tor and I2P network only, running in full mode. Blocks on RAID10 array, hardware is reliable.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (10 by maintainers)

Most upvoted comments

The main issue here is no core dump (in future please try and include it in the issue), and I don’t think there’s anything we can do with the gdb backtrace. I tested that using the 27.0 binary, and the associated debug symbols, works for inspecting a core dump. i.e:

# assuming you have bitcoind and bitcoind.dbg from
# https://bitcoincore.org/bin/bitcoin-core-27.0/bitcoin-27.0-x86_64-linux-gnu.tar.gz
# https://bitcoincore.org/bin/bitcoin-core-27.0/bitcoin-27.0-x86_64-linux-gnu-debug.tar.gz
# mv bitcoind.dbg bitcoind.dbg.txt (for now)

# I used trip to invoke a malloc failure
# Note that how/where the core dump is generated will depend on your system.

trip malloc -- ./bitcoind
> terminate called without an active exception
> Aborted (core dumped)

# get the dump
cp /var/lib/apport/coredump/core._root_segfault_bitcoind.0.d7bf5f78-1c2a-4e45-b21a-1445b80537f5.21806.6531920 dump

Give GDB bitcoind and the dump:

gdb bitcoind dump
GNU gdb (Ubuntu 15.0.50.20240403-0ubuntu1) 15.0.50.20240403-git
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bitcoind...
(No debugging symbols found in bitcoind)
[New LWP 21806]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by './bitcoind'.
Program terminated with signal SIGABRT, Aborted.
Download failed: Invalid argument.  Continuing without source file ./nptl/./nptl/pthread_kill.c.
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44

warning: 44	./nptl/pthread_kill.c: No such file or directory
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007b39a084526e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007b39a08288ff in __GI_abort () at ./stdlib/abort.c:79
#5  0x00005f813f8ae4f7 in ?? ()
#6  0x00005f814021fc76 in ?? ()
#7  0x00005f814021fce1 in ?? ()
#8  0x00005f814021eb7b in ?? ()
#9  0x00005f813f8ae3fc in ?? ()
#10 0x00005f813f8b463d in ?? ()
#11 0x00005f81402d8ec5 in ?? ()
#12 0x00007b39a082a264 in __libc_start_main_impl (main=0x5f813f8b2bf0, argc=1, argv=0x7fff15128688, init=0x5f81402d8e80, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff15128678) at ../csu/libc-start.c:343
#13 0x00005f813f8cb4ba in ?? ()

No symbols. Return bitcoind.dbg.txt to bitcoind.dgb and retry. Note that GDB will load the debug symbols automatically, but they can also be passed explicitly:

mv bitcoind.dbg.txt bitcoind.dbg
# gdb bitcoind dump
GNU gdb (Ubuntu 15.0.50.20240403-0ubuntu1) 15.0.50.20240403-git
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bitcoind...
Reading symbols from /root/segfault/bitcoind.dbg...
[New LWP 21806]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
Downloading separate debug info for /usr/local/bin/trip
Downloading separate debug info for system-supplied DSO at 0x7fff151c3000                                                                                                                                                                                         
[Thread debugging using libthread_db enabled]                                                                                                                                                                                                                     
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by './bitcoind'.
Program terminated with signal SIGABRT, Aborted.
Download failed: Invalid argument.  Continuing without source file ./nptl/./nptl/pthread_kill.c.
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44

warning: 44	./nptl/pthread_kill.c: No such file or directory
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007b39a084526e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007b39a08288ff in __GI_abort () at ./stdlib/abort.c:79
#5  0x00005f813f8ae4f7 in __gnu_cxx::__verbose_terminate_handler() [clone .cold] ()
#6  0x00005f814021fc76 in __cxxabiv1::__terminate(void (*)()) ()
#7  0x00005f814021fce1 in std::terminate() ()
#8  0x00005f814021eb7b in __cxa_allocate_exception ()
#9  0x00005f813f8ae3fc in operator new(unsigned long) [clone .cold] ()
#10 0x00005f813f8b463d in __gnu_cxx::new_allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, true> >::allocate (this=<optimized out>, __n=1)
    at /usr/include/c++/ext/new_allocator.h:103
#11 std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, true> >::allocate (__n=1, this=<optimized out>) at /usr/include/c++/bits/allocator.h:181
#12 std::allocator_traits<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, true> > >::allocate (__n=1, __a=...)
    at /usr/include/c++/bits/alloc_traits.h:460
#13 std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, true> > >::_M_allocate_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long&> (this=0x5f81406bbe80 <(anonymous namespace)::V2_MESSAGE_MAP>) at /usr/include/c++/bits/hashtable_policy.h:2033
#14 std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_Scoped_node::_Scoped_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long&> (__h=0x5f81406bbe80 <(anonymous namespace)::V2_MESSAGE_MAP>, this=<synthetic pointer>) at /usr/include/c++/bits/hashtable.h:286
#15 std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long&> (this=0x5f81406bbe80 <(anonymous namespace)::V2_MESSAGE_MAP>) at /usr/include/c++/bits/hashtable.h:1728
#16 std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long&> (this=0x5f81406bbe80 <(anonymous namespace)::V2_MESSAGE_MAP>) at /usr/include/c++/bits/hashtable.h:836
#17 std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned char, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char> > >::emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long&> (this=0x5f81406bbe80 <(anonymous namespace)::V2_MESSAGE_MAP>) at /usr/include/c++/bits/unordered_map.h:389
#18 (anonymous namespace)::V2MessageMap::V2MessageMap (this=0x5f81406bbe80 <(anonymous namespace)::V2_MESSAGE_MAP>) at net.cpp:928
#19 __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at net.cpp:940
#20 _GLOBAL__sub_I__Z22NET_MESSAGE_TYPE_OTHERB5cxx11 () at net.cpp:3899
#21 0x00005f81402d8ec5 in __libc_csu_init (argc=argc@entry=1, argv=argv@entry=0x7fff15128688, envp=0x7fff15128698) at elf-init.c:88
#22 0x00007b39a082a264 in __libc_start_main_impl (main=0x5f813f8b2bf0 <main(int, char**)>, argc=1, argv=0x7fff15128688, init=0x5f81402d8e80 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff15128678)
    at ../csu/libc-start.c:343
#23 0x00005f813f8cb4ba in _start () at ../sysdeps/x86_64/start.S:120
(gdb) 

where I can get trip program?

https://codeberg.org/pkal/trip

Was there anything unusual in the debug log before the crash?

Logs contains nothing else worth mentioning, machine continues to run as intended.

Do you mean the machine logs contain nothing worth mentioning or the Bitcoin Core debug.log?

Just in case you are not aware: Bitcoin Core writes to a debug.log file in your data directory (where the blockchain is stored too). It would be useful to know what Bitcoin Core logged at around 2024-08-23 23:47:55 BST.

Of course, sorry for not including it earlier. I am attaching first and last 100 lines of the log. In between, there is nothing useful, I think. At the end of the log, you will see nothing logged around time of the crash, unfortunately.

bitcoind.log

Was there anything unusual in the debug log before the crash?