rclcpp: spdlog generates a SEGFAULT when a context is being destructed
Bug report
Required Info:
- Operating System:
- Ubuntu 18.04
- Installation type:
- from source
- Version or commit hash:
- https://gist.githubusercontent.com/ivanpauno/2201f6fb5ad0fc5449d1f2e98f73a73d/raw/77582cbdc6bac5022039cc17fb0c03d0ed9f8289/ros2.repos
(used in https://github.com/ros2/rmw_fastrtps/pull/312#issuecomment-561769677).
- DDS implementation:
- rmw_fastrtps_cpp
- Client library (if applicable):
- rclcpp
Steps to reproduce issue
Unfortunately, I have been able to reproduce this only with the repos file specified above.
That repos file are using rclcpp master, and it only has changes in others repositories.
The error doesn’t seem to be a consequence of any of the changes in my branches, but just an existing error that was triggered randomly (see explanation below).
After building, run test_subscription_publisher_count_api, or test_publisher_subscription_count_api.
Expected behavior
No crashes.
Actual behavior
It segfaults.
Traceback of the segfault
#0 0x00007ffff4b2e3b8 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator
<char> > const, std::shared_ptr<spdlog::logger> >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<spdlog::logger> > >, std:
:__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<ch
ar> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_find_before_node (
this=0x7ffff4d7db98 <spdlog::details::registry::instance()::s_instance+120>, __n=2, __k="root", __code=18029476052510536322) at /usr/include/c++/7/bits/hashtable.h:1548
#1 0x00007ffff4b2a636 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator
<char> > const, std::shared_ptr<spdlog::logger> >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<spdlog::logger> > >, std:
:__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<ch
ar> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_erase (
this=0x7ffff4d7db98 <spdlog::details::registry::instance()::s_instance+120>, __k="root") at /usr/include/c++/7/bits/hashtable.h:1910
#2 0x00007ffff4b26ac5 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator
<char> > const, std::shared_ptr<spdlog::logger> >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<spdlog::logger> > >, std:
:__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<ch
ar> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::erase (
this=0x7ffff4d7db98 <spdlog::details::registry::instance()::s_instance+120>, __k="root") at /usr/include/c++/7/bits/hashtable.h:759
#3 0x00007ffff4b218ff in std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<spdlog::logger>, std::hash<std::__cxx11::basic_string<char, st
d::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, s
td::char_traits<char>, std::allocator<char> > const, std::shared_ptr<spdlog::logger> > > >::erase (this=0x7ffff4d7db98 <spdlog::details::registry::instance()::s_instance+120>, __x="root")
at /usr/include/c++/7/bits/unordered_map.h:814
#4 0x00007ffff4b1b3d2 in spdlog::details::registry::drop (this=0x7ffff4d7db20 <spdlog::details::registry::instance()::s_instance>, logger_name="root")
at /home/ivanpauno/ros2_ws/other_ws/ros2_node_participant_ws/install/spdlog_vendor/include/spdlog/details/registry.h:187
#5 0x00007ffff4b1bceb in spdlog::drop (name="root") at /home/ivanpauno/ros2_ws/other_ws/ros2_node_participant_ws/install/spdlog_vendor/include/spdlog/spdlog.h:113
#6 0x00007ffff4b12f4b in rcl_logging_external_shutdown () at /home/ivanpauno/ros2_ws/other_ws/ros2_node_participant_ws/src/ros2/rcl_logging/rcl_logging_spdlog/src/rcl_logging_spdlog.cpp:155
#7 0x00007ffff6faa420 in rcl_logging_fini () at /home/ivanpauno/ros2_ws/other_ws/ros2_node_participant_ws/src/ros2/rcl/rcl/src/rcl/logging.c:116
#8 0x00007ffff6fa721f in rcl_shutdown (context=0x5555559f7b50) at /home/ivanpauno/ros2_ws/other_ws/ros2_node_participant_ws/src/ros2/rcl/rcl/src/rcl/init.c:188
#9 0x00007ffff75ad229 in rclcpp::Context::shutdown (this=0x5555559f7940, reason="context destructor was called while still not shutdown")
at /home/ivanpauno/ros2_ws/other_ws/ros2_node_participant_ws/src/ros2/rclcpp/rclcpp/src/rclcpp/context.cpp:145
#10 0x00007ffff75ac0f1 in rclcpp::Context::~Context (this=0x5555559f7940, __in_chrg=<optimized out>)
at /home/ivanpauno/ros2_ws/other_ws/ros2_node_participant_ws/src/ros2/rclcpp/rclcpp/src/rclcpp/context.cpp:43
Additional information
The problem happens only with Context objects of static storage duration (it may happen for example with the default context).
When the context is destructed, it will try to shutdown, if its rcl_context_t is still valid (shutdown not manually called) it will shutdown the rcl context , which triggers the shutdown of the loggers, which shutdowns the external loggers, which calls spdlog::drop, which calls a method in a singleton object that was previously destructed, boom đź’Ą .
I checked carefully with the debugger that:
- The spdlog
registrysingleton gets destructed before theContext. - The context that was causing the problem was one of the contexts created here.
IMO, this error was randomly triggered by unrelated changes, because the order of destruction of static/global objects in different shared libraries is not ensured by the linker (and the order of destruction of different static/global objects in the same library is also not ensured, by the compiler).
Also, I think that we haven’t hit this before just because we usually call rclcpp::shutdown or Context::shutdown manually, but I don’t believe that that should be mandatory.
I’m (almost) sure that rclpy should never run into this problem.
Of course, the test can be fixed by avoiding to use a global variable, but I think that’s not a “real” fix, as the problem may be hit again in the future (for example, it may happen with the default context).
Some questions and ideas:
- The logger is being shut down by the context, but all the “state” of the logger is global, and not part of the context. Why the context should shut down it? What if you have two contexts, shut down one, and the other context is still alive and trying to log?
- Do we need to call
spdlog::drop? If not, I wouldn’t call it.
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 16 (11 by maintainers)
Commits related to this issue
- Include rmw_event_t instead of forward declaring it (#933) Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com> — committed to ApexAI/rclcpp by hidmic 5 years ago
- spdlog: do not drop logger in destructor For static objects, the order of destruction is in reverse of the order of construction, but the latter is hard to control except among static objects defined... — committed to OFS/opae-sdk by pcolberg 3 years ago
- spdlog: do not drop logger in destructor For static objects, the order of destruction is in reverse of the order of construction, but the latter is hard to control except among static objects defined... — committed to OFS/opae-sdk by pcolberg 3 years ago
- spdlog: do not drop logger in destructor For static objects, the order of destruction is in reverse of the order of construction, but the latter is hard to control except among static objects defined... — committed to OFS/opae-sdk by pcolberg 3 years ago
- spdlog: do not drop logger in destructor For static objects, the order of destruction is in reverse of the order of construction, but the latter is hard to control except among static objects defined... — committed to OFS/opae-sdk by pcolberg 3 years ago
- spdlog: do not drop logger in destructor For static objects, the order of destruction is in reverse of the order of construction, but the latter is hard to control except among static objects defined... — committed to OFS/opae-sdk by pcolberg 3 years ago
- spdlog: do not drop logger in destructor For static objects, the order of destruction is in reverse of the order of construction, but the latter is hard to control except among static objects defined... — committed to OFS/opae-sdk by pcolberg 3 years ago
spdlog allows this. spdlog logger objects can be created and used like any other object. Using the registry is not mandatory and can be bypassed easily, by creating the objects directly. Also, consider calling spdlog::shutdown() at the end of main to free any spdlog registry items in a predictable place.
Thanks @gabime !! Proposed fixed in https://github.com/ros2/rcl_logging/pull/23.
Ok, I will start by avoiding the use of
spdlog::drop.That’s true. I don’t really like the nifty counter idiom, but it’s fair using it here. I think that the we can use that idiom in rcl_logging_configure and rcl_logging_fini. So the first context creates the loggers, and the last one destroys them.
That will be needed to avoid the problem. I will work on a patch, test it locally, and see if they accept it.
Another reason why we didn’t run into this problem before, is this other issue https://github.com/ros2/rclcpp/issues/893. #906 fixed that problem. I haven’t backport it as the patch is not ABI compatible. So, the fixes of this issue shouldn’t be backported either (except we decide to backport #906).