rmw_fastrtps: Regression in Rolling: error "cannot publish data, at ./src/rmw_publish.cpp:66" during shutdown of binary

Bug report

In Rolling, I observe weird segfaults during shutdown of the test binary:

2024-05-26T18:55:07.8227778Z 12: [moveit_task_constructor_core-test-move-relative-1] [==========] 3 tests from 1 test suite ran. (138 ms total)
2024-05-26T18:55:07.8228588Z 12: [moveit_task_constructor_core-test-move-relative-1] [  PASSED  ] 3 tests.
2024-05-26T18:55:07.8229520Z 12: [moveit_task_constructor_core-test-move-relative-1] cannot publish data, at ./src/rmw_publish.cpp:66 during '__function__'
2024-05-26T18:55:07.8230659Z 12: [moveit_task_constructor_core-test-move-relative-1] Fail in delete datareader, at ./src/rmw_service.cpp:89 during '__function__'
2024-05-26T18:55:07.8233380Z 12: [ERROR] [moveit_task_constructor_core-test-move-relative-1]: process has died [pid 5393, exit code -11, cmd '/home/runner/work/moveit_task_constructor/moveit_task_constructor/.work/target_ws/build/moveit_task_constructor_core/test/moveit_task_constructor_core-test-move-relative --ros-args --params-file /tmp/launch_params_ujuovdp5 --params-file /tmp/launch_params_htlg3d0o --params-file /tmp/launch_params_4g9zqee6 --params-file /tmp/launch_params_niamvcg5'].

Humble and Iron don’t exhibit this issue. WORKAROUND: Bypassing normal cleanup routines by calling quick_exit().

Required Info:

  • Operating System: Ubuntu 24.04
  • Installation type: deb package
  • Version or commit hash: 8.5.0
  • DDS implementation:fastrtps
  • Client library (if applicable): rclcpp

Steps to reproduce issue

git clone -b https://github.com/ros-planning/moveit_task_constructor.git src/mtc
(cd src/mtc; git checkout bb047c8)
docker run --rm -it -v $PWD/src:/root/ws_moveit/src  moveit/moveit2:rolling-source

in docker:

colcon build --mixin rel-with-deb-info
python3 -m launch_testing.launch_test /root/ws_moveit/src/mtc/core/test/test.launch.py test_binary:=/root/ws_moveit/build/moveit_task_constructor_core/test/moveit_task_constructor_core-test-move-relative

Expected behavior

Binary shuts down cleanly after tests.

Actual behavior

Binary throws error message and segfaults:

cannot publish data, at ./src/rmw_publish.cpp:66 during '__function__'
Fail in delete datareader, at ./src/rmw_service.cpp:89 during '__function__'

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (8 by maintainers)

Most upvoted comments

I have created a PR ros2/rclcpp#2556 to fix the issue in Executor.

For fastdds, when the program ends, fastdds resources start to release by Destructor. Note that the destruction of the static executor happens after this process. This results in an exception when the executor tries to access fastdds resources during its destruction.

...
    at /root/ws_moveit/src/Fast-DDS/src/cpp/fastdds/domain/DomainParticipantFactory.cpp:73
#10 0x00007ffff08a8cb4 in eprosima::fastdds::dds::DomainParticipantFactory::~DomainParticipantFactory (this=0x5555557a0680, __in_chrg=<optimized out>)
    at /root/ws_moveit/src/Fast-DDS/src/cpp/fastdds/domain/DomainParticipantFactory.cpp:86
#11 0x00007ffff08a8d4c in operator() (__closure=0x5555557a13f0, p=0x5555557a0680) at /root/ws_moveit/src/Fast-DDS/src/cpp/fastdds/domain/DomainParticipantFactory.cpp:100
#12 0x00007ffff08ad3e2 in std::_Sp_counted_deleter<eprosima::fastdds::dds::DomainParticipantFactory*, eprosima::fastdds::dds::DomainParticipantFactory::get_shared_instance()::<lambda(eprosima::fastdds::dds::DomainParticipantFactory*)>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose(void) (this=0x5555557a13e0) at /usr/include/c++/13/bits/shared_ptr_base.h:527
#13 0x00007ffff0586833 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x5555557a13e0) at /usr/include/c++/13/bits/shared_ptr_base.h:346
#14 0x00007ffff0586d83 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7ffff1393548 <eprosima::fastdds::dds::DomainParticipantFactory::get_shared_instance()::instance+8>, 
    __in_chrg=<optimized out>) at /usr/include/c++/13/bits/shared_ptr_base.h:1071
#15 0x00007ffff08aea5c in std::__shared_ptr<eprosima::fastdds::dds::DomainParticipantFactory, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (
    this=0x7ffff1393540 <eprosima::fastdds::dds::DomainParticipantFactory::get_shared_instance()::instance>, __in_chrg=<optimized out>) at /usr/include/c++/13/bits/shared_ptr_base.h:1524
#16 0x00007ffff08aea7c in std::shared_ptr<eprosima::fastdds::dds::DomainParticipantFactory>::~shared_ptr (
    this=0x7ffff1393540 <eprosima::fastdds::dds::DomainParticipantFactory::get_shared_instance()::instance>, __in_chrg=<optimized out>) at /usr/include/c++/13/bits/shared_ptr.h:175
#17 0x00007ffff628ea66 in __run_exit_handlers (status=0, listp=<optimized out>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:108
#18 0x00007ffff628ebae in __GI_exit (status=<optimized out>) at ./stdlib/exit.c:138
#19 0x00007ffff62711d1 in __libc_start_call_main (main=main@entry=0x555555578ae9 <main(int, char**)>, argc=argc@entry=10, argv=argv@entry=0x7fffffffbfd8) at ../sysdeps/nptl/libc_start_call_main.h:74
#20 0x00007ffff627128b in __libc_start_main_impl (main=0x555555578ae9 <main(int, char**)>, argc=10, argv=0x7fffffffbfd8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, 
    stack_end=0x7fffffffbfc8) at ../csu/libc-start.c:360
#21 0x0000555555577fd5 in _start ()

As for why cyclonedds doesn’t have this issue, I believe it’s because cyclonedds doesn’t release its resources when the program exits (fastdds use Destructor). Therefore, when the static executor is being destructed, it can still access those resources. These resources are eventually reclaimed by the operating system.

So I think this code should be changed. https://github.com/moveit/moveit_task_constructor/blob/bb047c894cd2a395a672abdd47ec76d709d1cb1e/core/src/introspection.cpp#L160

Adding rclcpp::shutdown() in the last test indeed solves the problem somehow.

I just wanted to quickly check an idea. Putting a shutdown in a test isn’t really appropriate.

I tried adding shutdown() before, but in main() (for better symmetry), which didn’t change anything!

Yes. This is a problem.

Why is it important where shutdown is called? I still consider this a bug in fastrtps.

It is used to gracefully shut down ROS 2 nodes and associated resources (The correct order of release).|

Calling shutdown() in the last test instead of main, means that the Node is still existing when shutdown is called. Maybe this gives a hint to the problem?

Yes, there are other reasons too. I’ll continue to investigate this.