rclcpp: rclcpp::exceptions::RCLError / rcl_wait() failed: error not set

Bug report

Required Info:

• Operating System:
  • Ubuntu 22.04
• Installation type:
  • bin deb
• Version or commit hash:
  • last Humble sync
• DDS implementation:
  • Cyclone + localhost_only
• Client library (if applicable):
  • rclcpp

Don’t know if it is rclcpp or something nav2 specific, already posted here: https://github.com/ros-planning/navigation2/issues/3097

I get an rclcpp::exceptions::RCLError / rcl_wait() failed: error not set error randomly during endurance testing on a real robot on amcl, controller_server and some other navigation node. All were working fine under galactic. Struggling to understand the origin of it.

Jul 27 02:14:09 bash[813572]: [controller_server-66] terminate called after throwing an instance of ‘rclcpp::exceptions::RCLError’ Jul 27 02:14:09 bash[813572]: [controller_server-66] what(): rcl_wait() failed: error not set Jul 27 02:14:09 bash[813572]: [controller_server-66] Stack trace (most recent call last) in thread 816228: Jul 27 02:14:09 bash[813572]: [controller_server-66] https://github.com/ros-planning/navigation2/issues/14 Object “”, at 0xffffffffffffffff, in Jul 27 02:14:09 bash[813572]: [controller_server-66] https://github.com/ros-planning/navigation2/pull/13 Object “/usr/lib/x86_64-linux-gnu/libc.so.6”, at 0x7fc4eee369ff, in Jul 27 02:14:10 bash[813572]: [controller_server-66] https://github.com/ros-planning/navigation2/pull/12 Object “/usr/lib/x86_64-linux-gnu/libc.so.6”, at 0x7fc4eeda4b42, in Jul 27 02:14:10 bash[813572]: [controller_server-66] https://github.com/ros-planning/navigation2/issues/11 Object “/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30”, at 0x7fc4ef0342c2, in Jul 27 02:14:10 bash[813572]: [controller_server-66] https://github.com/ros-planning/navigation2/pull/10 Object “/opt/ros/humble/lib/librclcpp.so”, at 0x7fc4ef44dc80, in rclcpp::executors::SingleThreadedExecutor::spin() Jul 27 02:14:10 bash[813572]: [controller_server-66] https://github.com/ros-planning/navigation2/pull/9 Object “/opt/ros/humble/lib/librclcpp.so”, at 0x7fc4ef445972, in rclcpp::Executor::get_next_executable(rclcpp::AnyExecutable&, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) Jul 27 02:14:10 bash[813572]: [controller_server-66] https://github.com/ros-planning/navigation2/pull/8 Object “/opt/ros/humble/lib/librclcpp.so”, at 0x7fc4ef4435cd, in rclcpp::Executor::wait_for_work(std::chrono::duration<long, std::ratio<1l, 1000000000l> >) Jul 27 02:14:10 bash[813572]: [controller_server-66] https://github.com/ros-planning/navigation2/pull/7 Object “/opt/ros/humble/lib/librclcpp.so”, at 0x7fc4ef43f9c8, in rclcpp::exceptions::throw_from_rcl_error(int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, rcutils_error_state_s const*, void (*)()) Jul 27 02:14:10 bash[813572]: [controller_server-66] https://github.com/ros-planning/navigation2/issues/6 Object “/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30”, at 0x7fc4ef00627d, in std::rethrow_exception(std::__exception_ptr::exception_ptr) Jul 27 02:14:10 bash[813572]: [controller_server-66] https://github.com/ros-planning/navigation2/pull/5 Object “/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30”, at 0x7fc4ef0062f6, in std::terminate() Jul 27 02:14:10 bash[813572]: [controller_server-66] https://github.com/ros-planning/navigation2/pull/4 Object “/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30”, at 0x7fc4ef00628b, in Jul 27 02:14:10 bash[813572]: [controller_server-66] https://github.com/ros-planning/navigation2/issues/3 Object “/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30”, at 0x7fc4eeffabfd, in Jul 27 02:14:10 bash[813572]: [controller_server-66] https://github.com/ros-planning/navigation2/pull/2 Object “/usr/lib/x86_64-linux-gnu/libc.so.6”, at 0x7fc4eed387f2, in abort Jul 27 02:14:10 bash[813572]: [controller_server-66] https://github.com/ros-planning/navigation2/pull/1 Object “/usr/lib/x86_64-linux-gnu/libc.so.6”, at 0x7fc4eed52475, in raise Jul 27 02:14:10 bash[813572]: [controller_server-66] #0 Object “/usr/lib/x86_64-linux-gnu/libc.so.6”, at 0x7fc4eeda6a7c, in pthread_kill Jul 27 02:14:10 bash[813572]: [controller_server-66] Aborted (Signal sent by tkill() 814467 1000) Jul 27 02:14:10 bash[813572]: [ERROR] [controller_server-66]: process has died [pid 814467, exit code -6, cmd ‘/opt/wyca/elodie/nav2_controller/lib/nav2_controller/controller_server --ros-args --params-file /home/wyca/.wyca/nav2_params/controller_server_params.yaml --params-file /home/wyca/.wyca/nav2_params/local_costmap_params.yaml --params-file /home/wyca/.wyca/nav2_params/costmaps_plugins_params.yaml --params-file /tmp/launch_params_est2liq2 -r /cmd_vel:=/twist_mux_root/twist_nav’].