rmw_zenoh: [Flaky] Exception in GraphListener thread: rcl_wait unexpectedly timed out

The same error log has been reported by several users in different contexts, using latest release, or commit

  1. in #455 - using v0.2.2 (release for jazzy)
2025-03-07T08:07:28.822902000Z [ERROR] [diagnostics_summarizer-29]: process has died [pid 662, exit code -6, cmd '/install/harvey/lib/harvey/diagnostics_summarizer --ros-args -r __node:=diagnostics_summarizer --params-file /tmp/launch_params_i1lgx4ix'].
2025-03-07T08:07:33.852035000Z [diagnostics_summarizer-29] [ERROR] [1741334848.118082488] [rclcpp]: caught std::exception exception in GraphListener thread: rcl_wait unexpectedly timed out
2025-03-07T08:07:33.882358000Z [diagnostics_summarizer-29] terminate called after throwing an instance of 'std::runtime_error'
2025-03-07T08:07:33.882532000Z [diagnostics_summarizer-29]   what():  rcl_wait unexpectedly timed out
2025-03-07T08:07:33.884842000Z [diagnostics_summarizer-29] Stack trace (most recent call last) in thread 1024:
2025-03-07T08:07:33.891791000Z [diagnostics_summarizer-29] #11   Object "", at 0xffffffffffffffff, in 
2025-03-07T08:07:33.891994000Z [diagnostics_summarizer-29] #10   Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc25b62bc3b, in 
2025-03-07T08:07:33.894063000Z [diagnostics_summarizer-29] #9    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc25b59eaa3, in 
2025-03-07T08:07:33.895319000Z [diagnostics_summarizer-29] #8    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33", at 0x7fc25b830db3, in 
2025-03-07T08:07:33.922034000Z [diagnostics_summarizer-29] #7    Object "/opt/ros/jazzy/lib/librclcpp.so", at 0x7fc25bb96ab7, in 
2025-03-07T08:07:33.928509000Z [diagnostics_summarizer-29] #6    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33", at 0x7fc25b7ff0c0, in std::rethrow_exception(std::__exception_ptr::exception_ptr)
2025-03-07T08:07:33.934079000Z [diagnostics_summarizer-29] #5    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33", at 0x7fc25b7e9a54, in std::terminate()
2025-03-07T08:07:33.945943000Z [diagnostics_summarizer-29] #4    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33", at 0x7fc25b7ff0d9, in 
2025-03-07T08:07:33.965063000Z [diagnostics_summarizer-29] #3    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33", at 0x7fc25b7e9ff4, in 
2025-03-07T08:07:33.970482000Z [diagnostics_summarizer-29] #2    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc25b52a8fe, in abort
2025-03-07T08:07:33.986060000Z [diagnostics_summarizer-29] #1    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc25b54727d, in raise
2025-03-07T08:07:34.030215000Z [diagnostics_summarizer-29] #0    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc25b5a0b2c, in pthread_kill
2025-03-07T08:07:34.064043000Z [diagnostics_summarizer-29] Aborted (Signal sent by tkill() 662 0)
  1. in #267 - using v0.2.2 (release for jazzy)
[foxglove_bridge-2] [ERROR] [1741672280.071278525] [rclcpp]: caught std::exception exception in GraphListener thread: rcl_wait unexpectedly timed out
[foxglove_bridge-2] terminate called after throwing an instance of 'std::runtime_error'
[foxglove_bridge-2]   what():  rcl_wait unexpectedly timed out
[ERROR] [foxglove_bridge-2]: process has died [pid 77, exit code -6, cmd '/opt/ros/jazzy/lib/foxglove_bridge/foxglove_bridge --ros-args --log-level error --ros-args -r __node:=data_bridge -r __ns:=/groundstation --params-file /tmp/launch_params_veefrsbb'].
  1. via email - using jazzy branch, commit fe1a0146eb1f6465a56dd014077d8b0664ef1a84
[lifecycle_manager-2] [ERROR] [1741548982.344915581] [rclcpp]: caught std::exception exception in GraphListener thread: rcl_wait unexpectedly timed out
[lifecycle_manager-2] terminate called after throwing an instance of 'std::runtime_error'
[lifecycle_manager-2]   what():  rcl_wait unexpectedly timed out
[lifecycle_manager-4] [ERROR] [1741548982.587766326] [rclcpp]: caught std::exception exception in GraphListener thread: rcl_wait unexpectedly timed out
[lifecycle_manager-4] terminate called after throwing an instance of 'std::runtime_error'
[lifecycle_manager-4]   what():  rcl_wait unexpectedly timed out
[lifecycle_manager-18] [ERROR] [1741548983.299309956] [rclcpp]: caught std::exception exception in GraphListener thread: rcl_wait unexpectedly timed out
[lifecycle_manager-18] terminate called after throwing an instance of 'std::runtime_error'
[lifecycle_manager-18]   what():  rcl_wait unexpectedly timed out
[ERROR] [lifecycle_manager-2]: process has died [pid 175258, exit code -6, cmd '/home/gd/xxxxx/g_ws/install/nav2_lifecycle_manager/lib/nav2_lifecycle_manager/lifecycle_manager --ros-args -r __node:=lifecycle_manager_map_server_amcl --params-file /tmp/launch_params_d_t5atuk --params-file /tmp/launch_params_51hi__lq --params-file /tmp/launch_params_n239xeyi --params-file /tmp/launch_params_yb0p02ir --params-file /tmp/launch_params_h316r13z'].

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 28 (18 by maintainers)

Most upvoted comments

@Yadunund I’ll do one this week.

Based on @JEnoch analysis I’ve prepared https://github.com/ros2/rmw_zenoh/pull/575. The PR seems to solve the flaky test on the same VM used by @JEnoch to run the tests.

I believe https://github.com/ros2/rmw_zenoh/pull/528 provides a fix to the problem. @evshary could you validate the validate on nav2_system_tests?

If I run

colcon test --packages-select nav2_system_tests --ctest-args -R test_waypoint_follower --event-handlers console_direct+

on my side I get the same error as @evshary (see #527 (comment)).

I.e. I do NOT see any timed out error

[rclcpp]: caught std::exception exception in GraphListener thread: rcl_wait unexpectedly timed out

That error was solved by #528 on my side and on the CI.

Did you manage to trigger the rcl_wait unexpectedly timed out error in any other way?

I confirm that this test does not give this error after #528. In the next few days I will do more tests with the code of my project.

I am also experiencing the same issue even with #528.

I believe #528 provides a fix to the problem. @evshary could you validate the validate on nav2_system_tests?

The rcl_wait error disappeared on my side. The failure in the log is caused by another issue.

stdout_stderr.log

Looking at the code:

  • the GraphListener::run_loop() calls rcl_wait(&wait_set_, -1) - i.e. with an infinite timeout.
  • rcl_wait() converts -1 into a INT64_MAX nanoseconds timeout, and calls rmw_wait() with it.
  • In rmw_zenoh_cpp the rmw_wait() implementation calls std::condition_variable::wait_for() with this “INT64_MAX” timeout and a predicate on wait_set_data->triggered.
  • Thus this returns only when wait_set_data->triggered is set to true, which is supposed to be done only after the insertion of an event in one of the queues for subscriptions, guard_conditions, services, clients or events.
  • Then the code checks those queues and returns RMW_RET_TIMEOUT if all the queues are empty.

So the issue seems to be some race condition between the insertion of an event, the setting of wait_set_data->triggered to true and the wakeup of this condition_variable and the check of for a new event.