rmw_zenoh: [Flaky] Exception in GraphListener thread: rcl_wait unexpectedly timed out
The same error log has been reported by several users in different contexts, using latest release, or commit
- in #455 - using v0.2.2 (release for jazzy)
2025-03-07T08:07:28.822902000Z [ERROR] [diagnostics_summarizer-29]: process has died [pid 662, exit code -6, cmd '/install/harvey/lib/harvey/diagnostics_summarizer --ros-args -r __node:=diagnostics_summarizer --params-file /tmp/launch_params_i1lgx4ix'].
2025-03-07T08:07:33.852035000Z [diagnostics_summarizer-29] [ERROR] [1741334848.118082488] [rclcpp]: caught std::exception exception in GraphListener thread: rcl_wait unexpectedly timed out
2025-03-07T08:07:33.882358000Z [diagnostics_summarizer-29] terminate called after throwing an instance of 'std::runtime_error'
2025-03-07T08:07:33.882532000Z [diagnostics_summarizer-29] what(): rcl_wait unexpectedly timed out
2025-03-07T08:07:33.884842000Z [diagnostics_summarizer-29] Stack trace (most recent call last) in thread 1024:
2025-03-07T08:07:33.891791000Z [diagnostics_summarizer-29] #11 Object "", at 0xffffffffffffffff, in
2025-03-07T08:07:33.891994000Z [diagnostics_summarizer-29] #10 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc25b62bc3b, in
2025-03-07T08:07:33.894063000Z [diagnostics_summarizer-29] #9 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc25b59eaa3, in
2025-03-07T08:07:33.895319000Z [diagnostics_summarizer-29] #8 Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33", at 0x7fc25b830db3, in
2025-03-07T08:07:33.922034000Z [diagnostics_summarizer-29] #7 Object "/opt/ros/jazzy/lib/librclcpp.so", at 0x7fc25bb96ab7, in
2025-03-07T08:07:33.928509000Z [diagnostics_summarizer-29] #6 Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33", at 0x7fc25b7ff0c0, in std::rethrow_exception(std::__exception_ptr::exception_ptr)
2025-03-07T08:07:33.934079000Z [diagnostics_summarizer-29] #5 Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33", at 0x7fc25b7e9a54, in std::terminate()
2025-03-07T08:07:33.945943000Z [diagnostics_summarizer-29] #4 Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33", at 0x7fc25b7ff0d9, in
2025-03-07T08:07:33.965063000Z [diagnostics_summarizer-29] #3 Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33", at 0x7fc25b7e9ff4, in
2025-03-07T08:07:33.970482000Z [diagnostics_summarizer-29] #2 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc25b52a8fe, in abort
2025-03-07T08:07:33.986060000Z [diagnostics_summarizer-29] #1 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc25b54727d, in raise
2025-03-07T08:07:34.030215000Z [diagnostics_summarizer-29] #0 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc25b5a0b2c, in pthread_kill
2025-03-07T08:07:34.064043000Z [diagnostics_summarizer-29] Aborted (Signal sent by tkill() 662 0)
- in #267 - using v0.2.2 (release for jazzy)
[foxglove_bridge-2] [ERROR] [1741672280.071278525] [rclcpp]: caught std::exception exception in GraphListener thread: rcl_wait unexpectedly timed out
[foxglove_bridge-2] terminate called after throwing an instance of 'std::runtime_error'
[foxglove_bridge-2] what(): rcl_wait unexpectedly timed out
[ERROR] [foxglove_bridge-2]: process has died [pid 77, exit code -6, cmd '/opt/ros/jazzy/lib/foxglove_bridge/foxglove_bridge --ros-args --log-level error --ros-args -r __node:=data_bridge -r __ns:=/groundstation --params-file /tmp/launch_params_veefrsbb'].
- via email - using jazzy branch, commit fe1a0146eb1f6465a56dd014077d8b0664ef1a84
[lifecycle_manager-2] [ERROR] [1741548982.344915581] [rclcpp]: caught std::exception exception in GraphListener thread: rcl_wait unexpectedly timed out
[lifecycle_manager-2] terminate called after throwing an instance of 'std::runtime_error'
[lifecycle_manager-2] what(): rcl_wait unexpectedly timed out
[lifecycle_manager-4] [ERROR] [1741548982.587766326] [rclcpp]: caught std::exception exception in GraphListener thread: rcl_wait unexpectedly timed out
[lifecycle_manager-4] terminate called after throwing an instance of 'std::runtime_error'
[lifecycle_manager-4] what(): rcl_wait unexpectedly timed out
[lifecycle_manager-18] [ERROR] [1741548983.299309956] [rclcpp]: caught std::exception exception in GraphListener thread: rcl_wait unexpectedly timed out
[lifecycle_manager-18] terminate called after throwing an instance of 'std::runtime_error'
[lifecycle_manager-18] what(): rcl_wait unexpectedly timed out
[ERROR] [lifecycle_manager-2]: process has died [pid 175258, exit code -6, cmd '/home/gd/xxxxx/g_ws/install/nav2_lifecycle_manager/lib/nav2_lifecycle_manager/lifecycle_manager --ros-args -r __node:=lifecycle_manager_map_server_amcl --params-file /tmp/launch_params_d_t5atuk --params-file /tmp/launch_params_51hi__lq --params-file /tmp/launch_params_n239xeyi --params-file /tmp/launch_params_yb0p02ir --params-file /tmp/launch_params_h316r13z'].
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 28 (18 by maintainers)
@Yadunund I’ll do one this week.
Based on @JEnoch analysis I’ve prepared https://github.com/ros2/rmw_zenoh/pull/575. The PR seems to solve the flaky test on the same VM used by @JEnoch to run the tests.
I believe https://github.com/ros2/rmw_zenoh/pull/528 provides a fix to the problem. @evshary could you validate the validate on
nav2_system_tests?I confirm that this test does not give this error after #528. In the next few days I will do more tests with the code of my project.
I am also experiencing the same issue even with #528.
The
rcl_waiterror disappeared on my side. The failure in the log is caused by another issue.stdout_stderr.log
Looking at the code:
GraphListener::run_loop()callsrcl_wait(&wait_set_, -1)- i.e. with an infinite timeout.rcl_wait()converts-1into aINT64_MAXnanoseconds timeout, and callsrmw_wait()with it.rmw_zenoh_cppthermw_wait()implementation callsstd::condition_variable::wait_for()with this “INT64_MAX” timeout and a predicate onwait_set_data->triggered.wait_set_data->triggeredis set totrue, which is supposed to be done only after the insertion of an event in one of the queues for subscriptions, guard_conditions, services, clients or events.RMW_RET_TIMEOUTif all the queues are empty.So the issue seems to be some race condition between the insertion of an event, the setting of
wait_set_data->triggeredto true and the wakeup of this condition_variable and the check of for a new event.