rmw_zenoh: Hang loading components

I am experience a hang while loading many components with rmw_zenoh.

Terminal 1:

$ ros2 run rmw_zenoh_cpp rmw_zenohd

Terminal 2:

$ export RMW_IMPLEMENTATION=rmw_zenoh_cpp
$ ros2 run rclcpp_components component_container

Terminal 3:

$ export RMW_IMPLEMENTATION=rmw_zenoh_cpp
$ for i in {1..100}; do ros2 component load --node-namespace /talker${i} /ComponentManager composition composition::Talker; done
...
[INFO] [1716659586.681864543] [rmw_zenoh_cpp]: Successfully connected to a Zenoh router with id 7d769066d659797f52ea9cc8c559737.
Loaded component 76 into '/ComponentManager' container node as '/talker79/talker'
[INFO] [1716659589.848647783] [rmw_zenoh_cpp]: Successfully connected to a Zenoh router with id 7d769066d659797f52ea9cc8c559737.
Loaded component 77 into '/ComponentManager' container node as '/talker80/talker'
[INFO] [1716659593.097119225] [rmw_zenoh_cpp]: Successfully connected to a Zenoh router with id 7d769066d659797f52ea9cc8c559737.
Loaded component 78 into '/ComponentManager' container node as '/talker81/talker'
[INFO] [1716659596.418323636] [rmw_zenoh_cpp]: Successfully connected to a Zenoh router with id 7d769066d659797f52ea9cc8c559737.

I would expect the for loop above to eventually finish but it does not.

Let’s attach to the process and investigate:

$ sudo gdb attach -p `pgrep -f talker82`
...
(gdb) info threads
  Id   Target Id                                      Frame 
* 1    Thread 0x7f13abacd740 (LWP 1649280) "ros2"     syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  2    Thread 0x7f13a9d2c640 (LWP 1649281) "ros2-ust" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  3    Thread 0x7f13a952b640 (LWP 1649282) "ros2-ust" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  4    Thread 0x7f13a104e640 (LWP 1649283) "ros2"     __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f13aabeb280 <(anonymous namespace)::g_signal_handler_sem>) at ./nptl/futex-internal.c:57
  5    Thread 0x7f13a084d640 (LWP 1649284) "app-0"    0x00007f13abbf5e2e in epoll_wait (epfd=3, events=0x556946c61670, maxevents=1024, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  6    Thread 0x7f13a064c640 (LWP 1649285) "net-0"    0x00007f13abbf5e2e in epoll_wait (epfd=8, events=0x556946c64680, maxevents=1024, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  7    Thread 0x7f13a044b640 (LWP 1649286) "acc-0"    0x00007f13abbf5e2e in epoll_wait (epfd=12, events=0x556946c93120, maxevents=1024, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  8    Thread 0x7f138ffff640 (LWP 1649288) "tx-0"     0x00007f13abbf5e2e in epoll_wait (epfd=16, events=0x7f139000a5e0, maxevents=1024, timeout=462) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  9    Thread 0x7f138fdfe640 (LWP 1649289) "rx-0"     futex_wait (private=0, expected=2, futex_word=0x7f13aabeaa60 <rclpy::LoggingGuard::logging_mutex_>) at ../sysdeps/nptl/futex-internal.h:146
  10   Thread 0x7f138fbfd640 (LWP 1649290) "rx-1"     syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  11   Thread 0x7f138f9fc640 (LWP 1649291) "ros2"     __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f138f9fbde0, op=137, expected=0, futex_word=0x556946f65780) at ./nptl/futex-internal.c:57
...
(gdb) thread 1
[Switching to thread 1 (Thread 0x7f13abacd740 (LWP 1649280))]
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
38  in ../sysdeps/unix/sysv/linux/x86_64/syscall.S
(gdb) bt
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f13a26a66b4 in std::sys::pal::unix::futex::futex_wait () at library/std/src/sys/pal/unix/futex.rs:62
#2  std::sys::sync::mutex::futex::Mutex::lock_contended () at library/std/src/sys/sync/mutex/futex.rs:68
#3  0x00007f13a29b3ba5 in std::sys::sync::mutex::futex::Mutex::lock () at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys/sync/mutex/futex.rs:40
#4  std::sync::mutex::Mutex<alloc::boxed::Box<(dyn zenoh::net::routing::hat::HatTrait + core::marker::Send + core::marker::Sync), alloc::alloc::Global>>::lock<alloc::boxed::Box<(dyn zenoh::net::routing::hat::HatTrait + core::marker::Send + core::marker::Sync), alloc::alloc::Global>> (self=0x556946c7e120)
    at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sync/mutex.rs:317
#5  0x00007f13a29c4509 in zenoh::net::routing::dispatcher::face::{impl#4}::send_declare (self=0x5569468e6750, msg=...) at src/net/routing/dispatcher/face.rs:189
#6  0x00007f13a2a3dc20 in zenoh::session::Session::declare_queryable_inner (self=0x5569466acfb0, key_expr=0x7fff48bb4490, complete=true, origin=zenoh::sample::Locality::Any, callback=...) at src/session.rs:1249
#7  0x00007f13a280b6c6 in zenoh::queryable::{impl#20}::res_sync<zenoh::handlers::DefaultHandler> (self=...) at <***removed***>/.cargo/git/checkouts/zenoh-cc237f2570fab813/763a05f/zenoh/src/queryable.rs:632
#8  0x00007f13a28023c2 in zenoh_ext::publication_cache::PublicationCache::new (conf=...) at src/publication_cache.rs:160
#9  0x00007f13a28012dd in zenoh_ext::publication_cache::{impl#2}::res_sync (self=<error reading variable: Cannot access memory at address 0x89>) at src/publication_cache.rs:99
#10 0x00007f13a2779c0b in zenohcd::publication_cache::ze_declare_publication_cache (session=..., keyexpr=..., options=...) at <***removed***>/ws_rmw_zenoh/build/zenoh_c_vendor/zenoh_c_vendor-prefix/src/zenoh_c_vendor/src/publication_cache.rs:136
#11 0x00007f13a8ccfa7d in rmw_create_publisher (node=0x556946f65c40, type_supports=0x7f13a9f3eb80, topic_name=0x556946f67630 "/rosout", qos_profile=0x7fff48bb9b30, publisher_options=0x7fff48bb9bb0) at <***removed***>/ws_rmw_zenoh/src/rmw_zenoh/rmw_zenoh_cpp/src/rmw_zenoh.cpp:586
#12 0x00007f13aa26512c in rcl_publisher_init (publisher=0x7fff48bb9b28, node=0x556946f65120, type_support=0x7f13a9f3eb80, topic_name=0x7f13aa283c93 "/rosout", options=0x7fff48bb9b30) at <***removed***>/ws_rmw_zenoh/src/rcl/rcl/src/rcl/publisher.c:111
#13 0x00007f13aa25e0bb in rcl_logging_rosout_init_publisher_for_node (node=0x556946f65120) at <***removed***>/ws_rmw_zenoh/src/rcl/rcl/src/rcl/logging_rosout.c:276
#14 0x00007f13aa261c93 in rcl_node_init (node=0x556946f65120, name=0x556946f5b0a0 "_ros2cli_1649280", namespace_=0x7fff48bc0b48 "", context=0x556946cabbb0, options=0x7fff48bc07e0) at <***removed***>/ws_rmw_zenoh/src/rcl/rcl/src/rcl/node.c:297
#15 0x00007f13aaa22cec in rclpy::Node::Node (this=0x556946b6b170, node_name=0x556946f5b0a0 "_ros2cli_1649280", namespace_=0x7fff48bc0b48 "", context=..., pycli_args=..., use_global_arguments=true, enable_rosout=true) at <***removed***>/ws_rmw_zenoh/src/rclpy/rclpy/src/rclpy/node.cpp:464
#16 0x00007f13aaa2b7f5 in pybind11::detail::initimpl::construct_or_initialize<rclpy::Node, char const*, char const*, rclpy::Context&, pybind11::object, bool, bool, 0> () at /usr/include/pybind11/detail/init.h:61
#17 0x00007f13aaa299cb in pybind11::detail::initimpl::constructor<char const*, char const*, rclpy::Context&, pybind11::object, bool, bool>::execute<pybind11::class_<rclpy::Node, rclpy::Destroyable, std::shared_ptr<rclpy::Node> >, , 0>(pybind11::class_<rclpy::Node, rclpy::Destroyable, std::shared_ptr<rclpy::Node> >&)::{lambda(pybind11::detail::value_and_holder&, char const*, char const*, rclpy::Context&, pybind11::object, bool, bool)#1}::operator()(pybind11::detail::value_and_holder&, char const*, char const*, rclpy::Context&, pybind11::object, bool, bool) const (__closure=0x5569467ced28, v_h=..., args#0=0x556946f5b0a0 "_ros2cli_1649280", 
    args#1=0x7fff48bc0b48 "", args#2=..., args#3=..., args#4=true, args#5=true) at /usr/include/pybind11/detail/init.h:178
...

We can see from frame 5 in thread 1 that it wants to acquire the ctrl_lock: https://github.com/eclipse-zenoh/zenoh/blob/763a05f0d61a894cf3cbeb07223e2ea726777325/zenoh/src/net/routing/dispatcher/face.rs#L189

Another thread must be holding that lock:

(gdb) thread 9
[Switching to thread 9 (Thread 0x7f138fdfe640 (LWP 1649289))]
#0  futex_wait (private=0, expected=2, futex_word=0x7f13aabeaa60 <rclpy::LoggingGuard::logging_mutex_>) at ../sysdeps/nptl/futex-internal.h:146
146 ../sysdeps/nptl/futex-internal.h: No such file or directory.
(gdb) bt
#0  futex_wait (private=0, expected=2, futex_word=0x7f13aabeaa60 <rclpy::LoggingGuard::logging_mutex_>) at ../sysdeps/nptl/futex-internal.h:146
#1  __GI___lll_lock_wait (futex=futex@entry=0x7f13aabeaa60 <rclpy::LoggingGuard::logging_mutex_>, private=0) at ./nptl/lowlevellock.c:49
#2  0x00007f13abb6805d in lll_mutex_lock_optimized (mutex=0x7f13aabeaa60 <rclpy::LoggingGuard::logging_mutex_>) at ./nptl/pthread_mutex_lock.c:48
#3  ___pthread_mutex_lock (mutex=0x7f13aabeaa60 <rclpy::LoggingGuard::logging_mutex_>) at ./nptl/pthread_mutex_lock.c:128
#4  0x00007f13aaa1c75c in __gthread_mutex_lock (__mutex=0x7f13aabeaa60 <rclpy::LoggingGuard::logging_mutex_>) at /usr/include/x86_64-linux-gnu/c++/11/bits/gthr-default.h:749
#5  0x00007f13aaa1c77d in __gthread_recursive_mutex_lock (__mutex=0x7f13aabeaa60 <rclpy::LoggingGuard::logging_mutex_>) at /usr/include/x86_64-linux-gnu/c++/11/bits/gthr-default.h:811
#6  0x00007f13aaa1c79c in std::recursive_mutex::lock (this=0x7f13aabeaa60 <rclpy::LoggingGuard::logging_mutex_>) at /usr/include/c++/11/mutex:108
#7  0x00007f13aaa1c7e0 in std::lock_guard<std::recursive_mutex>::lock_guard (this=0x7f138fdf0408, __m=...) at /usr/include/c++/11/bits/std_mutex.h:229
#8  0x00007f13aaa1b742 in rclpy::LoggingGuard::LoggingGuard (this=0x7f138fdf0408) at <***removed***>/ws_rmw_zenoh/src/rclpy/rclpy/src/rclpy/logging.cpp:33
#9  0x00007f13aaa1b786 in rclpy::rclpy_thread_safe_logging_output_handler (location=0x7f13a8d29ed0 <rmw_zenoh_cpp::GraphCache::parse_del(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)::__rcutils_logging_location>, severity=30, name=0x7f13a8ceb6c5 "rmw_zenoh_cpp", 
    timestamp=1716659597585131569, format=0x7f13a8cebf30 "Received liveliness token to remove node /%s from the graph before all pub/subs/clients/services for this node have been removed. Removing all entities first...", args=0x7f138fdf04c0)
    at <***removed***>/ws_rmw_zenoh/src/rclpy/rclpy/src/rclpy/logging.cpp:51
#10 0x00007f13aa224d9c in vrcutils_log_internal (location=0x7f13a8d29ed0 <rmw_zenoh_cpp::GraphCache::parse_del(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)::__rcutils_logging_location>, severity=30, name=0x7f13a8ceb6c5 "rmw_zenoh_cpp", 
    format=0x7f13a8cebf30 "Received liveliness token to remove node /%s from the graph before all pub/subs/clients/services for this node have been removed. Removing all entities first...", args=0x7f138fdf04c0) at <***removed***>/ws_rmw_zenoh/src/rcutils/src/logging.c:1095
#11 0x00007f13aa224f7c in rcutils_log_internal (location=0x7f13a8d29ed0 <rmw_zenoh_cpp::GraphCache::parse_del(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)::__rcutils_logging_location>, severity=30, name=0x7f13a8ceb6c5 "rmw_zenoh_cpp", 
    format=0x7f13a8cebf30 "Received liveliness token to remove node /%s from the graph before all pub/subs/clients/services for this node have been removed. Removing all entities first...") at <***removed***>/ws_rmw_zenoh/src/rcutils/src/logging.c:1119
#12 0x00007f13a8c80fe4 in rmw_zenoh_cpp::GraphCache::parse_del (this=0x556946a04f70, keyexpr="@ros2_lv/0/b89f34a8f6a786788926e9c41d11be8e/295/295/NN/%talker25/talker", ignore_from_current_session=false) at <***removed***>/ws_rmw_zenoh/src/rmw_zenoh/rmw_zenoh_cpp/src/detail/graph_cache.cpp:603
#13 0x00007f13a8cc87fe in graph_sub_data_handler (sample=0x7f138fdf1a70, data=0x556946bbe550) at <***removed***>/ws_rmw_zenoh/src/rmw_zenoh/rmw_zenoh_cpp/src/rmw_init.cpp:73
#14 0x00007f13a26acead in zenohcd::closures::sample_closure::z_closure_sample_call (closure=0x556946d08ab0, sample=0x7f138fdf1a70) at <***removed***>/ws_rmw_zenoh/build/zenoh_c_vendor/zenoh_c_vendor-prefix/src/zenoh_c_vendor/src/closures/sample_closure.rs:52
#15 0x00007f13a2785be1 in zenohcd::liveliness::zc_liveliness_declare_subscriber::{closure#0} (sample=...) at <***removed***>/ws_rmw_zenoh/build/zenoh_c_vendor/zenoh_c_vendor-prefix/src/zenoh_c_vendor/src/liveliness.rs:186
#16 0x00007f13a2a426c8 in zenoh::session::Session::handle_data (self=0x556946bf5ed0, local=false, key_expr=0x7f138fdf52d0, info=..., payload=..., attachment=...) at src/session.rs:1712
#17 0x00007f13a2a4ce44 in zenoh::session::{impl#13}::send_declare (self=0x556946bf5ed0, msg=...) at src/session.rs:2195
#18 0x00007f13a2a58414 in zenoh::session::{impl#16}::send_declare (self=0x556946bf5ed0, ctx=...) at src/session.rs:2701
#19 0x00007f13a2a19fdf in zenoh::net::routing::hat::p2p_peer::pubsub::propagate_forget_simple_subscription (tables=0x556946c7e040, res=0x7f138fdf7548) at src/net/routing/hat/p2p_peer/pubsub.rs:175
#20 0x00007f13a2a1a8b1 in zenoh::net::routing::hat::p2p_peer::pubsub::undeclare_client_subscription (tables=0x556946c7e040, face=0x7f138fdf8028, res=0x7f138fdf7548) at src/net/routing/hat/p2p_peer/pubsub.rs:205
#21 0x00007f13a2a1aeb7 in zenoh::net::routing::hat::p2p_peer::pubsub::forget_client_subscription (tables=0x556946c7e040, face=0x7f138fdf8028, res=0x7f138fdf7548) at src/net/routing/hat/p2p_peer/pubsub.rs:236
#22 0x00007f13a2b1e7dd in zenoh::net::routing::hat::p2p_peer::pubsub::{impl#0}::undeclare_subscription (self=0x1, tables=0x556946c7e040, face=0x7f138fdf8028, res=0x7f138fdf7548, _node_id=0) at src/net/routing/hat/p2p_peer/pubsub.rs:275
#23 0x00007f13a2b60bd5 in zenoh::net::routing::dispatcher::pubsub::undeclare_subscription (hat_code=..., tables=0x556946c7e030, face=0x7f138fdf8028, expr=0x7f138fdf8000, node_id=0) at src/net/routing/dispatcher/pubsub.rs:108
#24 0x00007f13a29c4da9 in zenoh::net::routing::dispatcher::face::{impl#4}::send_declare (self=0x7f1390003bd0, msg=...) at src/net/routing/dispatcher/face.rs:208
#25 0x00007f13a29c25fe in zenoh::net::primitives::demux::{impl#1}::handle_message (self=0x7f1390003bd0, msg=...) at src/net/primitives/demux.rs:69
#26 0x00007f13a2b3267e in zenoh::net::runtime::{impl#6}::handle_message (self=0x7f1390009e10, msg=...) at src/net/runtime/mod.rs:385
#27 0x00007f13a2d41a79 in zenoh_transport::unicast::universal::transport::TransportUnicastUniversal::trigger_callback (self=0x7f1390136cc0, callback=..., msg=...) at src/unicast/universal/rx.rs:49
#28 0x00007f13a2d42ae3 in zenoh_transport::unicast::universal::transport::TransportUnicastUniversal::handle_frame (self=0x7f1390136cc0, frame=...) at src/unicast/universal/rx.rs:100
#29 0x00007f13a2d45b21 in zenoh_transport::unicast::universal::transport::TransportUnicastUniversal::read_messages (self=0x7f1390136cc0, batch=..., link=0x7f1390136d88) at src/unicast/universal/rx.rs:204
#30 0x00007f13a2ce826e in zenoh_transport::unicast::universal::link::rx_task::{async_fn#0} () at src/unicast/universal/link.rs:266
#31 0x00007f13a2ce4ba5 in zenoh_transport::unicast::universal::link::{impl#0}::start_rx::{async_block#0} () at src/unicast/universal/link.rs:124
#32 0x00007f13a2d50a94 in tokio_util::task::task_tracker::{impl#8}::poll<zenoh_transport::unicast::universal::link::{impl#0}::start_rx::{async_block_env#0}> (self=..., cx=0x7f138fdfc5a8) at <***removed***>/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-util-0.7.10/src/task/task_tracker.rs:669
...

Frame 24 on thread 9 indicates the thread should have acquired the ctrl_lock earlier in the function if it is calling undeclare_subscription(): https://github.com/eclipse-zenoh/zenoh/blob/763a05f0d61a894cf3cbeb07223e2ea726777325/zenoh/src/net/routing/dispatcher/face.rs#L208

This means that thread 9 is holding the ctrl_lock that thread 1 is waiting on.

Thread 9 is also waiting:

(gdb) thread 9
[Switching to thread 9 (Thread 0x7f138fdfe640 (LWP 1649289))]
#0  futex_wait (private=0, expected=2, futex_word=0x7f13aabeaa60 <rclpy::LoggingGuard::logging_mutex_>) at ../sysdeps/nptl/futex-internal.h:146
146 in ../sysdeps/nptl/futex-internal.h
(gdb) up 3
#3  ___pthread_mutex_lock (mutex=0x7f13aabeaa60 <rclpy::LoggingGuard::logging_mutex_>) at ./nptl/pthread_mutex_lock.c:128
128 ./nptl/pthread_mutex_lock.c: No such file or directory.
(gdb) print mutex.__data.__owner
$1 = 1649280
(gdb) thread find 1649280
Thread 1 has target id 'Thread 0x7f13abacd740 (LWP 1649280)'

Thread 9 is waiting on some resource held by thread 1.

Frame 9 on thread 9 wants to acquire logging_mutex_ in rclpy_thread_safe_logging_output_handler(): https://github.com/ros2/rclpy/blob/52f2a69325721e1368ac3176adcc348e1f53f93e/rclpy/src/rclpy/logging.cpp#L51

Frame 15 on thread 1 should have acquired logging_mutex_ right before rcl_node_init(): https://github.com/ros2/rclpy/blob/52f2a69325721e1368ac3176adcc348e1f53f93e/rclpy/src/rclpy/node.cpp#L464

This means that thread 1 is holding the logging_mutex_ that thread 9 is waiting on.

Hopefully I have read the situation correctly here and, at the very least, others are able to reproduce the problem. I will continue this investigation after the US holiday weekend.

ROS Distribution: Iron OS: Ubuntu Jammy rmw_zenohd: commit 1ac6bd2ae1d3a9f741167bb2c771e42bea50229f. rust: rustc 1.78.0 (9b00956e5 2024-04-29)

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 27 (14 by maintainers)

Most upvoted comments

I can see the value of a RMW safe replacement (or fix) to the RCUTILS_LOG_*_NAMED macros.

As it stands, that is probably not feasible. The way the RCUTILS macros work is with a global logging callback. Even if we did have some way to switch back and forth between the “console-only” and the “client-library” installed callback (which we currently don’t), I don’t see a good way we could do it in a thread-safe manner.

That said, we could add a new set of logging APIs to rcutils specifically for the RMW layer, that would only ever use the console. That would have the benefit of being able to reuse a bunch of the code that we use for the “full-featured” rcutils logging.

For now, I think the easiest thing to do (and backport) is to fix this locally in rmw_zenoh_cpp, and then consider one of these more generic solutions later.

#8  0x00007f90e1c28c0b in rclcpp_logging_output_handler (location=0x7f90df793f10 <rmw_zenoh_cpp::GraphCache::parse_del(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)::__rcutils_logging_location>, severity=30, name=0x7f90df7596ed "rmw_zenoh_cpp", timestamp=1718793033330027250, 

@clalancette and I took a closer look at the stack trace from https://github.com/ros2/rmw_zenoh/issues/182#issuecomment-2178911481 and discovered that it is indeed an ABBA deadlock resulting from the two threads trying to acquire both graph_mutex_ and the scoped_logging_guard. 🤦🏼

The problem arises from the RCUTILS_LOG_*_NAMED macros. By default vrcutils_log_internal logs to stdout/err via g_rcutils_logging_output_handler but this hander is overwritten via rcutils_logging_set_output_handler in rclcpp and rclpy to also include publishing to /rosout.

I’m working on a fix where we rely on our own logging functions in rmw_zenoh that prints to stderr/stdout.

The deadlock in rmw_wait will be addressed separately.

Thanks @hwoithe. Will keep digging…

@Yadunund done! #200 is ready for review and merge.

@JEnoch I am now able to make more progress with https://github.com/ros2/rmw_zenoh/pull/200. Thank you! Unfortunately, I’m still experiencing two scenarios which I will describe in separate comments.

It isn’t since this patch is not in a pull request yet. I will update the comment once we have a zenoh PR number for it.

Sounds great! I just wanted to make sure, since https://github.com/ros2/rmw_zenoh/issues/198 refers to a benchmarking issue. Thank you for the quick response.

Thanks for your testing and quick feedback. We’ll analyse your backtraces.

I don’t know the codebase well enough yet and haven’t found if Thread 5 is supposed to drop the write lock and/or ctrl_lock further in the call chain.

The purpose of the patch is precisely to release the ctrl_lock before propagating the liveliness message to the callbacks in rmw_zenoh. So either the patch is not complete, either another lock is in cause. @OlivierHecart , isn’t the problem in this line which still takes the ctrl_lock while calling wend_declare(): https://github.com/eclipse-zenoh/zenoh/blob/d070e7ce777222692d34f466c0f1a95f0de2f068/zenoh/src/net/routing/dispatcher/tables.rs#L177

Also, is this referencing the correct issue number? https://github.com/ros2/rmw_zenoh/commit/614954c248bdb7e528dca116512279edf7937a12#diff-7d5bcea24badbde5330943e2f5ff4c2234863cb59a9f75c049d1b659620d0ef9L24

It isn’t since this patch is not in a pull request yet. I will update the comment once we have a zenoh PR number for it.