rmw_cyclonedds: Program termination with Ctrl+C stops in the middle.

@ivanpauno @hsgwa The way I read it, the problem was solved with changing the firewall settings, and it is not really a bug in Cyclone if it can’t receive data because of a firewall. Hanging during shutdown is a consequence of that.

I do want everything to work fine within a machine if at least the loopback interface works, and so I am working on supporting using multiple network interfaces simultaneously.

The interesting options as I see it are:

• If hanging during shutdown because of a firewall is a bigger problem than not communicating because of a firewall, then we can change the default to Internal/multipleReceiveThread=false to prevent the hang.
• We could try to autodetect a firewall blocking network traffic on start-up and by default not use network interfaces that are blocked. I don’t know exactly how reliable that would be, though.
• We can just wait for Cyclone to use the loopback interface and the ethernet in parallel. That should also fix it.

Thanks!

I reviewed network settings and found out that firewall was the cause. This issue can be reproduced as following steps.

1. Select interface to Ethernet

   1. Prepare XML

      <CycloneDDS>
          <Domain>
              <General>
                  <NetworkInterfaceAddress>YOUR_ETHERNET_INTERFACE</NetworkInterfaceAddress>
              </General>
              <Tracing>
                  <Category>config</Category>
                  <!-- save configs to cyclonedds.log -->
              </Tracing>
          </Domain>
      </CycloneDDS>
      

   2. Set CYCLONEDDS_URI

      $ export CYCLONEDDS_URI=/path/to/xml
      

2. Enable firewall

   $ sudo ufw enable
   

3. Run demo

   $ ros2 run demo_nodes_cpp talker
   

I tried to change cyclonedds configure, and these are results.

• If loopback is used, it will communicate and exit normally.
• When Internal/multipleReceiveThread=false, it does not communicate, but exits normally.

I apologize in advance if this has already been documented about FIrewall settings.

@iuhilnehc-ynos yes that’s the path, at least for the initial “trigger” message. https://github.com/eclipse-cyclonedds/cyclonedds/blob/81666fd987cbb2a7a61f24c96be923c0cd1b2995/src/core/ddsi/src/q_init.c#L778 schedules a periodic event to try again (https://github.com/eclipse-cyclonedds/cyclonedds/blob/81666fd987cbb2a7a61f24c96be923c0cd1b2995/src/core/ddsi/src/q_init.c#L784). The purpose of the packet is to unblock the recvmsg call, the content of the packet doesn’t matter, it just needs to return from recvmsg.

UDP/IP is unreliable and so it retries until the threads terminate, but as far as I know, the loopback interface is effectively reliable provided the socket receive buffer has space. If there is something present in the receive buffer, recvmsg should return it (and no matter what the contents are, it is sufficient to make it terminate); if there’s nothing present, surely this packet should be delivered.

I’ve seen it hang in this way before, even on machines doing absolutely nothing else (so no chance of socket buffer being full and the IP stack dropping the packet for that reason), but because of some routing issue that made it impossible for a process to send a packet to its own sockets. I have no clue why such a routing issue would exist …

It is hanging because it is not succeeding in waking up a receive thread by sending a datagram to it: in principle, Cyclone can have any number of threads for receiving data from the network, and each thread handles either a single socket or multiple sockets. If it is the former, it enters a blocking wait, and then termination requires receiving a datagram to unblock the recvmsg() call. For those threads it sends itself a minimalistic datagram and retries every second.

From the stack traces, you can tell it is one of those threads, and you can also see its timed-event handling being in a state consistent with those retries:

ros2/rcl#2  __pthread_cond_timedwait (cond=0x55a37be0a470, mutex=0x55a37be0a448, abstime=0x7fcf40a014d0) at pthread_cond_wait.c:656
ros2/rcl#3  0x00007fcf432b60fb in ddsrt_cond_waituntil (cond=0x55a37be0a470, mutex=0x55a37be0a448, abstime=1611795496341632475) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/ddsrt/src/sync/posix/sync.c:116
ros2/rcl#4  0x00007fcf432b61bd in ddsrt_cond_waitfor (cond=0x55a37be0a470, mutex=0x55a37be0a448, reltime=999899872) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/ddsrt/src/sync/posix/sync.c:137
ros2/rcl#5  0x00007fcf432726a1 in xevent_thread (xevq=0x55a37be0a3f0) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsi/src/q_xevent.c:1286

On Linux, by default, it creates 3 threads: one that handles all discovery traffic (multicast+unicast, so 2 sockets and exempt from this termination procedure), one that handles unicast application data (single socket), and one that handles multicast application data (also single socket). Chances are it is the multicast one here, but I can’t be quite certain.

Not being able to send datagrams to oneself is probably a network configuration issue, but I don’t know what bit of is causing the problem. So it is probably sensible to avoid having to do this by not having any receive thread handling a single socket. For that, set Internal/MultipleReceiveThreads to false.

@ivanpauno

@hsgwa can you share a multhithreaded backtrace (thread apply all bt)?

here it is. I added version info of related packages to the original post.

(gdb) thread apply all bt
Thread 9 (Thread 0x7fcf2b7fe700 (LWP 3519)):
#0  __libc_recvmsg (flags=0, msg=0x7fcf2b7fd1b0, fd=8) at ../sysdeps/unix/sysv/linux/recvmsg.c:28
ros2/rcl#1  __libc_recvmsg (fd=8, msg=0x7fcf2b7fd1b0, flags=0) at ../sysdeps/unix/sysv/linux/recvmsg.c:25
ros2/rcl#2  0x00007fcf432b52fc in ddsrt_recvmsg (sock=8, msg=0x7fcf2b7fd1b0, flags=0, rcvd=0x7fcf2b7fd188) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/ddsrt/src/sockets/posix/socket.c:455
ros2/rcl#3  0x00007fcf431e7c10 in ddsi_udp_conn_read (conn_cmn=0x55a37be0a7c0, buf=0x7fcf40000070 "", len=65536, allow_spurious=true, srcloc=0x7fcf2b7fd430) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsi/src/ddsi_udp.c:90
ros2/rcl#4  0x00007fcf431e6be5 in ddsi_conn_read (conn=0x55a37be0a7c0, buf=0x7fcf40000070 "", len=65536, allow_spurious=true, srcloc=0x7fcf2b7fd430) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/security/api/../../core/ddsi/include/dds/ddsi/ddsi_tran.h:254
ros2/rcl#5  0x00007fcf4326728c in do_packet (ts1=0x55a37be02500, gv=0x55a37be0afa0, conn=0x55a37be0a7c0, guidprefix=0x0, rbpool=0x55a37be17f50) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsi/src/q_receive.c:3252
ros2/rcl#6  0x00007fcf43268172 in recv_thread (vrecv_thread_arg=0x55a37be0dde0) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsi/src/q_receive.c:3500
ros2/rcl#7  0x00007fcf43269c88 in create_thread_wrapper (ptr=0x55a37be02500) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsi/src/q_thread.c:241
ros2/rcl#8  0x00007fcf432b67aa in os_startRoutineWrapper (threadContext=0x55a37be17fb0) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/ddsrt/src/threads/posix/threads.c:165
ros2/rcl#9  0x00007fcf446c7609 in start_thread (arg=<optimized out>) at pthread_create.c:477
ros2/rcl#10 0x00007fcf44a23293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7fcf40a02700 (LWP 3517)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fcf40a014d0, clockid=<optimized out>, expected=0, futex_word=0x55a37be0a498) at ../sysdeps/nptl/futex-internal.h:320
ros2/rcl#1  __pthread_cond_wait_common (abstime=0x7fcf40a014d0, clockid=<optimized out>, mutex=0x55a37be0a448, cond=0x55a37be0a470) at pthread_cond_wait.c:520
ros2/rcl#2  __pthread_cond_timedwait (cond=0x55a37be0a470, mutex=0x55a37be0a448, abstime=0x7fcf40a014d0) at pthread_cond_wait.c:656
ros2/rcl#3  0x00007fcf432b60fb in ddsrt_cond_waituntil (cond=0x55a37be0a470, mutex=0x55a37be0a448, abstime=1611795496341632475) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/ddsrt/src/sync/posix/sync.c:116
ros2/rcl#4  0x00007fcf432b61bd in ddsrt_cond_waitfor (cond=0x55a37be0a470, mutex=0x55a37be0a448, reltime=999899872) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/ddsrt/src/sync/posix/sync.c:137
ros2/rcl#5  0x00007fcf432726a1 in xevent_thread (xevq=0x55a37be0a3f0) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsi/src/q_xevent.c:1286
ros2/rcl#6  0x00007fcf43269c88 in create_thread_wrapper (ptr=0x55a37be02480) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsi/src/q_thread.c:241
ros2/rcl#7  0x00007fcf432b67aa in os_startRoutineWrapper (threadContext=0x55a37be112a0) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/ddsrt/src/threads/posix/threads.c:165
ros2/rcl#8  0x00007fcf446c7609 in start_thread (arg=<optimized out>) at pthread_create.c:477
ros2/rcl#9  0x00007fcf44a23293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7fcf41203700 (LWP 3516)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a37be0a2d0) at ../sysdeps/nptl/futex-internal.h:183
ros2/rcl#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55a37be0a280, cond=0x55a37be0a2a8) at pthread_cond_wait.c:508
ros2/rcl#2  __pthread_cond_wait (cond=0x55a37be0a2a8, mutex=0x55a37be0a280) at pthread_cond_wait.c:638
ros2/rcl#3  0x00007fcf432b5fc2 in ddsrt_cond_wait (cond=0x55a37be0a2a8, mutex=0x55a37be0a280) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/ddsrt/src/sync/posix/sync.c:92
ros2/rcl#4  0x00007fcf4325ab88 in dqueue_thread (q=0x55a37be0a280) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsi/src/q_radmin.c:2502
ros2/rcl#5  0x00007fcf43269c88 in create_thread_wrapper (ptr=0x55a37be02440) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsi/src/q_thread.c:241
ros2/rcl#6  0x00007fcf432b67aa in os_startRoutineWrapper (threadContext=0x55a37be110d0) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/ddsrt/src/threads/posix/threads.c:165
ros2/rcl#7  0x00007fcf446c7609 in start_thread (arg=<optimized out>) at pthread_create.c:477
ros2/rcl#8  0x00007fcf44a23293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7fcf41a04700 (LWP 3515)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a37be0a230) at ../sysdeps/nptl/futex-internal.h:183
ros2/rcl#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55a37be0a1e0, cond=0x55a37be0a208) at pthread_cond_wait.c:508
ros2/rcl#2  __pthread_cond_wait (cond=0x55a37be0a208, mutex=0x55a37be0a1e0) at pthread_cond_wait.c:638
ros2/rcl#3  0x00007fcf432b5fc2 in ddsrt_cond_wait (cond=0x55a37be0a208, mutex=0x55a37be0a1e0) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/ddsrt/src/sync/posix/sync.c:92
ros2/rcl#4  0x00007fcf4325ab88 in dqueue_thread (q=0x55a37be0a1e0) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsi/src/q_radmin.c:2502
ros2/rcl#5  0x00007fcf43269c88 in create_thread_wrapper (ptr=0x55a37be02400) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsi/src/q_thread.c:241
ros2/rcl#6  0x00007fcf432b67aa in os_startRoutineWrapper (threadContext=0x55a37be09650) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/ddsrt/src/threads/posix/threads.c:165
ros2/rcl#7  0x00007fcf446c7609 in start_thread (arg=<optimized out>) at pthread_create.c:477
ros2/rcl#8  0x00007fcf44a23293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7fcf42205700 (LWP 3514)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fcf422044b0, clockid=<optimized out>, expected=0, futex_word=0x55a37be08d10) at ../sysdeps/nptl/futex-internal.h:320
ros2/rcl#1  __pthread_cond_wait_common (abstime=0x7fcf422044b0, clockid=<optimized out>, mutex=0x55a37be08cc0, cond=0x55a37be08ce8) at pthread_cond_wait.c:520
ros2/rcl#2  __pthread_cond_timedwait (cond=0x55a37be08ce8, mutex=0x55a37be08cc0, abstime=0x7fcf422044b0) at pthread_cond_wait.c:656
ros2/rcl#3  0x00007fcf432b60fb in ddsrt_cond_waituntil (cond=0x55a37be08ce8, mutex=0x55a37be08cc0, abstime=1611796476340858161) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/ddsrt/src/sync/posix/sync.c:116
ros2/rcl#4  0x00007fcf432b61bd in ddsrt_cond_waitfor (cond=0x55a37be08ce8, mutex=0x55a37be08cc0, reltime=1000000000000) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/ddsrt/src/sync/posix/sync.c:137
ros2/rcl#5  0x00007fcf4324acf6 in gcreq_queue_thread (q=0x55a37be08cb0) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsi/src/q_gc.c:133
ros2/rcl#6  0x00007fcf43269c88 in create_thread_wrapper (ptr=0x55a37be023c0) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsi/src/q_thread.c:241
ros2/rcl#7  0x00007fcf432b67aa in os_startRoutineWrapper (threadContext=0x55a37be11360) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/ddsrt/src/threads/posix/threads.c:165
ros2/rcl#8  0x00007fcf446c7609 in start_thread (arg=<optimized out>) at pthread_create.c:477
ros2/rcl#9  0x00007fcf44a23293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7fcf42da5700 (LWP 3513)):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7fcf4560c9e0 <rclcpp::SignalHandler::signal_handler_sem_>) at ../sysdeps/nptl/futex-internal.h:320
ros2/rcl#1  do_futex_wait (sem=sem@entry=0x7fcf4560c9e0 <rclcpp::SignalHandler::signal_handler_sem_>, abstime=0x0, clockid=0) at sem_waitcommon.c:112
ros2/rcl#2  0x00007fcf446d14e8 in __new_sem_wait_slow (sem=0x7fcf4560c9e0 <rclcpp::SignalHandler::signal_handler_sem_>, abstime=0x0, clockid=0) at sem_waitcommon.c:184
ros2/rcl#3  0x00007fcf454761fc in rclcpp::SignalHandler::wait_for_signal () at /home/hasegawa/ros2_rolling/src/ros2/rclcpp/rclcpp/src/rclcpp/signal_handler.cpp:335
ros2/rcl#4  0x00007fcf45475be6 in rclcpp::SignalHandler::deferred_signal_handler (this=0x7fcf4560ca20 <rclcpp::SignalHandler::get_global_signal_handler()::signal_handler>) at /home/hasegawa/ros2_rolling/src/ros2/rclcpp/rclcpp/src/rclcpp/signal_handler.cpp:256
ros2/rcl#5  0x00007fcf45476d59 in std::__invoke_impl<void, void (rclcpp::SignalHandler::*)(), rclcpp::SignalHandler*> (__f=@0x55a37bddb200: (void (rclcpp::SignalHandler::*)(class rclcpp::SignalHandler * const)) 0x7fcf45475594 <rclcpp::SignalHandler::deferred_signal_handler()>, __t=@0x55a37bddb1f8: 0x7fcf4560ca20 <rclcpp::SignalHandler::get_global_signal_handler()::signal_handler>) at /usr/include/c++/9/bits/invoke.h:73
ros2/rcl#6  0x00007fcf45476c73 in std::__invoke<void (rclcpp::SignalHandler::*)(), rclcpp::SignalHandler*> (__fn=@0x55a37bddb200: (void (rclcpp::SignalHandler::*)(class rclcpp::SignalHandler * const)) 0x7fcf45475594 <rclcpp::SignalHandler::deferred_signal_handler()>) at /usr/include/c++/9/bits/invoke.h:95
ros2/rcl#7  0x00007fcf45476bc3 in std::thread::_Invoker<std::tuple<void (rclcpp::SignalHandler::*)(), rclcpp::SignalHandler*> >::_M_invoke<0ul, 1ul> (this=0x55a37bddb1f8) at /usr/include/c++/9/thread:244
ros2/rcl#8  0x00007fcf45476b65 in std::thread::_Invoker<std::tuple<void (rclcpp::SignalHandler::*)(), rclcpp::SignalHandler*> >::operator() (this=0x55a37bddb1f8) at /usr/include/c++/9/thread:251
ros2/rcl#9  0x00007fcf45476b36 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rclcpp::SignalHandler::*)(), rclcpp::SignalHandler*> > >::_M_run (this=0x55a37bddb1f0) at /usr/include/c++/9/thread:195
ros2/rcl#10 0x00007fcf44be4d84 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
ros2/rcl#11 0x00007fcf446c7609 in start_thread (arg=<optimized out>) at pthread_create.c:477
ros2/rcl#12 0x00007fcf44a23293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7fcf43c9e700 (LWP 3512)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
ros2/rcl#1  0x00007fcf445cf45d in ?? () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
ros2/rcl#2  0x00007fcf446c7609 in start_thread (arg=<optimized out>) at pthread_create.c:477
ros2/rcl#3  0x00007fcf44a23293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7fcf4449f700 (LWP 3511)):
#0  __libc_recvmsg (flags=0, msg=0x7fcf4449d840, fd=3) at ../sysdeps/unix/sysv/linux/recvmsg.c:28
ros2/rcl#1  __libc_recvmsg (fd=3, msg=0x7fcf4449d840, flags=0) at ../sysdeps/unix/sysv/linux/recvmsg.c:25
ros2/rcl#2  0x00007fcf445c917a in ustcomm_recv_unix_sock () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
ros2/rcl#3  0x00007fcf445cf367 in ?? () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
ros2/rcl#4  0x00007fcf446c7609 in start_thread (arg=<optimized out>) at pthread_create.c:477
ros2/rcl#5  0x00007fcf44a23293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95


Thread 1 (Thread 0x7fcf444a6000 (LWP 3509)):
#0  __pthread_clockjoin_ex (threadid=140527764760320, thread_return=0x7ffd0385ffe0, clockid=<optimized out>, abstime=<optimized out>, block=<optimized out>) at pthread_join_common.c:145
ros2/rcl#1  0x00007fcf432b6f44 in ddsrt_thread_join (thread=..., thread_result=0x0) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/ddsrt/src/threads/posix/threads.c:380
ros2/rcl#2  0x00007fcf4326a23f in join_thread (ts1=0x55a37be02500) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsi/src/q_thread.c:381
ros2/rcl#3  0x00007fcf4324d40b in wait_for_receive_threads (gv=0x55a37be0afa0) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsi/src/q_init.c:795
ros2/rcl#4  0x00007fcf43250bfc in rtps_stop (gv=0x55a37be0afa0) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsi/src/q_init.c:1761
ros2/rcl#5  0x00007fcf43289884 in dds_domain_free (vdomain=0x55a37be0ad00) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsc/src/dds_domain.c:311
ros2/rcl#6  0x00007fcf4328dfb3 in dds_entity_deriver_delete (e=0x55a37be0ad00) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsc/src/dds__types.h:193
ros2/rcl#7  0x00007fcf4328eec8 in really_delete_pinned_closed_locked (e=0x55a37be0ad00, delstate=DIS_EXPLICIT) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsc/src/dds_entity.c:522
ros2/rcl#8  0x00007fcf4328ebab in dds_delete_impl_pinned (e=0x55a37be0ad00, delstate=DIS_EXPLICIT) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsc/src/dds_entity.c:409
ros2/rcl#9  0x00007fcf4328eb20 in dds_delete_impl (entity=1353190617, delstate=DIS_EXPLICIT) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsc/src/dds_entity.c:384
ros2/rcl#10 0x00007fcf4328ea60 in dds_delete (entity=1353190617) at /home/hasegawa/ros2_rolling/src/eclipse-cyclonedds/cyclonedds/src/core/ddsc/src/dds_entity.c:367
ros2/rcl#11 0x00007fcf433bc7cd in check_destroy_domain (domain_id=0) at /home/hasegawa/ros2_rolling/src/ros2/rmw_cyclonedds/rmw_cyclonedds_cpp/src/rmw_node.cpp:826
ros2/rcl#12 0x00007fcf433be12b in rmw_context_impl_t::clean_up (this=0x55a37be00270) at /home/hasegawa/ros2_rolling/src/ros2/rmw_cyclonedds/rmw_cyclonedds_cpp/src/rmw_node.cpp:1129
ros2/rcl#13 0x00007fcf433be1aa in rmw_context_impl_t::fini (this=0x55a37be00270) at /home/hasegawa/ros2_rolling/src/ros2/rmw_cyclonedds/rmw_cyclonedds_cpp/src/rmw_node.cpp:1140
ros2/rcl#14 0x00007fcf433bfd27 in rmw_destroy_node (node=0x55a37be2a190) at /home/hasegawa/ros2_rolling/src/ros2/rmw_cyclonedds/rmw_cyclonedds_cpp/src/rmw_node.cpp:1372
ros2/rcl#15 0x00007fcf448623ec in rmw_destroy_node (v1=0x55a37be2a190) at /home/hasegawa/ros2_rolling/src/ros2/rmw_implementation/rmw_implementation/src/functions.cpp:220
ros2/rcl#16 0x00007fcf448c87f3 in rcl_node_fini (node=0x55a37be0aab0) at /home/hasegawa/ros2_rolling/src/ros2/rcl/rcl/src/rcl/node.c:382
ros2/rcl#17 0x00007fcf453b3bf8 in rclcpp::node_interfaces::NodeBase::<lambda(rcl_node_t*)>::operator()(rcl_node_t *) const (__closure=0x55a37be2ee60, node=0x55a37be0aab0) at /home/hasegawa/ros2_rolling/src/ros2/rclcpp/rclcpp/src/rclcpp/node_interfaces/node_base.cpp:141
ros2/rcl#18 0x00007fcf453b597e in std::_Sp_counted_deleter<rcl_node_t*, rclcpp::node_interfaces::NodeBase::NodeBase(const string&, const string&, rclcpp::Context::SharedPtr, const rcl_node_options_t&, bool, bool)::<lambda(rcl_node_t*)>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose(void) (this=0x55a37be2ee50) at /usr/include/c++/9/bits/shared_ptr_base.h:471
ros2/rcl#19 0x000055a37a90a4c0 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x55a37be2ee50) at /usr/include/c++/9/bits/shared_ptr_base.h:155
ros2/rcl#20 0x000055a37a90959f in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x55a37be00878, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:730
ros2/rcl#21 0x00007fcf4533d8d0 in std::__shared_ptr<rcl_node_t, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x55a37be00870, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:1169
ros2/rcl#22 0x00007fcf4533d8f0 in std::shared_ptr<rcl_node_t>::~shared_ptr (this=0x55a37be00870, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr.h:103
ros2/rcl#23 0x00007fcf453b4a1c in rclcpp::node_interfaces::NodeBase::~NodeBase (this=0x55a37be00850, __in_chrg=<optimized out>) at /home/hasegawa/ros2_rolling/src/ros2/rclcpp/rclcpp/src/rclcpp/node_interfaces/node_base.cpp:157
ros2/rcl#24 0x00007fcf453b4a7c in rclcpp::node_interfaces::NodeBase::~NodeBase (this=0x55a37be00850, __in_chrg=<optimized out>) at /home/hasegawa/ros2_rolling/src/ros2/rclcpp/rclcpp/src/rclcpp/node_interfaces/node_base.cpp:169
ros2/rcl#25 0x00007fcf453abb52 in std::_Sp_counted_ptr<rclcpp::node_interfaces::NodeBase*, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x55a37be2eea0) at /usr/include/c++/9/bits/shared_ptr_base.h:377
ros2/rcl#26 0x000055a37a90a4c0 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x55a37be2eea0) at /usr/include/c++/9/bits/shared_ptr_base.h:155
ros2/rcl#27 0x000055a37a90959f in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x55a37be090f0, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:730
ros2/rcl#28 0x000055a37a90921c in std::__shared_ptr<rclcpp::node_interfaces::NodeBaseInterface, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x55a37be090e8, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:1169
ros2/rcl#29 0x000055a37a909266 in std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface>::~shared_ptr (this=0x55a37be090e8, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr.h:103
ros2/rcl#30 0x00007fcf453a270a in rclcpp::Node::~Node (this=0x55a37be090d0, __in_chrg=<optimized out>) at /home/hasegawa/ros2_rolling/src/ros2/rclcpp/rclcpp/src/rclcpp/node.cpp:187
ros2/rcl#31 0x00007fcf424b28f6 in demo_nodes_cpp::Talker::~Talker (this=0x55a37be090d0, __in_chrg=<optimized out>) at /home/hasegawa/ros2_rolling/src/ros2/demos/demo_nodes_cpp/src/topics/talker.cpp:33
ros2/rcl#32 0x00007fcf424b9989 in __gnu_cxx::new_allocator<demo_nodes_cpp::Talker>::destroy<demo_nodes_cpp::Talker> (this=0x55a37be090d0, __p=0x55a37be090d0) at /usr/include/c++/9/ext/new_allocator.h:153
ros2/rcl#33 0x00007fcf424b994d in std::allocator_traits<std::allocator<demo_nodes_cpp::Talker> >::destroy<demo_nodes_cpp::Talker> (__a=..., __p=0x55a37be090d0) at /usr/include/c++/9/bits/alloc_traits.h:497
ros2/rcl#34 0x00007fcf424b9805 in std::_Sp_counted_ptr_inplace<demo_nodes_cpp::Talker, std::allocator<demo_nodes_cpp::Talker>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x55a37be090c0) at /usr/include/c++/9/bits/shared_ptr_base.h:557
ros2/rcl#35 0x000055a37a90a4c0 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x55a37be090c0) at /usr/include/c++/9/bits/shared_ptr_base.h:155
ros2/rcl#36 0x000055a37a90959f in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x55a37be6af58, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:730
ros2/rcl#37 0x000055a37a90928a in std::__shared_ptr<void, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x55a37be6af50, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:1169
ros2/rcl#38 0x000055a37a9092aa in std::shared_ptr<void>::~shared_ptr (this=0x55a37be6af50, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr.h:103
ros2/rcl#39 0x000055a37a9094c0 in rclcpp_components::NodeInstanceWrapper::~NodeInstanceWrapper (this=0x55a37be6af50, __in_chrg=<optimized out>) at /home/hasegawa/ros2_rolling/install/rclcpp_components/include/rclcpp_components/node_instance_wrapper.hpp:26
ros2/rcl#40 0x000055a37a90de36 in std::_Destroy<rclcpp_components::NodeInstanceWrapper> (__pointer=0x55a37be6af50) at /usr/include/c++/9/bits/stl_construct.h:98
ros2/rcl#41 0x000055a37a90d27d in std::_Destroy_aux<false>::__destroy<rclcpp_components::NodeInstanceWrapper*> (__first=0x55a37be6af50, __last=0x55a37be6af80) at /usr/include/c++/9/bits/stl_construct.h:108
ros2/rcl#42 0x000055a37a90c0f6 in std::_Destroy<rclcpp_components::NodeInstanceWrapper*> (__first=0x55a37be6af50, __last=0x55a37be6af80) at /usr/include/c++/9/bits/stl_construct.h:137
ros2/rcl#43 0x000055a37a90accd in std::_Destroy<rclcpp_components::NodeInstanceWrapper*, rclcpp_components::NodeInstanceWrapper> (__first=0x55a37be6af50, __last=0x55a37be6af80) at /usr/include/c++/9/bits/stl_construct.h:206
ros2/rcl#44 0x000055a37a90bbc4 in std::vector<rclcpp_components::NodeInstanceWrapper, std::allocator<rclcpp_components::NodeInstanceWrapper> >::_M_erase_at_end (this=0x7ffd03863220, __pos=0x55a37be6af50) at /usr/include/c++/9/bits/stl_vector.h:1793
ros2/rcl#45 0x000055a37a90a432 in std::vector<rclcpp_components::NodeInstanceWrapper, std::allocator<rclcpp_components::NodeInstanceWrapper> >::clear (this=0x7ffd03863220) at /usr/include/c++/9/bits/stl_vector.h:1496
ros2/rcl#46 0x000055a37a9082d3 in main (argc=4, argv=0x7ffd03864098) at /home/hasegawa/ros2_rolling/build/demo_nodes_cpp/rclcpp_components/node_main_talker.cpp:61