rclcpp: Memory leaks: Error in destruction of rcl action client handle: the Node Handle was destructed too early.

Memory leaks are found when running test_buffer_client of tf2_ros. with the Address Sanitizer (https://en.wikipedia.org/wiki/AddressSanitizer) on.

Below is the error message. Let me know if you need a dockerfile to reproduce it.

root@5490e71bc767:/opt/ros_ws# ./build/tf2_ros/test_buffer_client
[==========] Running 4 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 4 tests from TestBufferClient
[ RUN      ] TestBufferClient.lookup_transform_available
[       OK ] TestBufferClient.lookup_transform_available (236 ms)
[ RUN      ] TestBufferClient.lookup_transform_unavailable
[       OK ] TestBufferClient.lookup_transform_unavailable (175 ms)
[ RUN      ] TestBufferClient.can_transform_available
[       OK ] TestBufferClient.can_transform_available (172 ms)
[ RUN      ] TestBufferClient.can_transform_unavailable
[ERROR] [1612723447.179594300] [rclcpp_action]: Error in destruction of rcl action client handle: the Node Handle was destructed too early. You will leak memory
[       OK ] TestBufferClient.can_transform_unavailable (170 ms)
[----------] 4 tests from TestBufferClient (753 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test case ran. (823 ms total)
[  PASSED  ] 4 tests.

=================================================================
==30==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 568 byte(s) in 1 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055cc2d2da in rcl_action_client_init (/opt/ros/foxy/lib/librcl_action.so+0x52da)

Direct leak of 56 byte(s) in 1 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055c635892  (/opt/ros/foxy/lib/librcl.so+0x15892)
    #2 0x7f055cd1dd0a in rclcpp::graph_listener::GraphListener::GraphListener(std::shared_ptr<rclcpp::Context>) (/opt/ros/foxy/lib/librclcpp.so+0xe2d0a)
    #3 0x7f055cd4181b in std::shared_ptr<rclcpp::graph_listener::GraphListener> rclcpp::Context::get_sub_context<rclcpp::graph_listener::GraphListener, std::shared_ptr<rclcpp::Context> >(std::shared_ptr<rclcpp::Context>&&) (/opt/ros/foxy/lib/librclcpp.so+0x10681b)

Indirect leak of 480 byte(s) in 2 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055c6412d8 in rcl_subscription_init (/opt/ros/foxy/lib/librcl.so+0x212d8)

Indirect leak of 456 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597d4708 in rmw_create_client (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x1c708)

Indirect leak of 432 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055c63113e in rcl_client_init (/opt/ros/foxy/lib/librcl.so+0x1113e)

Indirect leak of 384 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597d3026 in rmw_create_client (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x1b026)

Indirect leak of 384 byte(s) in 2 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597e2e60  (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x2ae60)
    #2 0x7f05597e0de1 in rmw_create_subscription (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x28de1)

Indirect leak of 360 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597d4d20 in rmw_create_client (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x1cd20)

Indirect leak of 360 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597d3110 in rmw_create_client (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x1b110)

Indirect leak of 240 byte(s) in 2 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597e2d18  (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x2ad18)
    #2 0x7f05597e0de1 in rmw_create_subscription (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x28de1)

Indirect leak of 201 byte(s) in 6 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055c5ea56f in rmw_allocate (/opt/ros/foxy/lib/librmw.so+0x256f)

Indirect leak of 192 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597d4eb0 in rmw_create_client (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x1ceb0)

Indirect leak of 164 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbc947 in operator new(unsigned long) (/lib/x86_64-linux-gnu/libasan.so.5+0x10f947)
    #1 0x7f05597e466e  (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x2c66e)

Indirect leak of 161 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbc947 in operator new(unsigned long) (/lib/x86_64-linux-gnu/libasan.so.5+0x10f947)
    #1 0x7f05597e43de  (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x2c3de)

Indirect leak of 160 byte(s) in 2 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597e1ec5  (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x29ec5)
    #2 0x7f05597e0de1 in rmw_create_subscription (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x28de1)

Indirect leak of 103 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055cbef37c in rcutils_strndup (/opt/ros/foxy/lib/librcutils.so+0xd37c)

Indirect leak of 98 byte(s) in 2 object(s) allocated from:
    #0 0x7f055cfbc947 in operator new(unsigned long) (/lib/x86_64-linux-gnu/libasan.so.5+0x10f947)
    #1 0x7f05597e4156  (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x2c156)

Indirect leak of 96 byte(s) in 2 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055c5ea56f in rmw_allocate (/opt/ros/foxy/lib/librmw.so+0x256f)
    #2 0x7f05597e0de1 in rmw_create_subscription (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x28de1)

Indirect leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x7f055cfbc947 in operator new(unsigned long) (/lib/x86_64-linux-gnu/libasan.so.5+0x10f947)
    #1 0x7f055978faf7 in rmw_fastrtps_shared_cpp::__rmw_create_guard_condition(char const*) (/opt/ros/foxy/lib/librmw_fastrtps_shared_cpp.so+0x2caf7)

Indirect leak of 24 byte(s) in 1 object(s) allocated from:
    #0 0x7f055cfbc947 in operator new(unsigned long) (/lib/x86_64-linux-gnu/libasan.so.5+0x10f947)
    #1 0x7f055978fae7 in rmw_fastrtps_shared_cpp::__rmw_create_guard_condition(char const*) (/opt/ros/foxy/lib/librmw_fastrtps_shared_cpp.so+0x2cae7)

SUMMARY: AddressSanitizer: 4983 byte(s) leaked in 49 allocation(s).

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 20 (3 by maintainers)

Most upvoted comments

Perhaps something is wrong with this Dockerfile since it mixes rolling and foxy?

Yes, that’s a problem (we haven’t backported all of the fixes from rolling to foxy). You should change that to:

RUN cd $ROS_WS \
        &&  git clone -b ros2 --depth 1 https://github.com/ros2/geometry2 $ROS_WS/src/geometry2 \
        && . /opt/ros/${ROS_DISTRO}/setup.sh \
        &&  rm -rf ${ROS_WS}/build && VERBOSE=1 CXX="g++" CXXFLAGS="--coverage -g -fsanitize=address" colcon build --event-handlers console_direct+

(notice that we checkout the ros2 branch now)

+2
clalancette on Mar 4, 2021

Can you give us the stack trace of the leak?

+1
clalancette on Mar 5, 2021

Just find out that the same memory leaks can be exposed by running three other binaries (besides tf2_ros/test_buffer_client) from geometry2, with Address Sanitizers on:

  • test_tf2/test_message_filter
  • tf2_ros/buffer_server
  • tf2_ros/tf2_ros_test_time_reset

The leaks can be reproduced on the ros:foxy docker with the same dockerfile above. I have not tested with the recent rclcpp though (due to difficulties with source compilation).

+1
zhoulaifu on Feb 15, 2021

You can use this dockerfile to reproduce the environmental setting.

In the docker container, you have a rox:foxy system with geometry2 compiled with the Address Sanitizer (ASan). Then run the following binary from the tf2_ros package to reproduce the bug. ${ROS_WS}/build/tf2_ros/test_buffer_client

Please let me know if anything unclear.

+1
zhoulaifu on Feb 8, 2021
Read more comments on GitHub
← rclcpp: High CPU usage when using DDS intra-process communication
rclcpp: Possible memory leak in spin_some() →