geometry2: rclpy BufferClient lookup_transform not working, timeout
Bug report
Required Info:
- Operating System:
- Docker + Ubuntu 20.04
- Installation type:
- From binaries
- Version or commit hash:
- Foxy
- DDS implementation:
- Fast-RTPS,
- Client library (if applicable):
- rclpy
Steps to reproduce issue
Calling BufferClient.lookup_transform always timeouts although if I call the buffer server via ros2 cli it works.
Create a node like the following:
#!/usr/bin/env python3
import rclpy
from rclpy.node import Node
from tf2_ros.buffer_client import BufferClient
from rclpy.duration import Duration
from std_msgs.msg import Empty
class TestNode(Node):
def __init__(
self,
):
super().__init__("test_node")
self.cb_group = None
self.buffer = BufferClient(
self,
"/tf2_buffer_server",
timeout_padding=1.0,
callback_group=self.cb_group,
)
self.sub = self.create_subscription(
msg_type=Empty,
topic="/test_buffer_client",
callback=self.buffer_cb,
qos_profile=1,
)
def buffer_cb(self, msg):
_ = self.buffer.lookup_transform(
"map",
"odom",
rclpy.time.Time(seconds=0, nanoseconds=0),
timeout=Duration(seconds=1.0),
)
self.get_logger().info("Success")
def main(args=None):
rclpy.init(args=args)
node = TestNode()
executor = None
rclpy.spin(node, executor)
if __name__ == "__main__":
main()
Make sure the buffer server is already running and there is a valid transform between map->odom. Then publish to the topic that calls BufferClient.lookup_transform.
ros2 topic pub -1 /test_buffer_client std_msgs/msg/Empty
The traceback is:
Traceback (most recent call last):
File "/workspace/rover/ros2/install/navigation/lib/navigation/navigation_test", line 33, in <module>
sys.exit(load_entry_point('navigation', 'console_scripts', 'navigation_test')())
File "/workspace/rover/ros2/build/navigation/navigation/node_test.py", line 53, in main
rclpy.spin(node, executor)
File "/opt/ros/foxy/lib/python3.8/site-packages/rclpy/__init__.py", line 191, in spin
executor.spin_once()
File "/opt/ros/foxy/lib/python3.8/site-packages/rclpy/executors.py", line 711, in spin_once
raise handler.exception()
File "/opt/ros/foxy/lib/python3.8/site-packages/rclpy/task.py", line 239, in __call__
self._handler.send(None)
File "/opt/ros/foxy/lib/python3.8/site-packages/rclpy/executors.py", line 426, in handler
await call_coroutine(entity, arg)
File "/opt/ros/foxy/lib/python3.8/site-packages/rclpy/executors.py", line 351, in _execute_subscription
await await_or_execute(sub.callback, msg)
File "/opt/ros/foxy/lib/python3.8/site-packages/rclpy/executors.py", line 118, in await_or_execute
return callback(*args)
File "/workspace/rover/ros2/build/navigation/navigation/node_test.py", line 33, in buffer_cb
_ = self.buffer.lookup_transform(
File "/opt/ros/foxy/lib/python3.8/site-packages/tf2_ros/buffer_client.py", line 87, in lookup_transform
return self.__process_goal(goal)
File "/opt/ros/foxy/lib/python3.8/site-packages/tf2_ros/buffer_client.py", line 201, in __process_goal
raise tf2.TimeoutException("The LookupTransform goal sent to the BufferServer did not come back in the specified time. Something is likely wrong with the server")
tf2.TimeoutException: The LookupTransform goal sent to the BufferServer did not come back in the specified time. Something is likely wrong with the server
Expected behavior
It shouldn’t crash since it works with the ros2 cli. If you use the following it works:
ros2 action send_goal /tf2_buffer_server tf2_msgs/action/LookupTransform '{target_frame: map, source_frame: odom}'
Actual behavior
It timeouts every time.
Additional notes
I modified https://github.com/ros2/geometry2/blob/ros2/tf2_ros_py/tf2_ros/buffer_client.py#L61 to accept also a callback group that is passed to the ActionClient. Then, in the TestNode example above I set the self.cb_group=ReentrantCallbackGroup() and the executor to be a MultiThreadedExecutor. Only that way it works.
I am missing something?
About this issue
- Original URL
- State: open
- Created 5 years ago
- Reactions: 3
- Comments: 20 (14 by maintainers)
For people running into this problem:
Setting
spin_threadtotruedid not fix my problem of doingtransformlookups within a callback. Adding arclpy.executors.MultiThreadedExecutordid.For better or worse, that is by design. The problem with doing this is that you are blocking the executor, but the executor needs to run in order to get the transform.
The solution here is to make sure to never run blocking functions like
lookupTransformfrom within a callback, or you use the MultiThreadedExecutor, or you instantiate theTransformListenerwithspin_thread = True.Look very cool. I would be glad if we could get something like this upstream with a PR. I also think that the “normal” non service client buffer could very much benefit from this.
I wan’t the only one working on an implementation of the events executor for Python and somebody finished a more polished version first. It has been added to rolling in https://github.com/ros2/rclpy/pull/1391 and I have backported this implementation to jazzy in https://github.com/ros2/rclpy/pull/1435.
This tip from @reinzor worked for me:
To add a small detail, for me it was just switching this line in
mainfrom:to
This blog is quite relevant for this issue to better understand it; https://karelics.fi/deadlocks-in-rclpy/
I would propose to the maintainers to put the service client of the BufferClient in a separate callback group, like is done for the subscribers of the transform listener
BufferClientdoesn’t have anyTransformListener. Also usingMultiThreadedExecutordoes not work, you have to pass aReentrantCallbackGroupto the client inside theBufferClientclass -which currently is not exposed- as already mentioned in the issue.