ros_control: Multiple controller_manager spawn calls freeze a controller manager

Using https://github.com/davetcoleman/ros_control_boilerplate/tree/0.4.0/rrbot_control as an example, replace the ros_control_controller_manager node in https://github.com/davetcoleman/ros_control_boilerplate/blob/0.4.0/rrbot_control/launch/rrbot_hardware.launch#L21 with:

<node name="ros_control_controller_manager" pkg="controller_manager" type="controller_manager" respawn="false"
	  output="screen" args="spawn joint_state_controller" />
<node name="ros_control_controller_manager" pkg="controller_manager" type="controller_manager" respawn="false"
	  output="screen" args="spawn position_trajectory_controller" />

Build and run roslaunch rrbot_control rrbot_hardware.launch and the controller manager stops responding, either to service calls (ie, /rrbot/controller_manager/list_controllers) or Ctrl-C (if run outside roslaunch). The same thing happens if using spawner instead of controller_manager. I haven’t tested it on a controller manager running outside of ros_control_boilerplate.

If it’s intended that only one spawn call is allowed, I didn’t see it in the documentation.

About this issue

Original URL
State: open
Created 7 years ago
Comments: 21 (14 by maintainers)

Most upvoted comments

TL;DR: This is no bug in the controller_manager, but an issue with the boilerplate template.

I went through the code of the ros_control_boilerplate and have found out that it is running its control loop with a ROS timer in conjunction with an AsyncSpinner with two threads.

The service handlers in controller_manager are protected by locks. So if two switch requests come in at the same time, the first one will be run, the second one will block. In case of the boilerplate template these two service calls block both available spinner threads, so the control loop cannot be run. This results in a deadlock because the first service handler waits for the control loop to process the switching.

We could improve the situation a little bit by resolving the deadlock with a timed lock (so it will recover after some time), but the boilerplate controller loop will freeze for a certain time. IMHO it would be best to migrate the boilerplate code to a sleep-based loop.

mathias-luedtke on Dec 16, 2017

And what about a dedicated Callback Queue and AsyncSpinner pair to handle the controller_manager update loop? Would it be ok, or it may suffer the same deadlock problem?

This should work. But I don’t see any benefit, especially if the main ends with ros::waitForShutdown().

mathias-luedtke on May 2, 2018