rclcpp: Executor callbacks are no longer in a predictable order
Bug report
Required Info:
- Operating System:
- Ubuntu 20.04
- Installation type:
- Source
- Version or commit hash:
- DDS implementation:
- Fast-RTPS
- Client library (if applicable):
- rclcpp
Steps to reproduce issue
- Build the https://github.com/boschresearch/ros2_response_time_analysis project against ROS2 Jazzy (note that some minimal changes are required).
- Observe that the results of running the test show that the callbacks are executed in an arbitrary order as compared to the previous ordering which was with respect to the order in which the subscriptions/services were added to the node.
Expected behavior
With ROS2 Eloquent through Iron, the callbacks all executed in the following order, where the ordering was defined by the order in which the subscriptions/services were added to the node.
Actual behavior
With ROS2 Jazzy, the ordering is now arbitrary and not related to the order in which messages were received or the order in which subscriptions were added to the node. The execution order of timers also appears to be affected.
NOTE: While the low, medium, and high subscriptions are processed in the same order that they are transmitted, I believe this to be a fluke as the service and timer callbacks are not processed in the order they were registered or sent. For reference, the callbacks are registered in the following order:
- high
- medium
- low
- srv_high
- srv_medium
- srv_low
- timers (as directed by the test)
And the topics/services are sent in the following order:
- low
- medium
- high
- srv_high
- srv_low
- low
- medium
- high
- srv_high
- srv_low
- srv_medium
- srv_medium
- high
With timers set to expire at 200, 200, 2300, and 2300 milliseconds into the test.
Additional information
I believe this change is a result of #2142 due to the EntityCollection class being based on std::unordered_map where the key is the handle returned by rcl for the timer/subscription/etc. The entities are then added to the wait set in whatever order they are iterated through by the std::unordered_map and thus processed in that order. I haven’t had the opportunity to test it, but I believe the ordering could be different from system to system or even run to run as the rcl handle values change.
The default AllocatorMemoryStrategy used in previous versions of rclcpp utilized std:vector for storing rcl handles, which maintained the order in which they were added to the node.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 2
- Comments: 25 (6 by maintainers)
Oops. I accidentally replied using my personal account.
I’ve created https://github.com/ros2/rclcpp/issues/2535 for the timer-specific issue that has been discussed.
The
EventsExecutorindeed allows to define custom “queue” implementations. The queue class defines the ordering in which ready entities are processed in one iteration of “spin”.We currently only have a FIFO queue, based on the order in which entities are triggered. It’s possible to define queues with fixed order of entities (either arbitrary or user-defined). Custom queues can live outside of rclcpp, in the users workspaces.
We should consider if/when we will deprecate the waitset-based executors like
SingleThreadedExecutordue to their problems (mostly performance).For me, the concern is that the executor has behaved in the same manner likely since its inception and certainly across many releases of ROS2, and there is published work that describes this behavior. It’s very likely that some users of the default executor have made use of this behavior and will break when migrating to Jazzy.
It’s also concerning to me that, while the previous behavior was not necessarily defined, it was at least somewhat deterministic in that whatever callbacks were ready in the wait set were executed in the order that the callbacks were registered. The new approach is completely non-deterministic because the order is based on the hash of the memory addresses of the
rclhandles for the waitables (e.g. the keys of the unordered maps). This seems like a step backwards to me since determinism is something that has been discussed as a need for users of ROS2.This change certainly caught me off guard and caused me to spend a few hours today understanding what changed, which is why I thought it was important to raise it as a potential issue.
Having a fixed order of callbacks is a design flaw of the
SingleThreadedExecutor. IMO users shouldn’t rely on it.Callbacks should be part of an event-driven pipeline and so they should be invoked in the order in which they are triggered, this is what happens with other executors such as the
EventsExecutor.