rclpy: Runtime error when a long async callback is triggered
Bug report
Required Info:
- Operating System:
- Ubuntu 20.04
- Installation type:
- Binaries
- Version or commit hash:
- galactic 1.0.8-1focal.20220209.144341
- DDS implementation:
- Fast-RTPS
- Client library (if applicable):
- rclpy
Steps to reproduce issue
import asyncio
import logging
import rclpy
from std_msgs.msg import String
logging.getLogger('asyncio').setLevel(logging.DEBUG)
async def cb(msg):
print('Received a message:')
await asyncio.sleep(0.1)
print(msg.data)
async def main():
rclpy.init()
node = rclpy.create_node('my_node')
node.create_subscription(String, '/test', cb, 10)
while True:
rclpy.spin_once(node, timeout_sec=0.001)
await asyncio.sleep(0.001)
asyncio.run(main())
Run the subscriber and then run:
ros2 topic pub -1 /test std_msgs/msg/String "data: hello"
Expected behavior
When published to /test, the node should print Received a message:, wait for 0.1 seconds, and then print the message data.
From https://github.com/ros2/rclpy/pull/166#issue-285849213:
Any callback can now be a coroutine function (async def). If the callback is a coroutine and it becomes blocked the executor will move on and resume executing it later. The details of this are handled in the class Task which is a class very similar to asyncio.Task. asyncio can’t be used directly because it is not thread safe.
Actual behavior
Executing <Task pending name='Task-1' coro=<main() running at my_listener.py:19> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f2ed81ddb80>()] created at /usr/lib/python3.8/asyncio/base_events.py:422> cb=[_run_until_complete_cb() at /usr/lib/python3.8/asyncio/base_events.py:184] created at /usr/lib/python3.8/asyncio/base_events.py:595> took 0.199 seconds
Received a message:
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/mostafa/.vscode/extensions/ms-python.python-2022.8.0/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
cli.main()
File "/home/mostafa/.vscode/extensions/ms-python.python-2022.8.0/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
run()
File "/home/mostafa/.vscode/extensions/ms-python.python-2022.8.0/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
File "/usr/lib/python3.8/runpy.py", line 265, in run_path
return _run_module_code(code, init_globals, run_name,
File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "my_listener.py", line 21, in <module>
asyncio.run(main())
File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "my_listener.py", line 18, in main
rclpy.spin_once(node, timeout_sec=0.001)
File "/home/mostafa/dev/github/rclpy/issues/long-await/install/rclpy/lib/python3.8/site-packages/rclpy/__init__.py", line 176, in spin_once
executor.spin_once(timeout_sec=timeout_sec)
File "/home/mostafa/dev/github/rclpy/issues/long-await/install/rclpy/lib/python3.8/site-packages/rclpy/executors.py", line 712, in spin_once
raise handler.exception()
File "/home/mostafa/dev/github/rclpy/issues/long-await/install/rclpy/lib/python3.8/site-packages/rclpy/task.py", line 242, in __call__
self._handler.send(None)
File "/home/mostafa/dev/github/rclpy/issues/long-await/install/rclpy/lib/python3.8/site-packages/rclpy/executors.py", line 418, in handler
await call_coroutine(entity, arg)
File "/home/mostafa/dev/github/rclpy/issues/long-await/install/rclpy/lib/python3.8/site-packages/rclpy/executors.py", line 343, in _execute_subscription
await await_or_execute(sub.callback, msg)
File "/home/mostafa/dev/github/rclpy/issues/long-await/install/rclpy/lib/python3.8/site-packages/rclpy/executors.py", line 104, in await_or_execute
return await callback(*args)
File "my_listener.py", line 10, in cb
await asyncio.sleep(0.1)
File "/usr/lib/python3.8/asyncio/tasks.py", line 659, in sleep
return await future
RuntimeError: await wasn't used with future
Additional information
Observation
Reducing the timeout in the asyncio.sleep line makes the issue disappear. In the example on provided, on my system, a timeout of ~0.002 seconds or smaller works fine.
Guess
This piece might play a role:
My theory is that self._handler.send(None) is non-blocking, causing the line self._executing = False to be executed before the async callback is finished (while self.done is still False). So this causes the function to be assumed as paused (kind of like a yielded generator), so the already-running coro object gets called again by this piece:
This causes some kind of clash, hence RuntimeError: await wasn't used with future.
So as a quick test, I commented out the self._executing = False line. This caused the second call to be avoided, (so no runtime error anymore), but the callback didn’t finish (the message wasn’t printed out).
Another minimal example
rclpy practically does something like this:
import asyncio
async def cb():
print('a')
await asyncio.sleep(1)
print('b')
async def main():
_handler = cb()
while True:
_handler.send(None)
asyncio.run(main())
Which also throws RuntimeError: await wasn't used with future.
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 2
- Comments: 15 (4 by maintainers)
Did you mean something like this @charlielito?
The output should be:
Interestingly, I’ve been using a decorator that takes an asyncio function and makes it compatible with rclpy. It goes like this:
This mimics rclpy awaitable behavior:
I was using this as a temporary workaround until an upstream fix comes out, but maybe it could be the upstream fix?
That decorator assumes we’re only dealing with asyncio awaitables. To handle both rclpy and asyncio awaiables in one, we could improve the decorator to check if future is None:
Now rclpy.Task can automatically apply this transformation to all the coroutine functions and it turns them into the rclpy-style.
What are your thoughts? Should I update the PR to use this concept?
I was not aware that using a
ReentrantCallbackGroupwould work! Thanks for pointing that out. I also didn’t notice theself.timer.cancel()part.I don’t think recreating the callback group here is an issue given you’re trying to have it execute in parallel, but I believe if you recreate the group each time with a
MutuallyExclusiveCallbackGroupas opposed to storing it as a class instance variable, you might get unexpected behaviors.Though the PR to make
ReentrantCallbackGroupdidn’t make it through, it would be good to have an asyncio + rclpy guide, as it seems like people are rolling their own quite frequently and there’s no single agreed upon approach AFAIK. I took the liberty of modifying your example to show parallel execution:output:
@sloretz Curious if you agree with this approach or would suggest an alternative. I’d be happy to create a docs guide around this.
The decorator is a cool idea. One problem could be if if
asyncio’ssend()expectations could vary by cPython version. They have freedom to change it since it’s an implementation detail.There’s another idea worth looking at.
rclpy’s problem withasynciois pretty similar toasyncio’s problem with concurrent.futures.Future. They solve it by providing asyncio.wrap_future which wraps aconcurrent.futures.Futureobject with an asyncio one. It works by adding done calbacks to make one future complete the other one. It seems like this can be implemented with only public APIs.Say we had an api
rclpy.wrap_asyncio_future()which accepted anasynciofuture and returned anrclpyone. It could use asyncio.isfuture to make sure the argument is an asyncio future. Then it could create anrclpy.Futureto be used as the return value. The method would then get the asyncio event loop with future.get_loop(), and then use loop.call_soon_thread_safe to call asyncio.future.add_done_callback(). The done callback would complete therclpy.Futurewhen theasyncioone was completed.We could integrate it into
Taskby checking the return value of the firstsend(). If the return value is an asyncio future, then the task could wrap it with an rclpy one and await it instead.It’s still important to note somewhere that one must use a single threaded executor, and
spin_oncehas to be called repeatedly in an asyncio event loop in a non-blocking way. We could probably make it more convenient with an awaitable spin function, but that’s probably a separate discussion.That works! Thanks!
Folks, how would you implement asyncio timeouts with a rclpy Future to be able to detect a timeouts if something never returns (like a service or action async call)