colcon-core: asyncio error messages after CancelledError with Python 3.5 / Ubuntu 16.04

Our environment

  1. Ubuntu 16.04
  2. ROS 2 Bouncy
  3. colcon version-check
colcon-argcomplete 0.2.2: newer version available (0.2.3)
colcon-bash 0.3.1: up-to-date
colcon-cmake 0.2.4: up-to-date
colcon-core 0.3.13: newer version available (0.3.14)
colcon-defaults 0.2.1: up-to-date
colcon-devtools 0.2.1: up-to-date
colcon-lcov-result 0.2.0: up-to-date
colcon-library-path 0.2.0: up-to-date
colcon-metadata 0.2.1: up-to-date
colcon-mixin 0.1.0: newer version available (0.1.1)
colcon-notification 0.2.5: up-to-date
colcon-output 0.2.3: up-to-date
colcon-package-information 0.2.1: up-to-date
colcon-package-selection 0.2.1: up-to-date
colcon-parallel-executor 0.2.2: up-to-date
colcon-pkg-config 0.1.0: up-to-date
colcon-powershell 0.3.3: up-to-date
colcon-python-setup-py 0.2.1: up-to-date
colcon-recursive-crawl 0.2.0: up-to-date
colcon-ros 0.3.5: up-to-date
colcon-test-result 0.3.0: up-to-date
colcon-zsh 0.3.1: up-to-date
  1. build command: colcon build --merge-install

After we had a code error like this:

error: ‘m_pcl_sub’ was not declared in this scope
     const auto & loaned_message = m_pcl_sub->take();

colcon printed out a whole bunch of errors:

Starting >>> euclidean_cluster                                                                                
--- stderr: apex_tf                                                                                           
/home/user/1816/apex_ws/src/apex_app/common/apex_tf/src/local_tf.cpp: In member function ‘virtual void apex_app::common::apex_tf::LocalTf::TfSubTask::task_function()’:
/home/user/1816/apex_ws/src/apex_app/common/apex_tf/src/local_tf.cpp:142:35: error: ‘m_pcl_sub’ was not declared in this scope
     const auto & loaned_message = m_pcl_sub->take();
                                   ^
make[2]: *** [CMakeFiles/apex_tf.dir/src/local_tf.cpp.o] Error 1
make[1]: *** [CMakeFiles/apex_tf.dir/all] Error 2
make: *** [all] Error 2
---
Failed   <<< apex_tf	[ Exited with code 2 ]
Finished <<< lifecycle [9.53s]                                                                
Aborted  <<< velodyne_driver
Aborted  <<< euclidean_cluster
Aborted  <<< lidar_integration
Aborted  <<< motion_model
Aborted  <<< performance_test
Aborted  <<< udp_driver

Summary: 169 packages finished [1min 52s]
  1 package failed: apex_tf
  6 packages aborted: euclidean_cluster lidar_integration motion_model performance_test udp_driver velodyne_driver
  1 package had stderr output: apex_tf
  25 packages not processed
[116.165s] ERROR:concurrent.futures:exception calling callback for <Future at 0x7f7aeed4a9e8 state=finished returned NoneType>
Traceback (most recent call last):
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 297, in _invoke_callbacks
    callback(self)
  File "/usr/lib/python3.5/asyncio/futures.py", line 442, in _call_set_state
    dest_loop.call_soon_threadsafe(_set_state, destination, source)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 532, in call_soon_threadsafe
    handle = self._call_soon(callback, args)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
    self._check_closed()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
[116.166s] ERROR:concurrent.futures:exception calling callback for <Future at 0x7f7aeed4a128 state=finished returned NoneType>
Traceback (most recent call last):
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 297, in _invoke_callbacks
    callback(self)
  File "/usr/lib/python3.5/asyncio/futures.py", line 442, in _call_set_state
    dest_loop.call_soon_threadsafe(_set_state, destination, source)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 532, in call_soon_threadsafe
    handle = self._call_soon(callback, args)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
    self._check_closed()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
[118.025s] ERROR:concurrent.futures:exception calling callback for <Future at 0x7f7aeee97198 state=finished returned NoneType>
Traceback (most recent call last):
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 297, in _invoke_callbacks
    callback(self)
  File "/usr/lib/python3.5/asyncio/futures.py", line 442, in _call_set_state
    dest_loop.call_soon_threadsafe(_set_state, destination, source)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 532, in call_soon_threadsafe
    handle = self._call_soon(callback, args)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
    self._check_closed()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
[118.026s] ERROR:concurrent.futures:exception calling callback for <Future at 0x7f7aeed1bfd0 state=finished returned NoneType>
Traceback (most recent call last):
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 297, in _invoke_callbacks
    callback(self)
  File "/usr/lib/python3.5/asyncio/futures.py", line 442, in _call_set_state
    dest_loop.call_soon_threadsafe(_set_state, destination, source)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 532, in call_soon_threadsafe
    handle = self._call_soon(callback, args)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
    self._check_closed()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
[118.026s] ERROR:concurrent.futures:exception calling callback for <Future at 0x7f7aeeb2fbe0 state=finished returned NoneType>
Traceback (most recent call last):
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 297, in _invoke_callbacks
    callback(self)
  File "/usr/lib/python3.5/asyncio/futures.py", line 442, in _call_set_state
    dest_loop.call_soon_threadsafe(_set_state, destination, source)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 532, in call_soon_threadsafe
    handle = self._call_soon(callback, args)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
    self._check_closed()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
[118.027s] ERROR:concurrent.futures:exception calling callback for <Future at 0x7f7aeeb2f780 state=finished returned NoneType>
Traceback (most recent call last):
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 297, in _invoke_callbacks
    callback(self)
  File "/usr/lib/python3.5/asyncio/futures.py", line 442, in _call_set_state
    dest_loop.call_soon_threadsafe(_set_state, destination, source)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 532, in call_soon_threadsafe
    handle = self._call_soon(callback, args)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
    self._check_closed()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
[118.869s] ERROR:concurrent.futures:exception calling callback for <Future at 0x7f7aeeaf1518 state=finished returned NoneType>
Traceback (most recent call last):
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 297, in _invoke_callbacks
    callback(self)
  File "/usr/lib/python3.5/asyncio/futures.py", line 442, in _call_set_state
    dest_loop.call_soon_threadsafe(_set_state, destination, source)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 532, in call_soon_threadsafe
    handle = self._call_soon(callback, args)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
    self._check_closed()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
[118.869s] ERROR:concurrent.futures:exception calling callback for <Future at 0x7f7aeeaf1a58 state=finished returned NoneType>
Traceback (most recent call last):
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 297, in _invoke_callbacks
    callback(self)
  File "/usr/lib/python3.5/asyncio/futures.py", line 442, in _call_set_state
    dest_loop.call_soon_threadsafe(_set_state, destination, source)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 532, in call_soon_threadsafe
    handle = self._call_soon(callback, args)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
    self._check_closed()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
[138.736s] ERROR:concurrent.futures:exception calling callback for <Future at 0x7f7aeead00f0 state=finished returned NoneType>
Traceback (most recent call last):
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 297, in _invoke_callbacks
    callback(self)
  File "/usr/lib/python3.5/asyncio/futures.py", line 442, in _call_set_state
    dest_loop.call_soon_threadsafe(_set_state, destination, source)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 532, in call_soon_threadsafe
    handle = self._call_soon(callback, args)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
    self._check_closed()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
[138.737s] ERROR:concurrent.futures:exception calling callback for <Future at 0x7f7aeead09e8 state=finished returned NoneType>
Traceback (most recent call last):
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 297, in _invoke_callbacks
    callback(self)
  File "/usr/lib/python3.5/asyncio/futures.py", line 442, in _call_set_state
    dest_loop.call_soon_threadsafe(_set_state, destination, source)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 532, in call_soon_threadsafe
    handle = self._call_soon(callback, args)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
    self._check_closed()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
[138.754s] ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<_fd2callback() done, defined at /usr/local/lib/python3.5/dist-packages/colcon_core/subprocess.py:220> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/lib/python3.5/asyncio/futures.py:431, Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>
[138.755s] ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<_fd2callback() done, defined at /usr/local/lib/python3.5/dist-packages/colcon_core/subprocess.py:220> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/lib/python3.5/asyncio/futures.py:431, Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>
[138.755s] ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<_fd2callback() done, defined at /usr/local/lib/python3.5/dist-packages/colcon_core/subprocess.py:220> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/lib/python3.5/asyncio/futures.py:431, Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>
[138.755s] ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<_fd2callback() done, defined at /usr/local/lib/python3.5/dist-packages/colcon_core/subprocess.py:220> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/lib/python3.5/asyncio/futures.py:431, Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>
[138.755s] ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<_fd2callback() done, defined at /usr/local/lib/python3.5/dist-packages/colcon_core/subprocess.py:220> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/lib/python3.5/asyncio/futures.py:431, Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>
[138.755s] ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<_fd2callback() done, defined at /usr/local/lib/python3.5/dist-packages/colcon_core/subprocess.py:220> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/lib/python3.5/asyncio/futures.py:431, Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>
[138.755s] ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<_fd2callback() done, defined at /usr/local/lib/python3.5/dist-packages/colcon_core/subprocess.py:220> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/lib/python3.5/asyncio/futures.py:431, Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>
[138.755s] ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<_fd2callback() done, defined at /usr/local/lib/python3.5/dist-packages/colcon_core/subprocess.py:220> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/lib/python3.5/asyncio/futures.py:431, Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>
[138.755s] ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<_fd2callback() done, defined at /usr/local/lib/python3.5/dist-packages/colcon_core/subprocess.py:220> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/lib/python3.5/asyncio/futures.py:431, Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>
[138.756s] ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<_fd2callback() done, defined at /usr/local/lib/python3.5/dist-packages/colcon_core/subprocess.py:220> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/lib/python3.5/asyncio/futures.py:431, Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>

Google search didn’t turn out anything similar. Any idea where to start digging?

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 16 (13 by maintainers)

Most upvoted comments

Yup, just re-tested now and the issue on 16.04 is now fixed. Thanks!

No worries! We probably won’t be looking into it any further— this just clarifies for us that a colcon migration is not in the cards until we’re no longer targeting 16.04.

I can reproduce the error messages on Ubuntu 16.04 / Python 3.5.

The asyncio error messages appear after the invocation has finished and shouldn’t affect the behavior / outcome of the invocation. Since the problem exists only with Python 3.5 this might be a problem within asyncio itself.

The code running the coroutines mentioned in the error messages (_fd2callback) is correctly finishing all of them, see: https://github.com/colcon/colcon-core/blob/ccf68192a79f4f41a385c9d19cf090d79e49f05f/colcon_core/subprocess.py#L179-L184

I will keep this ticket open but I don’t expect that I will be able to come up with a patch for it. Hence I assigned the label “help wanted”. I am more than happy to review contributed patches which improve the ugly output.