pytest-xdist: `pytest-xdist` breaks asyncio-based code expecting to be run in the main thread

When asyncio-based software needs to register signal handlers, calling add_signal_handler() only works when the current thread is main and raises a RuntimeError otherwise.

When I first tried integrating pytest-xdist into aiohttp’s test suite a few years ago, we’ve faced a problem that about 15% of pytest invocations would hit this and had to disable the plugin:

• https://github.com/aio-libs/aiohttp/pull/3419
• https://github.com/aio-libs/aiohttp/issues/3450

The reproducer is to clone aio-libs/aiohttp, add -n auto to addopts somewhere in setup.cfg, and re-run make test until you see the traceback. On aiohttp side, it’s coming from the watcher.attach_loop() call invoked from the pytest plugin @ https://github.com/aio-libs/aiohttp/blob/742a8b6/aiohttp/pytest_plugin.py#L161.

Now that I’ve had some time to try to debug what’s happening, I’ve stuck a debugger right before the line that raises RuntimeError and confirmed that the thread wasn’t main:

>>> import threading
>>> threading.enumerate()
[<_MainThread(MainThread, started 140666216200000)>, <_DummyThread(Dummy-1, started daemon 140666196751936)>]
>>> threading.current_thread()
<_DummyThread(Dummy-1, started daemon                 
140666196751936)>

@asvetlov It should be possible to fix this on the asyncio side with ThreadedChildWatcher (https://bugs.python.org/issue35621 / https://github.com/python/cpython/pull/14344) but it only appeared in the stdlib since Python 3.8: https://stackoverflow.com/a/58614689/595220.

It’s hard to pinpoint where that thread is coming from but it appears to be execnet’s fault: https://github.com/pytest-dev/pytest-xdist/blob/c5fadcd734b008251fbaa438e0806235d9bd081b/src/xdist/workermanage.py#L41 + https://github.com/pytest-dev/execnet/blob/116637ab183bc078647f281b52e662650e8178e7/execnet/multi.py#L27.

@nicoddemus do you have any insight into this? I’m rather lost at this point. Is there any way to avoid this race condition?