pytest-xdist: getting failures on slower machines since 3.3.0 release

hi -

i dont have a local reproducer here but since pytest-xdist 3.3.0 was released SQLAlchemy is seeing failures on github action nodes that use the virtualized “arm64” architecture - example failures:

https://github.com/sqlalchemy/sqlalchemy/actions/runs/4962771769/jobs/8881240949

https://github.com/sqlalchemy/sqlalchemy/actions/runs/4964655654/jobs/8884945438

https://github.com/sqlalchemy/sqlalchemy/actions/runs/4973604221/jobs/8899553346

stack traces look like this:

=========================== sqlalchemy installation ============================
SQLAlchemy 2.0.14 (no user site)
Path: /github/workspace/.tox/github-nocext/lib/python3.10/site-packages/sqlalchemy/__init__.py
compiled extension not enabled; cannot import name 'collections' from 'sqlalchemy.cyextension' (/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/sqlalchemy/cyextension/__init__.py)
============================= test session starts ==============================
platform linux -- Python 3.10.11, pytest-7.3.1, pluggy-1.0.0
cachedir: .tox/github-nocext/.pytest_cache
rootdir: /github/workspace
configfile: pyproject.toml
plugins: xdist-3.3.0
created: 2/2 workers
INTERNALERROR> def worker_internal_error(self, node, formatted_error):
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>     
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E               File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 259, in get_event_loop
INTERNALERROR> E                 return asyncio.get_running_loop()
INTERNALERROR> E             RuntimeError: no running event loop
INTERNALERROR> E             
INTERNALERROR> E             During handling of the above exception, another exception occurred:
INTERNALERROR> E             
INTERNALERROR> E             Traceback (most recent call last):
INTERNALERROR> E               File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/_pytest/main.py", line 267, in wrap_session
INTERNALERROR> E                 config.hook.pytest_sessionstart(session=session)
INTERNALERROR> E               File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR> E               File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR> E                 return outcome.get_result()
INTERNALERROR> E               File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR> E                 raise ex[1].with_traceback(ex[2])
INTERNALERROR> E               File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/github/workspace/test/../lib/sqlalchemy/testing/plugin/pytestplugin.py", line 163, in pytest_sessionstart
INTERNALERROR> E                 asyncio._assume_async(plugin_base.post_begin)
INTERNALERROR> E               File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/sqlalchemy/testing/asyncio.py", line 53, in _assume_async
INTERNALERROR> E                 return _util_async_run(fn, *args, **kwargs)
INTERNALERROR> E               File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 243, in _util_async_run
INTERNALERROR> E                 loop = get_event_loop()
INTERNALERROR> E               File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 261, in get_event_loop
INTERNALERROR> E                 return asyncio.get_event_loop_policy().get_event_loop()
INTERNALERROR> E               File "/opt/_internal/cpython-3.10.11/lib/python3.10/asyncio/events.py", line 656, in get_event_loop
INTERNALERROR> E                 raise RuntimeError('There is no current event loop in thread %r.'
INTERNALERROR> E             RuntimeError: There is no current event loop in thread 'Dummy-1'.
INTERNALERROR> E           assert False
INTERNALERROR> 
INTERNALERROR> .tox/github-nocext/lib/python3.10/site-packages/xdist/dsession.py:197: AssertionError
[gw1] node down: Not properly terminated

replacing crashed worker gw1
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/_pytest/main.py", line 323, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/xdist/dsession.py", line 122, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/xdist/dsession.py", line 145, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/github/workspace/.tox/github-nocext/lib/python3.10/site-packages/xdist/dsession.py", line 235, in worker_errordown
INTERNALERROR>     self._active_nodes.remove(node)
INTERNALERROR> KeyError: <WorkerController gw1>

About this issue

Original URL
State: closed
Created a year ago
Comments: 27 (10 by maintainers)

Commits related to this issue

block pytest-xdist 3.3.0 Change-Id: I9c68ef747ad4c87b73751158c5b7966c28816ec8 References: https://github.com/pytest-dev/pytest-xdist/issues/907 — committed to sqlalchemy/sqlalchemy by zzzeek a year ago
Avoid unnecessary rinfo calls after creating gateways Hopefully this fixes #907, as seems this is the only change in #901 which is somehow related. — committed to nicoddemus/pytest-xdist by nicoddemus a year ago
Avoid unnecessary rinfo calls after creating gateways Hopefully this fixes #907, as seems this is the only change in #901 which is somehow related. — committed to nicoddemus/pytest-xdist by nicoddemus a year ago
Avoid unnecessary rinfo calls after creating gateways (#909) Hopefully this fixes #907, as seems this is the only change in #901 which is somehow related. --------- Co-authored-by: Ronny Pfann... — committed to pytest-dev/pytest-xdist by nicoddemus a year ago
⬆️ Update dependency pytest-xdist to v3.3.1 🔒 (#103) [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | ... — committed to ixm-one/pytest-cmake-presets by renovate[bot] a year ago
Bump pytest-xdist from 3.3.0 to 3.3.1 (#182) Bumps [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) from 3.3.0 to 3.3.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="... — committed to luqasz/librouteros by github-actions[bot] a year ago
Bump pytest-xdist from 3.3.0 to 3.3.1 (#1965) Bumps [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) from 3.3.0 to 3.3.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href=... — committed to software-mansion/protostar by dependabot[bot] a year ago
chore(deps): update dependency pytest-xdist to v3 (#116) [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Pa... — committed to fluencelabs/spell by renovate[bot] a year ago
build(deps-dev): Bump the dev group with 4 updates (#322) Bumps the dev group with 4 updates: [black](https://github.com/psf/black), [pytest-cov](https://github.com/pytest-dev/pytest-cov), [pytest... — committed to ryankanno/cookiecutter-py by dependabot[bot] 8 months ago
[MNT] [Dependabot](deps-dev): Update pytest-xdist requirement from <3.4,>=3.3 to >=3.3,<3.5 (#5551) Updates the requirements on [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) to permit t... — committed to sktime/sktime by dependabot[bot] 7 months ago
Update pytest-xdist requirement from <3.4,>=2.1.0 to >=2.1.0,<3.6 (#6517) Updates the requirements on [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) to permit the latest version. <deta... — committed to mitmproxy/mitmproxy by dependabot[bot] 7 months ago

Most upvoted comments

Will do! I’ll do a full investigation when I’m off mobile

On Monday, May 22, 2023, Bruno Oliveira @.***> wrote:

-v will currently force that extra rinfo call which might trigger the problem (but that was also true em 3.2.1).

See if removing -v helps.

— Reply to this email directly, view it on GitHub https://github.com/pytest-dev/pytest-xdist/issues/907#issuecomment-1558307345, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATLPNBHI2MQ6P34K73U2J43XHQEMNANCNFSM6AAAAAAYCSTYHA . You are receiving this because you were mentioned.Message ID: @.***>

david-engelmann on May 23, 2023

all good for 3.3.1, thanks! 🚀 🐰 🎉

zzzeek on May 20, 2023

3.3.1 just released to PyPI. 🎉

Thanks for the report and help with testing folks!

nicoddemus on May 19, 2023