pytest-asyncio: Breaking change in 0.23.*

I’m glad to see more people are experiencing this issue. Apologies, but the latest version is an absolute nightmare. After 5 hours of trying to adjust my environment I still can’t get it working. I have downgraded to pytest-asyncio 0.21.1 and will try to hold on to it as long as possible. I hope there will be some future changes to make this migration easier, but for now it’s a huge headache.

@seifertm while I deeply appreciate your work, which is crucial for all python and pytest users dealing with async tests, I don’t understand why you had to change the way the event loop is set up: the previous way worked just fine.

Please consider restoring the previous behaviour.

As for use cases, a few people provided their setups and needs in #657 (and this was my comment)

@seifertm I’d like to have only one loop for all fixtures and tests, without additional decorators to all tests and fixtures. We have thousands of tests in hundreds of services where all tests and fixtures share one loop and it is crucial for them. Is there any workaround to emulate the old behaviour?

This way https://pytest-asyncio.readthedocs.io/en/v0.23.2/how-to-guides/run_session_tests_in_same_loop.html does not consider fixtures at all(

I’m afraid I’m not in a position to judge the approach: I don’t know enough about the previous design and the desired long-term design. I can see how switching from event loop to even loop policy still allows folks to switch to alternative event loop implementations (we, for example, use uvloop) but also how it makes it harder to define custom scopes, or even different scopes depending on the test or test package.

IMHO the best solution is the one that makes the end developer write as little code as possible, and only code that is strictly related to what the dev wants to do. With this in mind, overriding the event_loop fixture was good enough: it’s 5 lines of code written in the proper conftest file. Markers instead are not good, as it’s lots of boilerplate for specifying the same setting all over a test suite: makers should better be used to specify exceptions, not rules.

Fixtures in v0.23 generally behave like tests and choose the event loop of the fixture scope. That means if a fixture has session scope it will run in the session-wide loop.

Sorry, but I don’t understand, I have not only session scoped fixtures but also module scoped fixtures, and they should use the same event loop as session scoped fixtures. Could you please describe how to achieve it?

We have thousands of tests in hundreds of services where all tests and fixtures share one loop and it is crucial for them.

I don’t think this use case was considered during the development of v0.22 and v0.23. Can you explain why you need all tests and fixtures to run the same loop?

We write blackbox tests for microservices using pytest and pytest-asyncio. Some session scoped fixtures for example create database connection pool, which all tests can use to check database state. Another session scoped fixture in background monitors logs of subprocesses (instances of application, that we test) and captures these logs to some list, which tests can check. These subprocesses can be started by any fixture and test as async context manager. And obviously, subprocess (asyncio.subprocess) should be started with the same loop as fixture that captures logs from it. And we have way more such examples.

I think the real problem is that the migration process is not clear. I would have expected that removing the event_loop fixture would be enough with asyncio_mode = auto, but it’s not.

Here is a minimal reproducible example (with GitHub CI!) of a real world application that I can’t migrate to 0.23 so far: https://github.com/ramnes/pytest-asyncio-706

@seifertm Any hint on how something like this should be migrated? (PR welcome on that repository.) If it’s not possible to migrate, then this would be the real issue: it wouldn’t be a breaking change but a loss of functionality. Otherwise we’ll probably have a few bits to add to the documentation here. 😃

I confirm, we use similar setup with 2 session level fixtures (one to redefine event loop, another for our own purposes), tests don’t work anymore, complain either about “The future belongs to a different loop than the one specified as the loop argument” or “Event loop is closed”.

I think the real problem is that the migration process is not clear. I would have expected that removing the event_loop fixture would be enough with asyncio_mode = auto, but it’s not.

Here is a minimal reproducible example (with GitHub CI!) of a real world application that I can’t migrate to 0.23 so far: https://github.com/ramnes/pytest-asyncio-706

@seifertm Any hint on how something like this should be migrated? (PR welcome on that repository.) If it’s not possible to migrate, then this would be the real issue: it wouldn’t be a breaking change but a loss of functionality. Otherwise we’ll probably have a few bits to add to the documentation here. 😃

@ramnes Thanks for the example project. I think this is the migration experience we should be aiming for.

The issue is that fixture scope is currently bound to loop scope. That means a session-scoped fixture will always be executed in a session-scoped loop, which is apparently a blocker for many users.

What does everyone think about introducing a separate loop_scope argument to pytest_asyncio.fixture, in order to control the fixture scope and loop scope separately? For example:

# Fixture runs once per session, but runs in the default event loop (=function-scoped loop)
@pytest_asyncio.fixture(scope="session")
async def my_fixture():
    ...

# Fixture runs once per session and runs in a session-scoped loop
@pytest_asyncio.fixture(scope="session", loop_scope="session")
async def my_fixture():
    ...

It says “This release is backwards-compatible with v0.21. Changes are non-breaking, unless you upgrade from v0.22.”

Can you explain why you need all tests and fixtures to run the same loop?

Here’s an example that’s similar to our tests that I don’t think can be upgraded to 0.23. Essentially, we have module-level async fixtures for setting up a relatively expensive Application, and function-scoped fixtures for small customizations to the app that are selected on a per-test basis. The function-scoped fixtures and tests need the same loop as the app because the application’s loop must be running to handle the async operations in the function-scoped fixtures and the tests themselves.

import asyncio
import pytest

class App:
    pass

@pytest.fixture(scope="module")
async def app():
    # create expensive app, once per module
    app = App()
    app.event_loop = asyncio.get_running_loop()
    yield app
    # app cleanup

@pytest.fixture(scope="function")
async def app_plugin(app):
    # enable a plugin (involves async operations on app's loop)
    print("await enable_plugin(app)")
    yield
    print("await disable_plugin(app)")
    assert app.event_loop.is_running()

@pytest.mark.asyncio(scope="module")
async def test_loops(app, app_plugin):
    print("await async_app_operation")
    assert app.event_loop is asyncio.get_running_loop()

Both asserts fail because both the function-scoped fixture (expected) and the test (unexpected) run with the function_scoped event loop (@pytest.mark.asyncio(scope="module") is ignored on the test due to the inclusion of the function-scoped fixture, which I assume is not intended? If the function-scope fixture is removed, the module-scoped event loop is used.

I believe @seifertm’s proposal of an explicit loop_scope decorator for async fixtures would address our case just fine.

If we wanted to workaround the issue today, I believe we could replace all of our function-scoped async fixtures with sync fixtures that use app.event_loop.run_until_complete(...).

What we really want is to recover the behavior of “everybody please use the module-level event loop”, so whatever path to that behavior works best for pytest-asyncio ought to be fine for us. In our case, all of our function-level async fixtures depend on our module-level async fixture, so it is conceivable that the situation is detectable, but I’m sure an explicit decorator on every fixture is easier to handle, if a bit more tedious to use. A single flag for defining a lowest or default level of event-loop creation would be simpler for us to use (e.g. asyncio-loop-min-level = "module" would mean function-level fixtures and tests use module-level event loop by default but not session, etc.), if that’s practical.

It also seems reasonable to me to expect a function-scoped fixture to run with the same loop as the function itself, so if I e.g. have @pytest.mark.asyncio(scope="module") on a test, it makes sense to use that scope (as the default, at least) on all function-scoped async fixtures used in the test. I’m not sure if that information is readily available at the right time, though, so I’d understand if it’s not practical to implement.

hi @seifertm , thank you for your reply and references.

If I got this right, pytest-asyncio 0.21.1 is perfect (get_event_loop deprecation aside), but you’re concerned by the mess folks do in an overridden event_loop fixture, you want to prevent them from shooting themselves in the foot, and would rather give them specific interfaces for fiddling with low-level stuff.

I’ll say something that might sound counter-intuitive: I don’t think you should care.

I think this library should perfectly do what it does, document where and how it can be customised, and point folks to docs when they abuse certain fixtures or fiddle with the library internals.

The documentation already provides a clear example of what is needed to customise the default event_loop fixture. If folks start adding tons of unnecessary stuff to it, it’s not this library’s fault, and it’s not something this library should handle. To the extreme that it should not be this library’s responsibility to close an unclosed event loop: if anything, it should raise an exception when it catches that the event loop hasn’t been closed.

If folks think it is necessary to do all the stuff they do in their event_loop fixtures, I think they’ll find a way to do it regardless of the constraints imposed by the library. Especially with a language like python where monkey patching is so easy.

Fixing wrong customisations on behalf of devs makes your work harder, as the library starts becoming a knight fighting off dragons. And there will always be dragons.

After upgrade to v 0.23 error in teardown:

@pytest.fixture
def server():
  from main import _create_fastapy_server
  app = _create_fastapy_server()
  return app

@pytest_asyncio.fixture
async def client_async(server):
    app = server
    async with (
      app.router.lifespan_context(app),
      AsyncClient(app=app, base_url="http://testserver") as client
    ):
      yield client

@pytest.mark.asyncio
async def test_server(client_async):
  """Start - Stop"""

Output:

tests/test_server.py:63 (test_server)
def finalizer() -> None:
        """Yield again, to finalize."""
    
        async def async_finalizer() -> None:
            try:
                await gen_obj.__anext__()
            except StopAsyncIteration:
                pass
            else:
                msg = "Async generator fixture didn't stop."
                msg += "Yield only once."
                raise ValueError(msg)
    
>       event_loop.run_until_complete(async_finalizer())

/home/tonal/.pyenv/versions/epool-smsserver/lib/python3.11/site-packages/pytest_asyncio/plugin.py:336: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/home/tonal/.pyenv/versions/3.11.6/lib/python3.11/asyncio/base_events.py:628: in run_until_complete
    self._check_closed()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_UnixSelectorEventLoop running=False closed=True debug=True>

    def _check_closed(self):
        if self._closed:
>           raise RuntimeError('Event loop is closed')
E           RuntimeError: Event loop is closed

/home/tonal/.pyenv/versions/3.11.6/lib/python3.11/asyncio/base_events.py:519: RuntimeError
sys:1: RuntimeWarning: coroutine '_wrap_asyncgen_fixture.<locals>._asyncgen_fixture_wrapper.<locals>.finalizer.<locals>.async_finalizer' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
/home/tonal/.pyenv/versions/epool-smsserver/lib/python3.11/site-packages/aiohttp/client.py:357: ResourceWarning: Unclosed client session <aiohttp.client.ClientSession object at 0x7fbfa55246f0>
  _warnings.warn(
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/home/tonal/.pyenv/versions/epool-smsserver/lib/python3.11/site-packages/aiohttp/connector.py:277: ResourceWarning: Unclosed connector <aiohttp.connector.TCPConnector object at 0x7fbfa5b67ad0>
  _warnings.warn(f"Unclosed connector {self!r}", ResourceWarning, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/home/tonal/.pyenv/versions/3.11.6/lib/python3.11/asyncio/sslproto.py:119: ResourceWarning: unclosed transport <asyncio._SSLProtocolTransport object>
  _warnings.warn(
/home/tonal/.pyenv/versions/3.11.6/lib/python3.11/asyncio/selector_events.py:864: ResourceWarning: unclosed transport <_SelectorSocketTransport fd=17>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback

Yes, we have already downgraded to 0.21.1. I don’t think it’s gonna be a problem for us for a long time (at least until Python 3.13).

I’ll think of a way to control the fixture scope independently of the event loop scope.

Thank you! Looking forward to the news.

The version 0.23.0 changelog is already mentioning the breaking change: https://github.com/pytest-dev/pytest-asyncio/releases/tag/v0.23.0

I went through the same, the new way to do it can be seen in this PR https://github.com/pytest-dev/pytest-asyncio/pull/662/files Would be nice to have some documentation on to how to migrate to the new way.

We also needed to downgrade to 0.21.1 to make our project work. Waiting for a new update. Thanks!

What does everyone think about introducing a separate loop_scope argument to pytest_asyncio.fixture, in order to control the fixture scope and loop scope separately? For example:

@seifertm 2 questions about this proposal:

1st question: Let’s say that we have a test with 2 async fixtures with different loop_scope. Should I expect pytest-asyncio to raise an error? Let’s say we have the following 2 fixtures:

@pytest_asyncio.fixture(scope="session", loop_scope="session")
async def database_connection():
    ....

@pytest_asyncio.fixture(scope="session")  # << no loop_scope defined, so default=`function` is used
async def random_csv_file():
    ....


async def mytest(random_csv_file, database_connection):
    ...

In this case, the 2 fixtures are using 2 different loop scopes, but it should be possible to run the random_csv_file one with the session scope, right? When thinking of all different scopes (package/module/…) there seems to be some kind of compatibility/loop scope overrides that can be made, but seems like it should be investigated carefully.

2nd question: Can we have some extra configuration for the mode=auto that also allows us to define the default loop_scopes?

Hello. These breaking changes are really annoying.

=============================== warnings summary ===============================
tests/test_orders.py::test_order_tracking_db[29556]
  /home/***/***/***/***/.venv/lib/python3.12/site-packages/pytest_asyncio/plugin.py:647: DeprecationWarning: There is no current event loop
    old_loop = asyncio.get_event_loop()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================= 2 passed, 1 warning in 1.77s =========================
Finished running tests!

Anyway test is not failed and async code was executed successfully. I suggest to include gists for cases of using ORMs in tests, because there are many approaches to achieve tearUp and tearDown, e.g.:

@pytest.fixture(scope="session")
def event_loop_policy():
    return uvloop.EventLoopPolicy()

@pytest.fixture(scope="session")
def db_engine():
    settings = DBSettings()  # type:ignore
    engine = create_async_engine(
        f"{str(settings.dsn)}",
        pool_pre_ping=True,
        pool_size=10,
        max_overflow=50,
        connect_args={
            "server_settings": {"jit": "off"},
        },
        echo=True,
        isolation_level="READ_COMMITTED",
        pool_recycle=600,
    )
    yield engine
    engine.sync_engine.dispose()

@pytest_asyncio.fixture(scope="session")
async def db_session(db_engine):
    async with AsyncSession(bind=db_engine) as session:
        yield session
        await session.rollback()
        await session.close()

@pytest_asyncio.fixture(scope="session")
async def client(db_session):
    def _get_db_override():
        return db_session

    app = app_factory()
    app.dependency_overrides[connection] = _get_db_override
    async with AsyncClient(app=app, base_url="http://test") as client:
        try:
            yield client
        finally:
            await client.aclose()

It works as expected, but with warnings. I have seen approaches with

@pytest_asyncio.fixture(scope="function")
async def db_session(db_engine):

that causes errors if used with parametrize functions. Package is a ‘must have’ if working with asyncio in my opinion. Make it easier to use, please.

It was good enough but a terrible DX honestly. It took me quite a while to figure out the first time I’ve seen this problem (and thought it was a bug or very bad design). If the goal here is that we don’t need these five lines anymore, I’m all in and will just pin my dependency until we update our code. Let’s not revert to something ugly if the new approach is better.

Also, the semantic versioning meaning of 0.* versions is that they can introduce breaking changes anytime. If you don’t want to hit this kind of breaking changes in the first place, just pin your dependencies…

I did not follow exactly the issue, but I wanted to mention that 0.23 broke all our pipelines at work with some strange errors and the same with one of the open source project I maintain: https://github.com/FreeOpcUa/opcua-asyncio . The solution so far has been to revert to 0.21 everywhere

It says “This release is backwards-compatible with v0.21. Changes are non-breaking, unless you upgrade from v0.22.”

That was the original intention, yes. I can reproduce the difference between v0.21.1 and v0.23 and I agree that this is a breaking change (by accident).

With regards to the migration and as a workaround: The fundamental idea of v0.23 is that each pytest scope (session, package, module, class, and function) provides a separate event loop. You can decide for each test in which loop they run via the new scope kwarg to the asyncio mark.

@tkukushkin In your specific example you want to run the test in the same loop as the fixture. The some_async_fixture fixture has session scope. In order to achieve your goal, you should mark test_something accordingly:

@pytest.mark.asyncio(scope="session")
async def test_something(some_async_fixture):
    assert asyncio.get_running_loop() is some_async_fixture

See also the part about asyncio event loops in the Concepts section of the docs.

@redtomato74 I’d like to hear more about your use case for two different event loops. I suggest you open a separate issue for this.