pytest: Unexpected order of tests using parameterized fixtures

@kchomski-reef thanks for the great insight. I haven’t checked reorder_items code yet, but your reasoning seems correct. I only think that the assumptions on which this functionality is based are naive - that all paremetrized fixtures have the same weight. In real world f1 fixture from my example can have much more expensive setup/teardown, or there can be more fixtures depending on it. It would be great if I could control the order somehow. I tried indirect parametrization:

import itertools
import pytest

f1_params = (1, 2, 3)
@pytest.fixture(scope="module", params=f1_params)
def f1(request):
    pass
    
f2_params = ('a', 'b', 'c')
@pytest.fixture(scope="module", params=f2_params)
def f2(request):
    pass

@pytest.mark.parametrize('f1,f2', itertools.product(f1_params, f2_params), indirect=True)
def test(f1, f2):
    pass

But it breaks fixtures’ scopes (#570). The only working solutions for me now is moving one parametrized fixture to higher scope:

import pytest

@pytest.fixture(scope="session", params=(1, 2, 3))
def f1_meta(request):
    return request.param

@pytest.fixture(scope="module")
def f1(f1_meta):
    pass
    
@pytest.fixture(scope="module", params=('a', 'b', 'c'))
def f2(request):
    pass

def test(f1, f2):
    pass

But it seems hacky and it’s not scalable. Is there any way of achieving it? Some hook maybe?

Good point to have a new issue rather than stuck in a closed ticket. #3393 created.

@kchomski-reef thanks for the interest.

You are half right: the purpose is to reorder test itemswithin a scope to minimize fixture setup/teardown, when those fixtures are parametrized.

For example, consider this session fixture:

@pytest.fixture(scope='session', params=[1, 2])
def session_fix(request):
    # let's pretend this actually does some heavy weight setup/teardown 
    return request.param

Consider this test file:

def test_normal_1(): pass

def test_uses_fix_1(session_fix): pass
def test_uses_fix_2(session_fix): pass

def test_normal_2(): pass

If we don’t do any reordering, the tests would execute in this order:

test_normal_1
test_uses_fix_1[session_fix(1)]
test_uses_fix_1[session_fix(2)]
test_uses_fix_2[session_fix(1)]
test_uses_fix_2[session_fix(2)]
test_normal_2

IOW, this would cause session_fix to be created with parameter 1 for test_uses_fix_1, then be destroyed so a new session_fix can be created with parameter 2 and execute test_uses_fix_1 again with the new fixture. The same happens for test_uses_fix_2 and for every other test which uses those fixtures.

A key fact to understand all this and that is not clear in the documentation is that at any given time only a single instance of session_fix can exist, that’s why it needs to be destroyed between each test function call which parametrizes it.

The reorder algorithm thus tries to reorder the tests to minimize fixture setup/teardown:

test_normal_1
test_uses_fix_1[session_fix1]
test_uses_fix_2[session_fix1]
test_uses_fix_1[session_fix2]
test_uses_fix_2[session_fix2]
test_normal_2

This order now allow us to create session_fix(1) once, execute test_uses_fix_1 and test_uses_fix_2 with that fixture, then create session_fix(2) and execute the tests, and so on.

The same is valid for the other scopes other than function.

Let me know if something is not clear on my explanation.