pytest: Test fail (instead of skip) on empty parameter set

I need to fail (instead of skip) tests that got empty parameters set. My test looks like this:

def parameters_provider():
    # HTTP request to some external provider service.
    return service_request().result or []

@pytest.mark.parametrize("parameter", parameters_provider())
def test_something(parameter):
    assert parameter is not None

So if provider fails to return results, test gets empty parameters set and pytest skips it, but for me this is actually error - kind of no tests run and nothing actually checked, so test must fail.

After some investigation i’ve found pretty nasty way to achieve my goal, in conftest.py:

def pytest_runtest_teardown(item, nextitem):
    skip = item.get_marker("skip")
    if skip:
        reason = skip.kwargs.get("reason")
        if reason.startswith("got empty parameter set"):
            raise ValueError(reason)

I don’t like it because it depends on reason text which can change in any next pytest release. Is there any better way to do it?

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 1
  • Comments: 18 (11 by maintainers)

Commits related to this issue

Most upvoted comments

I don’t know, empty parameter sets seems such a rare occurrence to me that it should fail by default, as it is most likely an error.

I would propose to change the default to “error”, specially if users that intend for it to skip can do this:

@pytest.mark.skipif(not get_parameter_values(), reason='empty parameters')
@pytest.mark.parametrize('a', get_parameter_values())
def test_foo(a):
   pass

Then at least whoever wants to keep the existing “skip” behavior can have it.

@RonnyPfannschmidt can you please search that discussion about the points in favor of “skip”? It would help this discussion for sure.