pytest: Test fail (instead of skip) on empty parameter set
I need to fail (instead of skip) tests that got empty parameters set. My test looks like this:
def parameters_provider():
# HTTP request to some external provider service.
return service_request().result or []
@pytest.mark.parametrize("parameter", parameters_provider())
def test_something(parameter):
assert parameter is not None
So if provider fails to return results, test gets empty parameters set and pytest skips it, but for me this is actually error - kind of no tests run and nothing actually checked, so test must fail.
After some investigation i’ve found pretty nasty way to achieve my goal, in conftest.py:
def pytest_runtest_teardown(item, nextitem):
skip = item.get_marker("skip")
if skip:
reason = skip.kwargs.get("reason")
if reason.startswith("got empty parameter set"):
raise ValueError(reason)
I don’t like it because it depends on reason text which can change in any next pytest release. Is there any better way to do it?
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 1
- Comments: 18 (11 by maintainers)
Commits related to this issue
- fix #2527 - introduce a option to pic the empty parameterset action — committed to RonnyPfannschmidt/pytest by RonnyPfannschmidt 7 years ago
- fix #2527 - introduce a option to pic the empty parameterset action — committed to RonnyPfannschmidt/pytest by RonnyPfannschmidt 7 years ago
- fix #2527 - introduce a option to pic the empty parameterset action — committed to RonnyPfannschmidt/pytest by RonnyPfannschmidt 7 years ago
I don’t know, empty parameter sets seems such a rare occurrence to me that it should fail by default, as it is most likely an error.
I would propose to change the default to “error”, specially if users that intend for it to skip can do this:
Then at least whoever wants to keep the existing “skip” behavior can have it.
@RonnyPfannschmidt can you please search that discussion about the points in favor of “skip”? It would help this discussion for sure.