pytest-cov: Regression in 2.6.0: running coverage combine after pytest-cov now errors out
We run coverage in parallel=True mode, because we have some tests that spawn subprocesses, and we want to capture coverage for all of them. To get a complete report for codecov, the CI does:
pytest --cov ...
coverage combine
bash <(curl -s https://codecov.io/bash)
As far as I know, this was mandatory on pytest-cov 2.5.1 and earlier. Since 2.6.0 came out yesterday, our tests have been failing because coverage combine says No data to combine and it errors out, failing the test suite.
I think this may be happening because pytest-cov is now effectively running coverage combine itself, so that instead of being mandatory to run coverage combine, it’s now mandatory to not to run coverage combine? Possibly because of changes in #178? For now that seems to be working for us. But it’d be nice to confirm that our understanding is correct, and I figured you’d rather have a heads-up than not.
About this issue
- Original URL
- State: open
- Created 6 years ago
- Reactions: 1
- Comments: 16 (8 by maintainers)
Commits related to this issue
- temporarily pin pytest-cov to previous version to work around 'No data to combine error' The latest pytest-cov 2.6.0 seems to be running coverage combine itself: https://github.com/pytest-dev/pytest... — committed to anthrotype/afdko by anthrotype 6 years ago
- temporarily pin pytest-cov to previous version to work around 'No data to combine error' The latest pytest-cov 2.6.0 seems to be running coverage combine itself: https://github.com/pytest-dev/pytest... — committed to adobe-type-tools/afdko by anthrotype 6 years ago
- temporarily pin pytest-cov to previous version to work around 'No data to combine error' The latest pytest-cov 2.6.0 seems to be running coverage combine itself: https://github.com/pytest-dev/pytest... — committed to adobe-type-tools/afdko by anthrotype 6 years ago
- temporarily pin pytest-cov to previous version to work around 'No data to combine error' The latest pytest-cov 2.6.0 seems to be running coverage combine itself: https://github.com/pytest-dev/pytest... — committed to adobe-type-tools/afdko by anthrotype 6 years ago
- temporarily pin pytest-cov to previous version to work around 'No data to combine error' The latest pytest-cov 2.6.0 seems to be running coverage combine itself: https://github.com/pytest-dev/pytest... — committed to adobe-type-tools/afdko by anthrotype 6 years ago
- Fix for failing `pytest-cov` upgrade. This patch is a work-around for an issue with `pytest-cov`. For additional information, please see: https://github.com/pytest-dev/pytest-cov/issues/222 — committed to jbenden/deployer by jbenden 6 years ago
- Temporarily restrict pytest-cov version See https://github.com/pytest-dev/pytest-cov/issues/222 — committed to astrojuanlu/poliastro by astrojuanlu 6 years ago
- Temporarily restrict pytest-cov version See https://github.com/pytest-dev/pytest-cov/issues/222 — committed to astrojuanlu/poliastro by astrojuanlu 6 years ago
- New pytest-cov seems to be handling combining? https://github.com/pytest-dev/pytest-cov/issues/222 — committed to wtgee/POCS by wtgee 6 years ago
- downgraded dependency for pytest-cov: https://github.com/pytest-dev/pytest-cov/issues/222 — committed to AtteqCom/zsl by skiwichu 5 years ago
This auto-combine is problematic when you have several runs of pytest (say across different versions of python) and want to combine them. Please at least add an option to prevent pytest-cov doing this combining.
My take on this is that while pytest users tend to naturally gravitate to pytest-cov, it is actually not necessary and running coverage.py directly works just as well, and the whole setup has less moving parts https://coverage.readthedocs.io/en/stable/#quick-start. So, unless you need pytest-cov to achieve something specific, I suggest not to use it “just because we use pytest”.