spack: Coverage on unit tests fails when tracking child processes
As of 596d463714d46929f81db62b171c5a772679bb12 using concurrency=multiprocess to run:
$ coverage run bin/spack test
results in a few .coverage* files being corrupted. This may compromise the execution of:
$ coverage combine
$ codecov --required
and in the end the coverage report from codecov (see data from January 2018, where it dropped from ~75% to ~50%).
#6872 has been merged as a work-around to get more stable measurements, but the underlying issue with pytest + multiprocessing + coverage needs to be investigated further.
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Comments: 15 (10 by maintainers)
Commits related to this issue
- Removing 'coverage combine' in test script According to what was discovered in #6887, one of the problems is calling 'coverage combine' twice without the '-a' flag. This removes the first call within... — committed to epfl-scitas/spack by deleted user 9 years ago
- Removing 'coverage combine' in test script According to what was discovered in #6887, one of the problems is calling 'coverage combine' twice without the '-a' flag. This removes the first call within... — committed to epfl-scitas/spack by alalazo 9 years ago
- Removing 'coverage combine' in test script According to what was discovered in #6887, one of the problems is calling 'coverage combine' twice without the '-a' flag. This removes the first call within... — committed to epfl-scitas/spack by alalazo 9 years ago
- Removing 'coverage combine' in test script According to what was discovered in #6887, one of the problems is calling 'coverage combine' twice without the '-a' flag. This removes the first call within... — committed to epfl-scitas/spack by alalazo 9 years ago
- Restore multiprocessing in unit tests (#6949) * Revert "Travis: use --concurrency=multiprocessing only on build tests (#6872)" This reverts commit 596d463714d46929f81db62b171c5a772679bb12. * Re... — committed to spack/spack by alalazo 9 years ago
- Restore multiprocessing in unit tests (#6949) * Revert "Travis: use --concurrency=multiprocessing only on build tests (#6872)" This reverts commit 596d463714d46929f81db62b171c5a772679bb12. * Re... — committed to ch741/spack by alalazo 9 years ago
I just fixed this by raising an error if combine can’t read any of the files it found.
Bottomline: yes, we have a buglet in our tests somewhere related to multiprocessing,
coveragehas a bug (because it doesn’t realize that a.coveragefile is already there before deleting it) , Travis has numerous unrelated bugs that cause random failures of jobs andcodecovhad a bug that once fixed broke our CI 😆I think I finally got the the bottom of this twisted issue. Between 2.0.10 and 2.0.11 https://github.com/codecov/codecov-python/pull/112 was merged. This means that we started calling:
twice.
Now, doing this:
will result very likely in some error. If you check your
.coveragenow you’ll find it has data in it. The corrupted files will instead be empty and are not deleted like the good ones by the command.Running again:
(which is what effectively
codecov2.0.11 does for us) will account only for the corrupted files and leave you with an empty report! The content is:Commit 088c193 scores 74.45% on codecov (last stable number). It has all 10 reports on
codecov, but still the log of Travis shows:codecovversion is 2.0.10. On January 2 version 2.0.11 went out.At this point I start to think that we had this bug with
pytest+coveragesince quite some time, but never noticed it becausecodecovwas able to deal nicely with the merged coverage file (despite the failures above) until version 2.0.11.