spack: Coverage on unit tests fails when tracking child processes

As of 596d463714d46929f81db62b171c5a772679bb12 using concurrency=multiprocess to run:

$ coverage run bin/spack test

results in a few .coverage* files being corrupted. This may compromise the execution of:

$ coverage combine
$ codecov --required

and in the end the coverage report from codecov (see data from January 2018, where it dropped from ~75% to ~50%).

#6872 has been merged as a work-around to get more stable measurements, but the underlying issue with pytest + multiprocessing + coverage needs to be investigated further.

@adamjstewart @tgamblin @scheibelp @becker33

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Comments: 15 (10 by maintainers)

Commits related to this issue

Most upvoted comments

I just fixed this by raising an error if combine can’t read any of the files it found.

Bottomline: yes, we have a buglet in our tests somewhere related to multiprocessing, coverage has a bug (because it doesn’t realize that a .coverage file is already there before deleting it) , Travis has numerous unrelated bugs that cause random failures of jobs and codecov had a bug that once fixed broke our CI 😆

I think I finally got the the bottom of this twisted issue. Between 2.0.10 and 2.0.11 https://github.com/codecov/codecov-python/pull/112 was merged. This means that we started calling:

$ coverage combine

twice.

Now, doing this:

$ coverage run --concurrency=multiprocessing bin/spack test
$ coverage combine
Coverage.py warning: Couldn't read data from '/home/mculpo/PycharmProjects/spack/.coverage.nuvolari.1043.077127': CoverageException: Doesn't seem to be a coverage.py data file

will result very likely in some error. If you check your .coverage now you’ll find it has data in it. The corrupted files will instead be empty and are not deleted like the good ones by the command.

Running again:

$ coverage combine

(which is what effectively codecov 2.0.11 does for us) will account only for the corrupted files and leave you with an empty report! The content is:

!coverage.py: This is a private format, don't read it directly!{}

Commit 088c193 scores 74.45% on codecov (last stable number). It has all 10 reports on codecov, but still the log of Travis shows:

Coverage.py warning: Couldn't read data from '/home/travis/build/spack/spack/.coverage.travis-job-spack-spack-323456025.travisci.net.9618.332776': CoverageException: Doesn't seem to be a coverage.py data file
Coverage.py warning: Couldn't read data from '/home/travis/build/spack/spack/.coverage.travis-job-spack-spack-323456025.travisci.net.9686.644486': CoverageException: Doesn't seem to be a coverage.py data file
Coverage.py warning: Couldn't read data from '/home/travis/build/spack/spack/.coverage.travis-job-spack-spack-323456025.travisci.net.9719.277474': CoverageException: Doesn't seem to be a coverage.py data file

codecov version is 2.0.10. On January 2 version 2.0.11 went out.

At this point I start to think that we had this bug with pytest + coverage since quite some time, but never noticed it because codecov was able to deal nicely with the merged coverage file (despite the failures above) until version 2.0.11.