pytest-cov: Using pytest-cov for coverage results in top-level package name of "." when using coverage.py doesn't

First, versions:

$ python --version
Python 2.7.11
$ pip freeze|grep coverage
coverage==4.4.1
$ pip freeze|grep pytest-cov
pytest-cov==2.5.1

Happens on Ubuntu 12.04, Ubuntu 16.04, and Mac OS X El Capitan 10.11.6. Can’t test anything else, but seems safe to say it would be the same.

I have a project laid out like this:

proj/ex_secure/__init__.py
proj/ex_secure/base.py
proj/ex_secure/metrics.py
proj/ex_secure/keys.py
proj/ex_secure/utils.py
proj/tests/test_base.py
proj/tests/test_metrics.py
proj/tests/test_keys.py
proj/.gitignore
proj/.pep8
proj/README.rst
proj/setup.cfg
proj/setup.py

All the example commands below I run from within the proj directory.

First, I run coverage directly:

$ coverage run --branch --source=ex_secure -m pytest -s --junitxml=pytests.xml
... output that doesn't matter ...
$ coverage xml
$ coverage report
Name                              Stmts   Miss Branch BrPart  Cover
-------------------------------------------------------------------
ex_secure/__init__.py                 1      0      0      0   100%
ex_secure/base.py                   274     55     74     12    74%
ex_secure/keys.py                    31      6      6      0    78%
ex_secure/metrics.py                 10      2      4      0    57%
ex_secure/util.py                   238    238     98      0     0%

And the top package of the XML file looks like so:

<package branch-rate="0.3058" complexity="0" line-rate="0.4769" name="ex_secure">

Next, I run pytest with the pytest-cov plugin:

$ pytest -s --junitxml=pytests.xml --cov-report xml --cov-report term --cov-branch --cov=ex_secure
... output that doesn't matter ...
---------- coverage: platform darwin, python 2.7.11-final-0 ----------
Name                              Stmts   Miss Branch BrPart  Cover
-------------------------------------------------------------------
ex_secure/__init__.py                 1      0      0      0   100%
ex_secure/base.py                   274     55     74     12    74%
ex_secure/keys.py                    31      6      6      0    78%
ex_secure/metrics.py                 10      2      4      0    57%
ex_secure/util.py                   238    238     98      0     0%

And the top package of the XML file looks like so:

<package branch-rate="0.3058" complexity="0" line-rate="0.4769" name=".">

Notice the name of the package is now . instead of ex_secure. However, if I then run coverage xml to re-generate the report:

$ coverage xml

The XML file looks right again:

<package branch-rate="0.3058" complexity="0" line-rate="0.4769" name="ex_secure">

So I tried running pytest a different way:

$ pytest -s --junitxml=pytests.xml --cov-report xml --cov-report term --cov-branch --cov=ex_secure.base --cov=ex_secure.keys --cov=ex_secure.metrics --cov=ex_secure.util
... output that doesn't matter ...
---------- coverage: platform darwin, python 2.7.11-final-0 ----------
Name                              Stmts   Miss Branch BrPart  Cover
-------------------------------------------------------------------
ex_secure/base.py                   274     55     74     12    74%
ex_secure/keys.py                    31      6      6      0    78%
ex_secure/metrics.py                 10      2      4      0    57%

Now the XML also looks right:

<package branch-rate="0.4722" complexity="0" line-rate="0.6766" name="ex_secure">

But this has several downsides:

I have to enumerate everything, and I miss coverage info for new files added if they are not enumerated.
Can’t cover __init__.py
utils.py is not in the report anymore because none of its lines are covered, resulting in incorrectly high coverage results
Branch rate and line rate are now different (because of missing __init__.py and utils.py

About this issue

Original URL
State: open
Created 7 years ago
Reactions: 11
Comments: 21 (4 by maintainers)

Most upvoted comments

Just to add, I’m having the same problem. We have a slightly weird package layout, with a main package and an extra module in the same level, and pytest-cov does the same name='.' thing, which breaks our Sonar integration.

tonnydourado on Dec 29, 2017

I did some debugging and I might have an answer:

This is the XML report generation using pytest-cov:

# In coverage/control.py xml_report()
ipdb> self.source_match
<TreeMatcher [u'/Users/nwilliams/proj/ex_secure']>
# In coverage/xmlreport.py report()
ipdb> morfs, outfile
(None, <open file 'coverage.xml', mode 'w' at 0x10cdd3e40>)
ipdb> self.packages
{}
# In coverage/xmlreport.py xml_file()
ipdb> filename
u'/Users/nwilliams/proj/ex_secure/__init__.py'
ipdb> self.source_paths
set([u'/Users/nwilliams/proj/ex_secure'])
ipdb> dirname
u'.'
ipdb> package_name
u'.'
# Back in coverage/xmlreport.py report()
ipdb> self.packages
{'.': [{}, 0, 0, 0, 0]}

This is the XML report generation using coverage xml immediately after running pytest with pytest-cov:

# In coverage/control.py xml_report()
ipdb> self.source_match
None
# In coverage/xmlreport.py report()
ipdb> morfs, outfile
([], <open file 'coverage.xml', mode 'w' at 0x10537e5d0>)
ipdb> self.packages
{}
# In coverage/xmlreport.py xml_file()
ipdb> filename
u'/Users/nwilliams/proj/ex_secure/__init__.py'
ipdb> self.source_paths
set([])
ipdb> dirname
u'ex_secure'
ipdb> package_name
u'ex_secure'
# Back in coverage/xmlreport.py report()
ipdb> self.packages
{u'ex_secure': [{}, 0, 0, 0, 0]}

Upon further examination of the code, coverage xml does not allow the --sources argument and has no way to pass sources into xml_report. That’s why it works. On the other hand, pytest-cov is creating a single Coverage object and setting its sources, which get passed on to the xml_report, and so it strips that from all the file names before figuring out their package names.

So one could argue Coverage should be a little smarter here, but also pytest-cov is not calling xml_report the same way that coverage xml would, and that seems like a legit bug in pytest-cov to me.

beamerblvd on Oct 27, 2017

Can confirm I encounter exactly the problem described by https://github.com/pytest-dev/pytest-cov/issues/175#issue-269116254 and https://github.com/pytest-dev/pytest-cov/issues/175#issuecomment-354476616

the reasong https://github.com/pytest-dev/pytest-cov/issues/175#issuecomment-358335476 doesn’t work for me is that I have two sub-packages in one repository, so I need pytest-cov to actually generate the coverage report correctly

nzjrs on Mar 13, 2018

Yep, that’s correct. You would include /foo/ and /bob/ under the include section of [report] in your coveragerc. When you tell pytest-cov to cover “.”, coverage will check that file and only report on foo and bob.

Jnesselr on Mar 13, 2018