coveragepy: Coverage in parallel failing
Describe the bug When we run pytest using multiple processes with coverage on our tests, we get an error message that seems to be related to the integration of SQLite. The error we get is available from our integration test suite here: http://apollo.cs.man.ac.uk:8080/blue/organizations/jenkins/sPyNNaker8 Integration Tests/detail/neuron_recorder_agdr/10/pipeline/102
Replicated a sample of the output here in case this disappears!
[gw7] linux -- Python 3.6.7 /usr/bin/python3
self = <SqliteDb @0x7f9efba0c1d0 debug=None filename='/var/lib/jenkins/workspace/ation_Tests_neuron_recorder_agdr/.coverage' nest=0 con=None>
sql = 'insert into meta (key, value) values (?, ?)'
parameters = ('has_arcs', '1')
def execute(self, sql, parameters=()):
"""Same as :meth:`python:sqlite3.Connection.execute`."""
if self.debug:
tail = " with {!r}".format(parameters) if parameters else ""
self.debug.write("Executing {!r}{}".format(sql, tail))
try:
> return self.con.execute(sql, parameters)
E sqlite3.IntegrityError: UNIQUE constraint failed: meta.key
To Reproduce How can we reproduce the problem? Please be specific.
- What version of Python are you using? 3.6
- What version of coverage.py are you using? The output of
coverage debug sys
is helpful. 5.0 - What versions of what packages do you have installed? The output of
pip freeze
is helpful. See above link for details. - What code are you running? Give us a specific commit of a specific repo that we can check out. Integration tests in Jenkins; see above for details.
- What commands did you run?
py.test SpiNNUtils/unittests -rs -n auto --forked --show-progress --cov-branch --cov spynnaker8 --cov spynnaker --cov spinn_front_end_common --cov pacman --cov data_specification --cov spinnman --cov spinn_machine --cov spinn_storage_handlers --cov spalloc --cov spinn_utilities --junitxml junit/SpiNNUtils.xml --cov-report xml:coverage.xml --cov-append --timeout 1200
Expected behavior A clear and concise description of what you expected to happen. No error message; protection against writing from mulitple processes through checking for key in database before attempted insert.
Additional context Add any other context about the problem here.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 13
- Comments: 23 (8 by maintainers)
Commits related to this issue
- coverage version capped due to https://github.com/nedbat/coveragepy/issues/883 — committed to SpiNNakerManchester/sPyNNaker8 by Christian-B 5 years ago
- coverage version capped due to https://github.com/nedbat/coveragepy/issues/883 — committed to SpiNNakerManchester/SpiNNakerGraphFrontEnd by Christian-B 5 years ago
- Demonstration of https://github.com/nedbat/coveragepy/issues/883 — committed to Urth/coveragepy_883 by Urth 5 years ago
- Add .coveragerc to specify parallel runs Per upstream issue, this workaround keeps coverage from failing when trying to append to a nonexistent .coverage file. See: https://github.com/nedbat/coverag... — committed to sco1/flake8-annotations by sco1 4 years ago
- MAINT: https://github.com/nedbat/coveragepy/issues/883 — committed to MartinThoma/edapy by MartinThoma 4 years ago
- MAINT: https://github.com/nedbat/coveragepy/issues/883 — committed to MartinThoma/edapy by MartinThoma 4 years ago
- Use parallel=True to workaround nedbat/coveragepy#883 — committed to skirpichev/diofant by skirpichev 4 years ago
- Fix tests for Fedora 31+ and coverage Since Fedora 31, some macros that we used in RDO packages need to be fixed. Fixing this also exposed an issue when running coverage tests in parallel [1], so we ... — committed to softwarefactory-project/rdopkg by javierpena 4 years ago
- Drop coverage pinning Used parallel=True option to workaround nedbat/coveragepy#883 — committed to skirpichev/diofant by skirpichev 4 years ago
- Try solution in `https://github.com/nedbat/coveragepy/issues/883#issuecomment-650562896` — committed to Carreau/zarr-python by Carreau 4 years ago
- Try solution in `https://github.com/nedbat/coveragepy/issues/883#issuecomment-650562896` — committed to Carreau/zarr-python by Carreau 4 years ago
- py-coverage: update to 6.4.4. Version 6.4.4 — 2022-08-16 -------------------------- - Wheels are now provided for Python 3.11. .. _changes_6-4-3: Version 6.4.3 — 2022-08-06 ----------------------... — committed to NetBSD/pkgsrc by deleted user 2 years ago
I believe this is now fixed in commit b41be3f9
One solution is to add this to a .coveragerc file:
Removing
--cov-append
also prevents the error. I’m not sure which of these gives you the proper measurements, if either. I don’t know if there’s a better solution that can be built in to coverage.py or to pytest-cov.Is there any drawback to using:
?
And if there isn’t, why is it not the default ? (honest question)
The simplest way to avoid this issue for now is to pin coverage.py to 4.5.4. Another thing that would help me find a solution for you is to provide me with specific, complete instructions that would let me reproduce the problem.
Just to highlight: Adding the following section to the
setup.cfg
fixes the issue:edit: Strange … I think this worked a while ago. Now it doesn’t anymore.