coveragepy: sqlite3.OperationalError: disk I/O error - coverage/sqldata.py, line 1024
Describe the bug
Since coverage v5.0.2, I throw an exception during Python 3.6 testing (Python 3.4, 3.5, 3.7, and 3.8 work fine - source). Below is a screenshot of the exception thrown (source)
This problem does not occur with v5.0.1 (source).
Originally I thought this was a Travis CI issue (source), but it appears specific to the last release of coverage instead.
To Reproduce How can we reproduce the problem? Please be specific.
- Python v3.6.7
- Ubuntu 16.04.6 LTS
- Kernel: 4.15.0-1028-gcp (xenial container here)
- v5.0.2 of
coverage(pulled from PyPi/pip) - The following packages are installed:
apprise==0.8.2 asgiref==3.2.3 attrs==19.3.0 certifi==2019.11.28 chardet==3.0.4 Click==7.0 coverage==5.0.2 Django==3.0.2 entrypoints==0.3 filelock==3.0.12 flake8==3.7.9 idna==2.8 importlib-metadata==1.3.0 Markdown==3.1.1 mccabe==0.6.1 mock==3.0.5 more-itertools==8.0.2 oauthlib==3.1.0 packaging==20.0 pluggy==0.13.1 py==1.8.1 pycodestyle==2.5.0 pyflakes==2.1.1 pyparsing==2.4.6 pytest==5.3.2 pytest-cov==2.8.1 pytest-django==3.7.0 pytz==2019.3 PyYAML==5.3 requests==2.22.0 requests-oauthlib==1.3.0 six==1.13.0 sqlparse==0.3.0 toml==0.10.0 tox==3.14.3 urllib3==1.25.7 virtualenv==16.7.9 wcwidth==0.1.8 zipp==0.6.0 - I’m trying to build my package Apprise API which I hadn’t needed to build until of late. Here is the latest build status.
- The command ran was:
coverage run --parallel -m pytest apprise_api
Expected behavior
Not to get an exception (or maybe it just needs to be gracefully caught since it’s on an __exit__ call anyway)?
Additional context If i re-run the Travis-CI runner on a previous Python 3.6 test runner that has passed (in the past), it fails on Python 3.6 (more details and screenshots explained here).
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 4
- Comments: 63 (25 by maintainers)
Commits related to this issue
- Pin coverage to 5.0.1 workaround for https://github.com/nedbat/coveragepy/issues/915 — committed to r9y9/nnmnkwii by r9y9 4 years ago
- Improvements to debugging while chasing #915 — committed to nedbat/coveragepy by nedbat 4 years ago
- Debugging specific to #915 — committed to nedbat/coveragepy by nedbat 4 years ago
Wow, thanks everyone, here and on Hacker News. I’ve summed up the answer in Bug #915: solved!
@caronc Thanks for persisting! I’ll close this as not a coverage issue. The mock should be on NamedTemporaryFile, not on _TemporaryFileWrapper, since the first is the public interface, and is what you actually call in your product code. But I’m curious: the mock is called “mock_ntf” as if it had once been for NamedTemporaryFile. Why is that?
Happens in Ubuntu docker image with circleci too but slight different error message is thrown…
Couldn’t use data file ‘/home/circleci/project/test_results/.coverage’: database disk image is malformed
The bug is this code (also present in some of the other tests), which patches tempfile._TemporaryFileWrapper to inject OSErrors:
https://github.com/nedbat/apprise-api/blob/master/apprise_api/api/tests/test_add.py#L166
This causes this line to raise an OSError:
https://github.com/python/cpython/blob/3.6/Lib/tempfile.py#L556
triggering the _os.close in the exception handler.
However, the FileIO object created by _io.open continues to reference the file descriptor.
The os reuses the closed fd for opening the sqlite db file. When the FileIO is finalized by the CPython GC some time later, it closes that fd from under sqlite. That leads to the EBADF errors described above.
I’ve simplified the reproduction a bit, and added some questions in this plea for help: https://nedbatchelder.com/blog/202001/bug_915_please_help.html
If you run into this or a similar SQLite / coverage issue, make sure that you aren’t running
python 3.6.0. This problem disappeared for me when changing from3.6.0to3.6.9.@trinitronx The underlying error is different in your case, with a different cause. I don’t know if there is something coverage.py can do to make that situation better, but we should discuss it in a new issue.
The code was written with the assumption that any failure would happen in _io.open instead of _TemporaryFileWrapper. The _TemporaryFileWrapper constructor doesn’t actually do anything other than construct two objects, so it’s very rare for that to fail.
As for why this worked on other versions of Python and other environments, that’s because the problem only shows up when the GC happens to run after the same FD has been reused. If the GC runs at even a very slightly different time (such as between database opens instead of while the database is open) it won’t fail in the same way.
@caronc I do believe this is a genuine bug in Python, and I hope the core devs take bpo 39318 seriously and fix it, ideally with backports.
I understand about the difficulty of getting mocks to work. This article might help: Why your mock doesn’t work.
@caronc Thanks, this reproduces the error. The bad news is I still don’t have an idea why it’s happening, but this gives me something to work with.
@feu-de-bois Your error is described in #916, and I’m working on a fix for it.