coveragepy: `assert row is not None` triggered in `CoverageData._read_db()` when checking schema version

Describe the bug

In some of our jobs with 7.0.2, and not with 7.0.1, we commonly, but not always, get this traceback at the end. I expect it is indicative of another issue related to pytest-cov and pytest-xdist… but, none the less, it is a change in a patch release of coverage that may interest you.

https://github.com/Chia-Network/chia-blockchain/actions/runs/3825220672/jobs/6508036405#step:16:558

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/runner/work/chia-blockchain/chia-blockchain/venv/lib/python3.7/site-packages/_pytest/main.py", line 270, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/runner/work/chia-blockchain/chia-blockchain/venv/lib/python3.7/site-packages/_pytest/main.py", line 324, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/runner/work/chia-blockchain/chia-blockchain/venv/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/home/runner/work/chia-blockchain/chia-blockchain/venv/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/home/runner/work/chia-blockchain/chia-blockchain/venv/lib/python3.7/site-packages/pluggy/_callers.py", line 55, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/home/runner/work/chia-blockchain/chia-blockchain/venv/lib/python3.7/site-packages/pytest_cov/plugin.py", line 297, in pytest_runtestloop
INTERNALERROR>     self.cov_controller.finish()
INTERNALERROR>   File "/home/runner/work/chia-blockchain/chia-blockchain/venv/lib/python3.7/site-packages/pytest_cov/engine.py", line 44, in ensure_topdir_wrapper
INTERNALERROR>     return meth(self, *args, **kwargs)
INTERNALERROR>   File "/home/runner/work/chia-blockchain/chia-blockchain/venv/lib/python3.7/site-packages/pytest_cov/engine.py", line 338, in finish
INTERNALERROR>     self.cov.stop()
INTERNALERROR>   File "/home/runner/work/chia-blockchain/chia-blockchain/venv/lib/python3.7/site-packages/coverage/control.py", line 797, in combine
INTERNALERROR>     message=self._message,
INTERNALERROR>   File "/home/runner/work/chia-blockchain/chia-blockchain/venv/lib/python3.7/site-packages/coverage/data.py", line 158, in combine_parallel_data
INTERNALERROR>     new_data.read()
INTERNALERROR>   File "/home/runner/work/chia-blockchain/chia-blockchain/venv/lib/python3.7/site-packages/coverage/sqldata.py", line 815, in read
INTERNALERROR>     with self._connect():
INTERNALERROR>   File "/home/runner/work/chia-blockchain/chia-blockchain/venv/lib/python3.7/site-packages/coverage/sqldata.py", line 342, in _connect
INTERNALERROR>     self._open_db()
INTERNALERROR>   File "/home/runner/work/chia-blockchain/chia-blockchain/venv/lib/python3.7/site-packages/coverage/sqldata.py", line 285, in _open_db
INTERNALERROR>     self._read_db()
INTERNALERROR>   File "/home/runner/work/chia-blockchain/chia-blockchain/venv/lib/python3.7/site-packages/coverage/sqldata.py", line 302, in _read_db
INTERNALERROR>     assert row is not None
INTERNALERROR> AssertionError

To Reproduce How can we reproduce the problem? Please be specific. Don’t link to a failing CI job. Answer the questions below:

1. What version of Python are you using?

• Python 3.7 through 3.10 on all of Linux, macOS, and Windows

1. What version of coverage.py shows the problem? The output of coverage debug sys is helpful.

• 7.0.2

1. What versions of what packages do you have installed? The output of pip freeze is helpful.

• Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.1 PyYAML-6.0 Pygments-2.14.0 SecretStorage-3.3.3 aiofiles-22.1.0 aiohttp-3.8.3 aiohttp-cors-0.7.0 aiosignal-1.3.1 aiosqlite-0.17.0 altgraph-0.17.3 anyio-3.6.2 argon2-cffi-bindings-21.2.0 argon2_cffi-21.3.0 astroid-2.12.13 asttokens-2.2.1 async-timeout-4.0.2 attrs-22.2.0 backcall-0.2.0 bencode.py-4.0.0 bitstring-3.1.9 black-22.10.0 bleach-5.0.1 blspy-1.0.16 build-0.9.0 certifi-2022.12.7 cffi-1.15.1 cfgv-3.3.1 chardet-5.1.0 charset-normalizer-2.1.1 chia-blockchain-1.7.0b5.dev18 chia-rs-0.1.16 chiabip158-1.1 chiapos-1.0.11 chiavdf-1.0.8 click-8.1.3 clvm-0.9.7 clvm-tools-0.4.6 clvm-tools-rs-0.1.25 colorama-0.4.5 colorlog-6.7.0 commonmark-0.9.1 concurrent-log-handler-0.9.20 coverage-7.0.2 cryptography-38.0.3 decorator-5.1.1 diff-cover-7.3.0 dill-0.3.6 distlib-0.3.6 dnslib-0.9.23 dnspython-2.2.1 docutils-0.19 exceptiongroup-1.1.0 execnet-1.9.0 executing-1.2.0 filelock-3.8.0 flake8-6.0.0 frozenlist-1.3.3 identify-2.5.11 idna-3.4 importlib-metadata-6.0.0 iniconfig-1.1.1 ipython-8.7.0 isort-5.11.4 jaraco.classes-3.2.3 jedi-0.18.2 jeepney-0.8.0 keyring-23.9.3 keyrings.cryptfile-1.3.9 lazy-object-proxy-1.8.0 matplotlib-inline-0.1.6 mccabe-0.7.0 memory-profiler-0.61.0 more-itertools-9.0.0 multidict-6.0.4 mypy-0.991 mypy-extensions-0.4.3 nodeenv-1.7.0 packaging-21.3 parso-0.8.3 pathspec-0.10.3 pep517-0.13.0 pexpect-4.8.0 pickleshare-0.7.5 pkginfo-1.9.2 platformdirs-2.6.2 pluggy-1.0.0 portalocker-2.6.0 pre-commit-2.21.0 prompt-toolkit-3.0.36 psutil-5.9.1 ptyprocess-0.7.0 pure-eval-0.2.2 py3createtorrent-1.1.0 pycodestyle-2.10.0 pycparser-2.21 pycryptodome-3.16.0 pyflakes-3.0.1 pyinstaller-5.6.2 pyinstaller-hooks-contrib-2022.14 pylint-2.15.9 pyparsing-3.0.9 pytest-7.2.0 pytest-asyncio-0.20.3 pytest-cov-4.0.0 pytest-monitor-1.6.5 pytest-xdist-3.1.0 readme-renderer-37.3 requests-2.28.1 requests-toolbelt-0.10.1 rfc3986-2.0.0 rich-13.0.0 setproctitle-1.2.3 six-1.16.0 sniffio-1.3.0 sortedcontainers-2.4.0 stack-data-0.6.2 tomli-2.0.1 tomlkit-0.11.6 traitlets-5.8.0 twine-4.0.2 types-aiofiles-22.1.0.4 types-cryptography-3.3.23.2 types-pkg-resources-0.1.3 types-pyyaml-6.0.12.2 types-setuptools-65.6.0.2 typing-extensions-4.4.0 urllib3-1.26.13 virtualenv-20.17.1 watchdog-2.1.9 wcwidth-0.2.5 webencodings-0.5.1 wrapt-1.14.1 yarl-1.8.2 zipp-3.11.0 zstd-1.5.2.6

1. What code shows the problem? Give us a specific commit of a specific repo that we can check out. If you’ve already worked around the problem, please provide a commit before that fix.

• https://github.com/Chia-Network/chia-blockchain/pull/14257/commits/28b18935199cc30624faaa876fcf99d03855ad67

1. What commands did you run?

• pytest --cov=chia --cov=tests --cov-config=.coveragerc --cov-report= --durations=10 -n 4 -m "not benchmark" tests/core/util/test_cached_bls.py tests/core/util/test_config.py tests/core/util/test_file_keyring_synchronization.py tests/core/util/test_files.py tests/core/util/test_jsonify.py tests/core/util/test_keychain.py tests/core/util/test_keyring_wrapper.py tests/core/util/test_lockfile.py tests/core/util/test_lru_cache.py tests/core/util/test_significant_bits.py tests/core/util/test_streamable.py
• https://github.com/Chia-Network/chia-blockchain/blob/28b18935199cc30624faaa876fcf99d03855ad67/.github/workflows/test-single.yml#L214
• https://github.com/Chia-Network/chia-blockchain/actions/runs/3826406512/jobs/6510216659#step:16:1

Expected behavior

No tracebacks as is the case with 7.0.1. Unless… it is deemed that this is an indication of a “real” problem as I expressed concern about above.

Additional context

https://github.com/Chia-Network/chia-blockchain/pull/14255 is where I am reverting to 7.0.1 for now and includes links to a couple other PRs and CI runs where I setup our CI to run the problematic jobs 60 times over to provide a clear “statistical” comparison of the zero failures with 7.0.1 vs., well, many with 7.0.2.

I looked at the refactor of CoverageData._read_db() at https://github.com/nedbat/coveragepy/compare/7.0.1..7.0.2#diff-e5185ca855df54af4f94c983f99b52cfad9ddc17386353d09acdfa994dd72a33R287 (you have to expand the Load diff message for coverage/sqldata.py since GitHub doesn’t do it for you) and noticed that while this exception would have previously been caught by the except Exception as exc: clause the unpacking has been moved such that it is no longer mis-detected as a missing table.

Side note, this gave me a good chuckle.

https://github.com/nedbat/coveragepy/blob/7.0.2/CHANGES.rst#version-702--2023-01-02

Refactor: added many type annotations, including a number of refactorings. This should not affect outward behavior, but they were a bit invasive in some places, so keep your eyes peeled for oddities.