bazel: Python Coverage does not work

@chickenandpork @ulfjack If you use a pytest_runner to run your pytest tests, it’s a convenient place to put the lconv conversion code 😉 That way you can still use standard tools (and not need to use the fork or special shell scripts)

The following requires pytest-cov and coverage-lconv

import contextlib
import os
import pathlib
import sys

import coverage
import coverage_lcov.converter
import pytest

@contextlib.contextmanager
def coverage_decorator():
    if os.getenv("COVERAGE", None) == "1":
        coverage_dir = pathlib.Path(os.getenv("COVERAGE_DIR"))
        coverage_file = coverage_dir / ".coverage"
        coverage_manifest = pathlib.Path(os.getenv("COVERAGE_MANIFEST"))
        coverage_sources = coverage_manifest.read_text().splitlines()
        # @TODO: Handle config stuff
        cov = coverage.Coverage(data_file=str(coverage_file), include=coverage_sources)
        cov.start()

    try:
        yield
    finally:
        if os.getenv("COVERAGE", None) == "1":
            cov.stop()
            cov.save()

            # @TODO: Handle config stuff
            coverage_lcov.converter.Converter(
                relative_path=True,
                config_file=False,
                data_file_path=str(coverage_file),
            ).create_lcov(os.getenv("COVERAGE_OUTPUT_FILE"))


if __name__ == "__main__":
    with coverage_decorator():
        sys.exit(pytest.main(sys.argv[1:]))

FYI, I created https://github.com/bazelbuild/rules_python/pull/977 to make bazel coverage work with the hermetic Python toolchain from @rules_python. I found some issues whilst doing the work and I’ll copy them here for visibility. The issues were:

• I had to use coverage.py v6.5.0 because the latest version (7.0.4 has a types.py file in the package directory, which imports from Python’s stdlib types [1]. Somehow the Python interpreter is thinking that the from types import FrameType is referring to the currently interpreted file and everything breaks. I would have expected the package to use absolute imports and only attempt to import from coverage.types if we use coverage.types and not just a plain types import.
• The multi_python_versions example cannot show coverage for the more complex tests that are using subprocess to spawn a different Python interpreter. I am wondering if this is related to the fact that we are including coverage.py via the toolchain and not through other mechanisms [2].
• The __init__.py files in the root of the bzlmod example was breaking, when running under bazel coverage //:test. However, it started working when I renamed __init__.py to lib.py. I am suspecting that this has to do with the fact that the layer of indirection that coverage introduces could be something to do with that. Note that bazel test ... works regardless of file naming.

I think that all of these issues may be related to Python entrypoint template that is stored in this repository and not an issue with rules_python itself.