bazel: Custom test rule without `_lcov_merger` attribute always fails under `bazel coverage`
Description of the problem / feature request:
When a custom Starlark test rule without explicit coverage support in the form of the _lcov_merger
attribute is invoked via bazel coverage
with Bazel after 5b216b27435aeb9eb9c3bd3c552d6498e1050cc7, it fails with
FAIL: //:custom_test (see /home/fmeum/.cache/bazel/_bazel_fhenneke/767fd922858d516eb34a4dddb64ca1a0/execroot/__main__/bazel-out/k8-fastbuild/testlogs/custom_test/test.log)
ERROR: /home/fmeum/git/bazel-custom-test-coverage/BUILD:3:12: output 'custom_test/coverage.dat' was not created
ERROR: /home/fmeum/git/bazel-custom-test-coverage/BUILD:3:12: Testing //:custom_test failed: not all outputs were created or valid
Target //:custom_test up-to-date:
bazel-bin/custom_test
INFO: Elapsed time: 5.635s, Critical Path: 0.17s
INFO: 7 processes: 1 disk cache hit, 5 internal, 1 linux-sandbox.
FAILED: Build did NOT complete successfully
//:custom_test FAILED in 0.0s
/home/fmeum/.cache/bazel/_bazel_fhenneke/767fd922858d516eb34a4dddb64ca1a0/execroot/__main__/bazel-out/k8-fastbuild/testlogs/custom_test/test.log
The log says:
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //:custom_test
-----------------------------------------------------------------------------
--
Coverage runner: cannot locate file
With Bazel 4.2.1, the build succeeds:
INFO: Build completed successfully, 5 total actions
//:custom_test PASSED in 0.0s
Executed 1 out of 1 test: 1 test passes.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command
INFO: Build completed successfully, 5 total actions
Bugs: what’s the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
#!/usr/bin/env sh
touch WORKSPACE
cat << EOF > BUILD
load("//:rules.bzl", "custom_test")
custom_test(
name = "custom_test",
)
EOF
cat << EOF > rules.bzl
def _custom_test_impl(ctx):
executable = ctx.actions.declare_file(ctx.attr.name)
ctx.actions.write(executable, "", is_executable = True)
return [
DefaultInfo(
executable = executable,
)
]
custom_test = rule(
implementation = _custom_test_impl,
test = True,
)
EOF
bazel coverage //:custom_test
What operating system are you running Bazel on?
Linux
What’s the output of bazel info release
?
release 5.0.0-pre.20210831.2
I bisected the origin of this issue back to 5b216b27435aeb9eb9c3bd3c552d6498e1050cc7.
Have you found anything relevant by searching the web?
With 5b216b27435aeb9eb9c3bd3c552d6498e1050cc7
, the custom_test
rule in the reproducer now provides an InstrumentedFilesInfo
, which causes TestActionBuilder to collect coverage. However, since the rule does not have the :lcov_merger
attribute set, the LCOV_MERGER
env variable isn’t set when collect_coverage.sh
is invoked, which causes the error message reproduced above.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 19 (14 by maintainers)
Commits related to this issue
- Provide the lcov_merger to all Starlark tests Since 5b216b27, all Starlark-defined test rules either need to set the `_lcov_merger` attribute or fail if run with `bazel coverage`. This is fixed by at... — committed to fmeum/bazel by fmeum 3 years ago
- Provide the lcov_merger to all Starlark tests Since 5b216b27, all Starlark-defined test rules either need to set the `_lcov_merger` attribute or fail if run with `bazel coverage`. This is fixed by at... — committed to fmeum/bazel by fmeum 3 years ago
- Provide the default lcov_merger to all Starlark tests Since 5b216b27, all Starlark-defined test rules either need to set the `_lcov_merger` attribute or fail if run with `bazel coverage`. This is fix... — committed to fmeum/bazel by fmeum 3 years ago
- Provide the default lcov_merger to all Starlark tests Since 5b216b27, all Starlark-defined test rules either need to set the `_lcov_merger` attribute or fail if run with `bazel coverage`. This is fix... — committed to fmeum/bazel by fmeum 3 years ago
- Exit collect_coverage.sh early if LCOV_MERGER is not set. Rules that provide InstrumentedFilesInfo but do not define the _lcov_merger attribute will result in this script being called without the req... — committed to brentleyjones/bazel by c-mita 3 years ago
- Exit collect_coverage.sh early if LCOV_MERGER is not set. (#14359) Rules that provide InstrumentedFilesInfo but do not define the _lcov_merger attribute will result in this script being called witho... — committed to bazelbuild/bazel by brentleyjones 3 years ago
- Exit collect_coverage.sh early if LCOV_MERGER is not set. Rules that provide InstrumentedFilesInfo but do not define the _lcov_merger attribute will result in this script being called without the req... — committed to Bencodes/bazel by c-mita 3 years ago
- Fix bazel coverage false negative Previously this short circuit meant the tests weren't actually run, and they would always exit 0, in the case the test rule didn't set the lcov related attributes. ... — committed to keith/bazel by keith 2 years ago
- Fix bazel coverage false negative Previously this short circuit meant the tests weren't actually run, and they would always exit 0, in the case the test rule didn't set the lcov related attributes. ... — committed to bazelbuild/bazel by keith 2 years ago
- Fix bazel coverage false negative Previously this short circuit meant the tests weren't actually run, and they would always exit 0, in the case the test rule didn't set the lcov related attributes. ... — committed to brentleyjones/bazel by keith 2 years ago
- Fix bazel coverage false negative (#14836) Previously this short circuit meant the tests weren't actually run, and they would always exit 0, in the case the test rule didn't set the lcov related at... — committed to bazelbuild/bazel by brentleyjones 2 years ago
Yeah, the
diff_test
was just an example of a OSS rule that doesn’t have the_lcov_merger
attr set. Adding theexit 1
to the generated test script above will repro the issue and the test will pass incorrectly underbazel coverage
.I can open a new issue if that’s easier too.
With that patch it does remove the need to pass
--test_env=LCOV_MERGER=/usr/bin/true
to coverage invocationsHere’s a fix https://github.com/bazelbuild/bazel/pull/14807