bazel: Custom test rule without `_lcov_merger` attribute always fails under `bazel coverage`

Description of the problem / feature request:

When a custom Starlark test rule without explicit coverage support in the form of the _lcov_merger attribute is invoked via bazel coverage with Bazel after 5b216b27435aeb9eb9c3bd3c552d6498e1050cc7, it fails with

FAIL: //:custom_test (see /home/fmeum/.cache/bazel/_bazel_fhenneke/767fd922858d516eb34a4dddb64ca1a0/execroot/__main__/bazel-out/k8-fastbuild/testlogs/custom_test/test.log)
ERROR: /home/fmeum/git/bazel-custom-test-coverage/BUILD:3:12: output 'custom_test/coverage.dat' was not created
ERROR: /home/fmeum/git/bazel-custom-test-coverage/BUILD:3:12: Testing //:custom_test failed: not all outputs were created or valid
Target //:custom_test up-to-date:
  bazel-bin/custom_test
INFO: Elapsed time: 5.635s, Critical Path: 0.17s
INFO: 7 processes: 1 disk cache hit, 5 internal, 1 linux-sandbox.
FAILED: Build did NOT complete successfully
//:custom_test                                                           FAILED in 0.0s
  /home/fmeum/.cache/bazel/_bazel_fhenneke/767fd922858d516eb34a4dddb64ca1a0/execroot/__main__/bazel-out/k8-fastbuild/testlogs/custom_test/test.log

The log says:

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //:custom_test
-----------------------------------------------------------------------------
--
Coverage runner: cannot locate file

With Bazel 4.2.1, the build succeeds:

INFO: Build completed successfully, 5 total actions
//:custom_test                                                           PASSED in 0.0s

Executed 1 out of 1 test: 1 test passes.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command
INFO: Build completed successfully, 5 total actions

Bugs: what’s the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

#!/usr/bin/env sh
touch WORKSPACE

cat << EOF > BUILD
load("//:rules.bzl", "custom_test")

custom_test(
    name = "custom_test",
)
EOF

cat << EOF > rules.bzl
def _custom_test_impl(ctx):
    executable = ctx.actions.declare_file(ctx.attr.name)
    ctx.actions.write(executable, "", is_executable = True)
    return [
        DefaultInfo(
            executable = executable,
        )
    ]

custom_test = rule(
    implementation = _custom_test_impl,
    test = True,
)
EOF

bazel coverage //:custom_test

What operating system are you running Bazel on?

Linux

What’s the output of bazel info release?

release 5.0.0-pre.20210831.2

I bisected the origin of this issue back to 5b216b27435aeb9eb9c3bd3c552d6498e1050cc7.

Have you found anything relevant by searching the web?

With 5b216b27435aeb9eb9c3bd3c552d6498e1050cc7, the custom_test rule in the reproducer now provides an InstrumentedFilesInfo, which causes TestActionBuilder to collect coverage. However, since the rule does not have the :lcov_merger attribute set, the LCOV_MERGER env variable isn’t set when collect_coverage.sh is invoked, which causes the error message reproduced above.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 19 (14 by maintainers)

Commits related to this issue

Most upvoted comments

Yeah, the diff_test was just an example of a OSS rule that doesn’t have the _lcov_merger attr set. Adding the exit 1 to the generated test script above will repro the issue and the test will pass incorrectly under bazel coverage.

#!/usr/bin/env sh
touch WORKSPACE

cat << EOF > BUILD
load("//:rules.bzl", "custom_test")

custom_test(
    name = "custom_test",
)
EOF

cat << EOF > rules.bzl
def _custom_test_impl(ctx):
    executable = ctx.actions.declare_file(ctx.attr.name)
    ctx.actions.write(executable, "exit 1", is_executable = True)
    return [
        DefaultInfo(
            executable = executable,
        )
    ]

custom_test = rule(
    implementation = _custom_test_impl,
    test = True,
)
EOF

bazel coverage //:custom_test

I can open a new issue if that’s easier too.

With that patch it does remove the need to pass --test_env=LCOV_MERGER=/usr/bin/true to coverage invocations