bazel: Bazel coverage with `--combined_report=lcov` fails

Description of the problem / feature request:

Running bazel coverage --combined_report=lcov //... results in a NullPointerException

> bazel coverage --combined_report=lcov //...
INFO: Using default value for --instrumentation_filter: "//src/main/java/com/chris[/:]".
INFO: Override the above default with --instrumentation_filter
Internal error thrown during build. Printing stack trace: java.lang.NullPointerException
	at com.google.devtools.build.lib.bazel.coverage.BazelCoverageReportModule$1.getArgs(BazelCoverageReportModule.java:108)
	at com.google.devtools.build.lib.bazel.coverage.CoverageReportActionBuilder.generateCoverageReportAction(CoverageReportActionBuilder.java:244)
	at com.google.devtools.build.lib.bazel.coverage.CoverageReportActionBuilder.createCoverageActionsWrapper(CoverageReportActionBuilder.java:192)
	at com.google.devtools.build.lib.bazel.coverage.BazelCoverageReportModule$1.createCoverageReportActionsWrapper(BazelCoverageReportModule.java:93)
	at com.google.devtools.build.lib.analysis.BuildView.createResult(BuildView.java:436)

It appears that the default value of --coverage_report_generator causes this. Running it with --coverage_report_generator="@bazel_tools//tools/test/LcovMerger/java/com/google/devtools/lcovmerger:Main" works; though it should be pointed out that there are some SEVERE logs.

> bazel coverage --combined_report=lcov --coverage_report_generator="@bazel_tools//tools/test/LcovMerger/java/com/google/devtools/lcovmerger:Main" //...
INFO: Using default value for --instrumentation_filter: "//src/main/java/com/chris[/:]".
INFO: Override the above default with --instrumentation_filter
INFO: Analysed 5 targets (23 packages loaded).
INFO: Found 3 targets and 2 test targets...
INFO: LCOV coverage report is located at /home/chris.rice/.cache/bazel/_bazel_chris.rice/1baebf64cb080dc7a8e39a79a89b8741/execroot/__main__/bazel-out/_coverage/_coverage_report.dat
 and execpath is bazel-out/_coverage/_coverage_report.dat
INFO: From CoverageReport _coverage/_coverage_report.dat:
Oct 19, 2018 8:57:48 AM com.google.devtools.lcovmerger.Main getTracefiles
INFO: Found 2 tracefiles.
Oct 19, 2018 8:57:48 AM com.google.devtools.lcovmerger.Main parseFiles
SEVERE: Parsing file bazel-out/k8-fastbuild/testlogs/src/test/java/com/chris/foo/FooTest/coverage.dat
Oct 19, 2018 8:57:48 AM com.google.devtools.lcovmerger.Main parseFiles
SEVERE: Parsing file bazel-out/k8-fastbuild/testlogs/src/test/java/com/chris/hello/HelloWorldTest/coverage.dat
Oct 19, 2018 8:57:48 AM com.google.devtools.lcovmerger.Main getGcovInfoFiles
SEVERE: No gcov info file found.
INFO: Elapsed time: 6.037s, Critical Path: 5.47s
INFO: 15 processes: 10 linux-sandbox, 5 worker.
INFO: Build completed successfully, 35 total actions
//src/test/java/com/chris/foo:FooTest                                    PASSED in 0.7s
  /home/chris.rice/.cache/bazel/_bazel_chris.rice/1baebf64cb080dc7a8e39a79a89b8741/execroot/__main__/bazel-out/k8-fastbuild/testlogs/src/test/java/com/chris/foo/FooTest/coverage.dat
//src/test/java/com/chris/hello:HelloWorldTest                           PASSED in 0.6s
  /home/chris.rice/.cache/bazel/_bazel_chris.rice/1baebf64cb080dc7a8e39a79a89b8741/execroot/__main__/bazel-out/k8-fastbuild/testlogs/src/test/java/com/chris/hello/HelloWorldTest/coverage.dat

Bugs: what’s the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Clone this repo: https://github.com/chris-rice-lr/bazel_combined_report_repro and run

bazel coverage --combined_report=lcov //...

Running

bazel coverage --combined_report=lcov --coverage_report_generator="@bazel_tools//tools/test/LcovMerger/java/com/google/devtools/lcovmerger:Main" //...

will result in a single .dat file as expected

What operating system are you running Bazel on?

Fedora 28

What’s the output of `bazel info release`?

> bazel info release
Starting local Bazel server and connecting to it...
release 0.18.0- (@non-git)

If `bazel info release` returns “development version” or “(@non-git)”, tell us how you built Bazel.

I used the COPR repo: https://copr.fedorainfracloud.org/coprs/vbatts/bazel/

What’s the output of `git remote get-url origin ; git rev-parse master ; git rev-parse HEAD` ?

> git remote get-url origin ; git rev-parse master ; git rev-parse HEAD
git@github.com:chris-rice-lr/bazel_combined_report_repro.git
3b2b00c761c46fe8ea6a86deed47c8f10607ded3
3b2b00c761c46fe8ea6a86deed47c8f10607ded3

Have you found anything relevant by searching the web?

This thread helped. It showed that running with --coverage_report_generator set correctly made it work. https://groups.google.com/forum/#!searchin/bazel-discuss/combined_report|sort:date/bazel-discuss/itJkJ2T4mWw/qNKGEk_uFAAJ

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 9
Comments: 27 (13 by maintainers)

Commits related to this issue

Workadound bazel bug when using combined_report https://github.com/bazelbuild/bazel/issues/6450 — committed to regisd/jflex by regisd 5 years ago
Use a different disk cache for the coverage. Workaround suggested by codesuki https://github.com/bazelbuild/bazel/issues/6450#issuecomment-519751201 Against Internal error thrown during build. Print... — committed to jflex-de/jflex by regisd 5 years ago
Send code coverage reports (#608) * Build coverage reports with Bazel * Gave up with aggregated report (combined_report) https://github.com/bazelbuild/bazel/issues/6450 https://cirrus-... — committed to jflex-de/jflex by regisd 5 years ago
Pseudo-Merge commit 582504fabd9a1756cd869607b6a4b504e8ca7390 Author: Régis Décamps <regisd@google.com> Date: Tue Nov 26 00:18:30 2019 +0100 Send code coverage reports (#608) * Build covera... — committed to jflex-de/jflex by traviscibot 5 years ago
Repoint --coverage_report_generator default to CoverageOutputGenerator From https://github.com/bazelbuild/bazel/issues/6450#issuecomment-514933269: > The default value of the --coverage_report_gener... — committed to beasleyr-vmw/bazel by beasleyr-vmw 5 years ago

Most upvoted comments

So what is correct command to generate the coverage output?

georgewu2007 on Sep 11, 2019

@iirina LcovMerger seems to be missing from https://github.com/bazelbuild/bazel/tree/master/tools/test. Am I looking in the wrong place? If so, where should I find tools/test/LcovMerger/java/com/google/devtools/lcovmerger? Thanks.

EDIT: OK, I found it at the 0.18.1 tag, but I see that in newer versions of Bazel this has been renamed to CoverageOutputGenerator (https://github.com/bazelbuild/bazel/commit/8f4786b744e9f5820906a29674f668171813dba2). I am using 0.28.1. I will leave this here if other people run into this.

bogatuadrian on Jul 31, 2019

It should work at HEAD, and I expect it to be in the next release, which is probably 2.1.0.

ulfjack on Jan 7, 2020

The reason was using --disk_cache. This doesn’t seem to work well with CoverageOutputGenerator

codesuki on Aug 9, 2019

There are a lot of affected users who are running with various lower versions. And there is significant amount of work to upgrade to 2.0.0 (or 2.1.0). For bug like this, can we also cherrypick the fix to some of release versions, such as 1.0.0?

Also, code coverage is import functionality to guarantee code quality. Is it possible to add some tests to make sure it won’t break easily again?

Thanks!

speakmore on Feb 3, 2020

I get this traceback after setting the right flag:

--coverage_report_generator="@bazel_tools//tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator:Main"

Internal error thrown during build. Printing stack trace: java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=com.google.devtools.build.lib.skyframe.CoverageReportValue$CoverageReportKey@737c7335, actionIndex=1}' (requested by nodes 'File:[[<execution_root>]bazel-
out]_coverage/_coverage_report.dat')
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:499)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:368)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: null for File:[[<execution_root>]bazel-out/host/bin]external/bazel_tools/tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator/Main.jar
        at com.google.common.base.Preconditions.checkState(Preconditions.java:588)
        at com.google.devtools.build.lib.skyframe.PerActionFileCache.getMetadata(PerActionFileCache.java:53)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$DelegatingPairFileCache.getMetadata(SkyframeActionExecutor.java:1330)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.getInputMetadata(TreeNodeRepository.java:500)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:330)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildFromActionInputs(TreeNodeRepository.java:285)
        at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:100)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:101)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:75)
        at com.google.devtools.build.lib.exec.SpawnActionContextMaps$ProxySpawnActionContext.exec(SpawnActionContextMaps.java:362)
        at com.google.devtools.build.lib.bazel.coverage.CoverageReportActionBuilder$CoverageReportAction.execute(CoverageReportActionBuilder.java:125)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:1001)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:930)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$800(SkyframeActionExecutor.java:121)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:770)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:725)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:478)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:519)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:216)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:422)
        ... 4 more

fahhem on Nov 7, 2018