bazel: bazel C++ coverage for Envoy crashes JVM

With Bazel 0.22.0 and gcc 7.3.0, if you clone https://github.com/envoyproxy/envoy and run:

bazel coverage --experimental_cc_coverage test/...  --instrumentation_filter=//source/...,//include/... --coverage_report_generator=@bazel_tools//tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator:Main --combined_report=lcov

the following stack trace is emitted:

Internal error thrown during build. Printing stack trace: java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=com.google.devtools.build.lib.skyframe.CoverageReportValue$CoverageReportKey@2fe2e16c, actionIndex=1}' (requested by nodes 'File:[[<execution
_root>]bazel-out]_coverage/_coverage_report.dat')                                                              
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:514)                              
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:368)         
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)                                     
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)                                    
        at java.base/java.lang.Thread.run(Unknown Source)                                              
Caused by: java.lang.IllegalStateException: null for File:[[<execution_root>]bazel-out/host/bin]external/bazel_tools/tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator/Main.jar
        at com.google.devtools.build.lib.skyframe.ActionMetadataHandler.getMetadata(ActionMetadataHandler.java:206)                                   
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$DelegatingPairFileCache.getMetadata(SkyframeActionExecutor.java:1377)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.getInputMetadata(TreeNodeRepository.java:500)           
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:330)            
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)                                                                                                                                                                                       
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)         
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)                         
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)    
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)         
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)                                                                                                  
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)    
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)                               
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348) 
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
...
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildParentNode(TreeNodeRepository.java:348)
        at com.google.devtools.build.lib.remote.TreeNodeRepository.buildFromActionInputs(TreeNodeRepository.java:285)
        at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:104)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:102)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:76)
        at com.google.devtools.build.lib.exec.ProxySpawnActionContext.exec(ProxySpawnActionContext.java:44)
        at com.google.devtools.build.lib.bazel.coverage.CoverageReportActionBuilder$CoverageReportAction.execute(CoverageReportActionBuilder.java:125)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.executeAction(SkyframeActionExecutor.java:889)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:854)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:787)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:541)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:697)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:240)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:437)
        ... 4 more

This seems to occur when running hot_restart_test, which isn’t even a direct C++ unit test, but instead a shell script integration test wrapped around the Envoy binary, which is executed multiple times under the script. I’m guessing there might be some issue related to that.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 22 (14 by maintainers)

Most upvoted comments

Yes, I had build --remote_http_cache=http://localhost:9090 enabled for just caching. There is no remote execution happening though. Thanks for spotting this, if I disable that then I can get a coverage report.

Local disk cache also (confusingly) behind the RemoteSpawnCache interface. And yes, this confused me, too 😃