bazel: Bazel cquery crashing with a stack trace, asking to ping "b/150301500 or contact the blaze configurability team"
Description of the problem / feature request:
Bazel cquery will crash with something similar to:
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.IllegalStateException: query-requested node 'ConfiguredTargetKey{label=@org_golang_x_tools//cmd/goimports:goimports, config=BuildConfigurationValue.Key[4d498c45554873ec5759362a68427cef5b8c3da393a8813902396bb756aafdf5]}' was unavailable in the query environment graph. If you come across this error, please ping b/150301500 or contact the blaze configurability team.
or
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.NullPointerException: ToolchainContextKey{configurationKey=BuildConfigurationValue.Key[beffc25b26347f53bb6e6aa5e7c410d14dd4082be0e4b4b9ebdcf0cee34e129c], requiredToolchainTypeLabels=[@io_bazel_rules_go//go:toolchain], execConstraintLabels=[], forceExecutionPlatform=Optional.empty, debugTarget=false}
Bugs: what’s the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Any cquery invocation has a small chance to crash. We do have a sizeable build graph.
What operating system are you running Bazel on?
Seen this both on OSX and Linux.
What’s the output of bazel info release
?
release 5.0.0, though we have seen this with previous relases too.
Any other information, logs, or outputs that you want to share?
Here are the two recent stack traces:
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.IllegalStateException: query-requested node 'ConfiguredTargetKey{label=@org_golang_x_tools//cmd/goimports:goimports, config=BuildConfigurationValue.Key[4d498c45554873ec5759362a68427cef5b8c3da393a8813902396bb756aafdf5]}' was unavailable in the query environment graph. If you come across this error, please ping b/150301500 or contact the blaze configurability team.
at com.google.common.base.Preconditions.checkState(Preconditions.java:591)
at com.google.devtools.build.lib.query2.PostAnalysisQueryEnvironment.targetifyValues(PostAnalysisQueryEnvironment.java:428)
at com.google.devtools.build.lib.query2.PostAnalysisQueryEnvironment.targetifyValues(PostAnalysisQueryEnvironment.java:476)
at com.google.devtools.build.lib.query2.PostAnalysisQueryEnvironment.getReverseDeps(PostAnalysisQueryEnvironment.java:296)
at com.google.devtools.build.lib.query2.PostAnalysisQueryEnvironment.getReverseDeps(PostAnalysisQueryEnvironment.java:101)
at com.google.devtools.build.lib.query2.engine.AllRdepsFunction$1.process(AllRdepsFunction.java:106)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment$1.call(AbstractQueryEnvironment.java:135)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment$1.call(AbstractQueryEnvironment.java:132)
at com.google.common.util.concurrent.CombinedFuture$CallableInterruptibleTask.runInterruptibly(CombinedFuture.java:183)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at com.google.common.util.concurrent.CombinedFuture$CombinedFutureInterruptibleTask.execute(CombinedFuture.java:104)
at com.google.common.util.concurrent.CombinedFuture.handleAllCompleted(CombinedFuture.java:62)
at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:283)
at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:265)
at com.google.common.util.concurrent.AggregateFuture.access$200(AggregateFuture.java:42)
at com.google.common.util.concurrent.AggregateFuture$1.run(AggregateFuture.java:147)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:724)
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:126)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment$QueryTaskFutureImpl.addListener(AbstractQueryEnvironment.java:78)
at com.google.common.util.concurrent.AggregateFuture.init(AggregateFuture.java:128)
at com.google.common.util.concurrent.CombinedFuture.<init>(CombinedFuture.java:52)
at com.google.common.util.concurrent.Futures$FutureCombiner.call(Futures.java:719)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment.whenAllSucceedCall(AbstractQueryEnvironment.java:183)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment.whenSucceedsCall(AbstractQueryEnvironment.java:160)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment.eval(AbstractQueryEnvironment.java:130)
at com.google.devtools.build.lib.query2.engine.AllRdepsFunction.eval(AllRdepsFunction.java:88)
at com.google.devtools.build.lib.query2.engine.RdepsFunction.lambda$evalWithBoundedDepth$0(RdepsFunction.java:130)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment.lambda$transformAsync$0(AbstractQueryEnvironment.java:192)
at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213)
at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202)
at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:724)
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:126)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment$QueryTaskFutureImpl.addListener(AbstractQueryEnvironment.java:78)
at com.google.common.util.concurrent.AbstractTransformFuture.create(AbstractTransformFuture.java:39)
at com.google.common.util.concurrent.Futures.transformAsync(Futures.java:446)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment.transformAsync(AbstractQueryEnvironment.java:190)
at com.google.devtools.build.lib.query2.engine.RdepsFunction.evalWithBoundedDepth(RdepsFunction.java:132)
at com.google.devtools.build.lib.query2.engine.RdepsFunction.eval(RdepsFunction.java:75)
at com.google.devtools.build.lib.query2.engine.FunctionExpression.eval(FunctionExpression.java:55)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment.eval(AbstractQueryEnvironment.java:129)
at com.google.devtools.build.lib.query2.engine.RegexFilterExpression.eval(RegexFilterExpression.java:72)
at com.google.devtools.build.lib.query2.engine.FunctionExpression.eval(FunctionExpression.java:55)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment.eval(AbstractQueryEnvironment.java:129)
at com.google.devtools.build.lib.query2.common.AbstractBlazeQueryEnvironment.evalTopLevelInternal(AbstractBlazeQueryEnvironment.java:115)
at com.google.devtools.build.lib.query2.common.AbstractBlazeQueryEnvironment.evaluateQuery(AbstractBlazeQueryEnvironment.java:155)
at com.google.devtools.build.lib.query2.PostAnalysisQueryEnvironment.evaluateQuery(PostAnalysisQueryEnvironment.java:152)
at com.google.devtools.build.lib.buildtool.PostAnalysisQueryBuildTool.doPostAnalysisQuery(PostAnalysisQueryBuildTool.java:182)
at com.google.devtools.build.lib.buildtool.PostAnalysisQueryBuildTool.postProcessAnalysisResult(PostAnalysisQueryBuildTool.java:78)
at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:229)
at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:446)
at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:413)
at com.google.devtools.build.lib.runtime.commands.CqueryCommand.exec(CqueryCommand.java:152)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:585)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:231)
at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:550)
at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$1(GrpcServerImpl.java:614)
at io.grpc.Context$1.run(Context.java:579)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Exception
Bazel query error (37): INFO: Reading 'startup' options from /Users/kdavydov/repos/grail/.bazelrc: --host_jvm_args=-Xmx4G
Bazel query error (37): INFO: Reading 'startup' options from /Users/bzhang/grail/.bazelrc: --host_jvm_args=-Xmx4G
INFO: Options provided by the client:
Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'cquery' from /Users/bzhang/grail/.bazelrc:
Inherited 'build' options: --workspace_status_command=bazel/get_workspace_status.sh --define=gotags=bazel --stamp --copt=-ggdb -c opt --incompatible_enable_cc_toolchain_resolution --define=use_fast_cpp_protos=true --incompatible_strict_action_env --action_env=PATH=/usr/local/bin:/usr/sbin:/bin:/usr/bin --test_env=V23_CREDENTIALS --test_env=V23_CREDENTIALS_NO_LOCK=1 --test_env=V23_CREDENTIALS_NO_AGENT=1 --java_runtime_version=remotejdk_11 --tool_java_runtime_version=remotejdk_11 --tls_client_certificate=bazel/buildbuddy/buildbuddy-cert.pem --tls_client_key=bazel/buildbuddy/buildbuddy-key.pem --announce_rc
INFO: Found 93 targets...
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.NullPointerException: ToolchainContextKey{configurationKey=BuildConfigurationValue.Key[beffc25b26347f53bb6e6aa5e7c410d14dd4082be0e4b4b9ebdcf0cee34e129c], requiredToolchainTypeLabels=[@io_bazel_rules_go//go:toolchain], execConstraintLabels=[], forceExecutionPlatform=Optional.empty, debugTarget=false}
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:899)
at com.google.devtools.build.skyframe.DelegatingWalkableGraph.getDirectDeps(DelegatingWalkableGraph.java:124)
at com.google.devtools.build.lib.query2.PostAnalysisQueryEnvironment.targetifyValues(PostAnalysisQueryEnvironment.java:462)
at com.google.devtools.build.lib.query2.PostAnalysisQueryEnvironment.targetifyValues(PostAnalysisQueryEnvironment.java:476)
at com.google.devtools.build.lib.query2.PostAnalysisQueryEnvironment.getReverseDeps(PostAnalysisQueryEnvironment.java:296)
at com.google.devtools.build.lib.query2.PostAnalysisQueryEnvironment.getReverseDeps(PostAnalysisQueryEnvironment.java:101)
at com.google.devtools.build.lib.query2.engine.AllRdepsFunction$1.process(AllRdepsFunction.java:106)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment$1.call(AbstractQueryEnvironment.java:135)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment$1.call(AbstractQueryEnvironment.java:132)
at com.google.common.util.concurrent.CombinedFuture$CallableInterruptibleTask.runInterruptibly(CombinedFuture.java:183)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at com.google.common.util.concurrent.CombinedFuture$CombinedFutureInterruptibleTask.execute(CombinedFuture.java:104)
at com.google.common.util.concurrent.CombinedFuture.handleAllCompleted(CombinedFuture.java:62)
at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:283)
at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:265)
at com.google.common.util.concurrent.AggregateFuture.access$200(AggregateFuture.java:42)
at com.google.common.util.concurrent.AggregateFuture$1.run(AggregateFuture.java:147)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:724)
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:126)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment$QueryTaskFutureImpl.addListener(AbstractQueryEnvironment.java:78)
at com.google.common.util.concurrent.AggregateFuture.init(AggregateFuture.java:128)
at com.google.common.util.concurrent.CombinedFuture.<init>(CombinedFuture.java:52)
at com.google.common.util.concurrent.Futures$FutureCombiner.call(Futures.java:719)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment.whenAllSucceedCall(AbstractQueryEnvironment.java:183)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment.whenSucceedsCall(AbstractQueryEnvironment.java:160)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment.eval(AbstractQueryEnvironment.java:130)
at com.google.devtools.build.lib.query2.engine.AllRdepsFunction.eval(AllRdepsFunction.java:88)
at com.google.devtools.build.lib.query2.engine.RdepsFunction.lambda$evalWithBoundedDepth$0(RdepsFunction.java:130)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment.lambda$transformAsync$0(AbstractQueryEnvironment.java:192)
at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213)
at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202)
at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:724)
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:126)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment$QueryTaskFutureImpl.addListener(AbstractQueryEnvironment.java:78)
at com.google.common.util.concurrent.AbstractTransformFuture.create(AbstractTransformFuture.java:39)
at com.google.common.util.concurrent.Futures.transformAsync(Futures.java:446)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment.transformAsync(AbstractQueryEnvironment.java:190)
at com.google.devtools.build.lib.query2.engine.RdepsFunction.evalWithBoundedDepth(RdepsFunction.java:132)
at com.google.devtools.build.lib.query2.engine.RdepsFunction.eval(RdepsFunction.java:75)
at com.google.devtools.build.lib.query2.engine.FunctionExpression.eval(FunctionExpression.java:55)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment.eval(AbstractQueryEnvironment.java:129)
at com.google.devtools.build.lib.query2.engine.RegexFilterExpression.eval(RegexFilterExpression.java:72)
at com.google.devtools.build.lib.query2.engine.FunctionExpression.eval(FunctionExpression.java:55)
at com.google.devtools.build.lib.query2.engine.AbstractQueryEnvironment.eval(AbstractQueryEnvironment.java:129)
at com.google.devtools.build.lib.query2.common.AbstractBlazeQueryEnvironment.evalTopLevelInternal(AbstractBlazeQueryEnvironment.java:115)
at com.google.devtools.build.lib.query2.common.AbstractBlazeQueryEnvironment.evaluateQuery(AbstractBlazeQueryEnvironment.java:155)
at com.google.devtools.build.lib.query2.PostAnalysisQueryEnvironment.evaluateQuery(PostAnalysisQueryEnvironment.java:152)
at com.google.devtools.build.lib.buildtool.PostAnalysisQueryBuildTool.doPostAnalysisQuery(PostAnalysisQueryBuildTool.java:182)
at com.google.devtools.build.lib.buildtool.PostAnalysisQueryBuildTool.postProcessAnalysisResult(PostAnalysisQueryBuildTool.java:78)
at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:229)
at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:446)
at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:413)
at com.google.devtools.build.lib.runtime.commands.CqueryCommand.exec(CqueryCommand.java:152)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:585)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:231)
at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:550)
at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$1(GrpcServerImpl.java:614)
at io.grpc.Context$1.run(Context.java:579)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 3
- Comments: 15 (11 by maintainers)
Commits related to this issue
- bazel not blaze (part of issue #15079) Don't redirect people to internal Google services! part of #15079 Closes #15080. PiperOrigin-RevId: 436233697 — committed to bazelbuild/bazel by gkgoat1 2 years ago
- Don't call `bazel info` in `bazel` wrapper script (#1074) Summary: We were seeing `ci/bazel_build_deps.sh` fail on many peoples diffs. The failure was a strange looking bazel fatal error seemingly ... — committed to pixie-io/pixie by JamesMBartlett a year ago
We are running into this as well and following the error message to post here. In our case, after we upgraded one of the dependency rule set, it now constantly happens on
cquery
with test targets.I just hit this on successive cquery invocations with no changes to my Bazel files and no
bazel info
invocations in between. I did run slightly different queries (I was querying for deps of specific targets, with the failed second query looking at a different target than the first), but no other actions I’m aware of that match the things listed in this thread. I then re-ran the failed query and it succeeded, so it’s not reliably reproducible.There were no loading/analysis errors in any of the three queries.
@sgowroji https://github.com/bazelbuild/bazel/labels/cquery_aspects is specifically for deficiencies in cquery + aspect support. I don’t believe aspects are a part of this report?