bazel: IllegalStateException: Did not write as much data as expected

We’ve seen the following failure cause 2 flakes on our CI server (out of around 30 builds since upgrading from Bazel 0.15.0 to Bazel 0.16.1).

Unhandled exception thrown during build; message: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=@com_fasterxml_jackson_dataformat_jackson_dataformat_yaml//jar:jar BuildConfigurationValue.Key[56993b598966274b11f9c239ecf81d3c] false, actionIndex=0}' (requested by nodes 'external/com_fasterxml_jackson_dataformat_jackson_dataformat_yaml/jar/_ijar/jar/external/com_fasterxml_jackson_dataformat_jackson_dataformat_yaml/jar/jackson-dataformat-yaml-2.9.0-ijar.jar @com_fasterxml_jackson_dataformat_jackson_dataformat_yaml//jar:jar BuildConfigurationValue.Key[56993b598966274b11f9c239ecf81d3c] false')
INFO: Elapsed time: 178.732s, Critical Path: 6.46s
INFO: 186 processes: 186 remote cache hit.
FAILED: Build did NOT complete successfully
java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=@com_fasterxml_jackson_dataformat_jackson_dataformat_yaml//jar:jar BuildConfigurationValue.Key[56993b598966274b11f9c239ecf81d3c] false, actionIndex=0}' (requested by nodes 'external/com_fasterxml_jackson_dataformat_jackson_dataformat_yaml/jar/_ijar/jar/external/com_fasterxml_jackson_dataformat_jackson_dataformat_yaml/jar/jackson-dataformat-yaml-2.9.0-ijar.jar @com_fasterxml_jackson_dataformat_jackson_dataformat_yaml//jar:jar BuildConfigurationValue.Key[56993b598966274b11f9c239ecf81d3c] false')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:477)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:355)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Did not write as much data as expected.
	at com.google.protobuf.CodedOutputStream.checkNoSpaceLeft(CodedOutputStream.java:973)
	at com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:73)
	at com.google.devtools.build.lib.remote.util.DigestUtil.compute(DigestUtil.java:85)
	at com.google.devtools.build.lib.remote.TreeNodeRepository.getOrComputeDirectory(TreeNodeRepository.java:391)
	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:411)
	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
	at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:101)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:90)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:63)
	at com.google.devtools.build.lib.exec.SpawnActionContextMaps$ProxySpawnActionContext.exec(SpawnActionContextMaps.java:362)
	at com.google.devtools.build.lib.analysis.actions.SpawnAction.internalExecute(SpawnAction.java:287)
	at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:294)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:978)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:910)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$900(SkyframeActionExecutor.java:120)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:763)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:718)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:457)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:513)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:227)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:400)
	... 4 more

I haven’t seen it occur on my workstation - only on the CI build.

What operating system are you running Bazel on?

Ubuntu 14.04 host, running Bazel inside a docker container based on debian:jessie-backports.

What’s the output of bazel info release?

0.16.1

Have you found anything relevant by searching the web?

https://github.com/bazelbuild/rules_scala/issues/589 seems to be the same issue.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 16 (10 by maintainers)

Most upvoted comments

We haven’t seen it on 0.22 since applying the recommended workaround here https://github.com/bazelbuild/bazel/issues/6136#issuecomment-425710997

Presumably this will be resolved when 8db50b64b9600e392c3aa4644fdd87b2f2587ba2 is released with 0.23.

On Thu, Feb 14, 2019, at 14:03, Keith Smiley wrote:

We have seen this on macOS CI with 0.22.0

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bazelbuild/bazel/issues/6136#issuecomment-463818465, or mute the thread https://github.com/notifications/unsubscribe-auth/AANZTn4Evt8zMybc0gcBRdbnUsiW41taks5vNd1FgaJpZM4WlW9y.