bazel: IllegalStateException: Did not write as much data as expected

We’ve seen the following failure cause 2 flakes on our CI server (out of around 30 builds since upgrading from Bazel 0.15.0 to Bazel 0.16.1).

Unhandled exception thrown during build; message: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=@com_fasterxml_jackson_dataformat_jackson_dataformat_yaml//jar:jar BuildConfigurationValue.Key[56993b598966274b11f9c239ecf81d3c] false, actionIndex=0}' (requested by nodes 'external/com_fasterxml_jackson_dataformat_jackson_dataformat_yaml/jar/_ijar/jar/external/com_fasterxml_jackson_dataformat_jackson_dataformat_yaml/jar/jackson-dataformat-yaml-2.9.0-ijar.jar @com_fasterxml_jackson_dataformat_jackson_dataformat_yaml//jar:jar BuildConfigurationValue.Key[56993b598966274b11f9c239ecf81d3c] false')
INFO: Elapsed time: 178.732s, Critical Path: 6.46s
INFO: 186 processes: 186 remote cache hit.
FAILED: Build did NOT complete successfully
java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=@com_fasterxml_jackson_dataformat_jackson_dataformat_yaml//jar:jar BuildConfigurationValue.Key[56993b598966274b11f9c239ecf81d3c] false, actionIndex=0}' (requested by nodes 'external/com_fasterxml_jackson_dataformat_jackson_dataformat_yaml/jar/_ijar/jar/external/com_fasterxml_jackson_dataformat_jackson_dataformat_yaml/jar/jackson-dataformat-yaml-2.9.0-ijar.jar @com_fasterxml_jackson_dataformat_jackson_dataformat_yaml//jar:jar BuildConfigurationValue.Key[56993b598966274b11f9c239ecf81d3c] false')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:477)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:355)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Did not write as much data as expected.
	at com.google.protobuf.CodedOutputStream.checkNoSpaceLeft(CodedOutputStream.java:973)
	at com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:73)
	at com.google.devtools.build.lib.remote.util.DigestUtil.compute(DigestUtil.java:85)
	at com.google.devtools.build.lib.remote.TreeNodeRepository.getOrComputeDirectory(TreeNodeRepository.java:391)
	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:411)
	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
	at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:101)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:90)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:63)
	at com.google.devtools.build.lib.exec.SpawnActionContextMaps$ProxySpawnActionContext.exec(SpawnActionContextMaps.java:362)
	at com.google.devtools.build.lib.analysis.actions.SpawnAction.internalExecute(SpawnAction.java:287)
	at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:294)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:978)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:910)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$900(SkyframeActionExecutor.java:120)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:763)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:718)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:457)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:513)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:227)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:400)
	... 4 more

I haven’t seen it occur on my workstation - only on the CI build.

What operating system are you running Bazel on?

Ubuntu 14.04 host, running Bazel inside a docker container based on debian:jessie-backports.

What’s the output of bazel info release?

0.16.1

Have you found anything relevant by searching the web?

https://github.com/bazelbuild/rules_scala/issues/589 seems to be the same issue.