bazel: Remote worker throwing exception on upload of docker_build cache
Please provide the following information. The more we know about your system and use case, the more easily and likely we can help.
Description of the problem / feature request / question:
Remote worker throwing exception on upload of docker_build cache.
Genrule:
genrule(
name = "java8",
srcs = [],
outs = ["java8.tar"],
cmd = "docker pull openjdk:8; docker save openjdk:8 > $@",
local = 1,
)
Docker build:
load("@bazel_tools//tools/build_defs/docker:docker.bzl", "docker_build")
docker_build(
name = "image",
base = ":java8",
cmd = "echo 'hello world'",
)
Bazel build exceptiong:
bazel build //third_party/docker:image
INFO: Reading 'startup' options from /home/preston/.bazelrc: --host_jvm_args=-Dbazel.DigestFunction=SHA1
INFO: Found 1 target...
Target //third_party/docker:image failed to build
Use --verbose_failures to see the command lines of failed build steps.
Unhandled exception thrown during build; message: Unrecoverable error while evaluating node 'ACTION_EXECUTION:action 'CreateImage third_party/docker/image-layer.tar' (CreateImage[[Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image.config.sha256, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image.config, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image.layer.sha256, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image.layer, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image.metadata-name.sha256, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image.metadata, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/genfiles]third_party/docker/java8.tar, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/host/bin]external/bazel_tools/tools/build_defs/docker/create_image, Artifact:[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1[source]]external/bazel_tools/tools/build_defs/docker/create_image.py, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/host/internal]_middlemen/external_Sbazel_Utools_Stools_Sbuild_Udefs_Sdocker_Screate_Uimage-runfiles] -> [Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image-layer.tar]])' (requested by nodes 'ARTIFACT:third_party/docker/image-layer.tar //third_party/docker:image a1aa5b17a64171df0d4f40bcacc0492a (527716221 628672696)')
INFO: Elapsed time: 20.140s, Critical Path: 17.98s
java.lang.RuntimeException: Unrecoverable error while evaluating node 'ACTION_EXECUTION:action 'CreateImage third_party/docker/image-layer.tar' (CreateImage[[Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image.config.sha256, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image.config, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image.layer.sha256, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image.layer, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image.metadata-name.sha256, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image.metadata, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/genfiles]third_party/docker/java8.tar, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/host/bin]external/bazel_tools/tools/build_defs/docker/create_image, Artifact:[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1[source]]external/bazel_tools/tools/build_defs/docker/create_image.py, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/host/internal]_middlemen/external_Sbazel_Utools_Stools_Sbuild_Udefs_Sdocker_Screate_Uimage-runfiles] -> [Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image-layer.tar]])' (requested by nodes 'ARTIFACT:third_party/docker/image-layer.tar //third_party/docker:image a1aa5b17a64171df0d4f40bcacc0492a (527716221 628672696)')
at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:438)
at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:501)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: gRPC terminated prematurely: java.lang.RuntimeException: java.lang.IllegalArgumentException
at com.google.devtools.build.lib.remote.GrpcActionCache.uploadChunks(GrpcActionCache.java:532)
at com.google.devtools.build.lib.remote.GrpcActionCache.uploadAllResults(GrpcActionCache.java:413)
at com.google.devtools.build.lib.remote.RemoteSpawnStrategy.execLocally(RemoteSpawnStrategy.java:121)
at com.google.devtools.build.lib.remote.RemoteSpawnStrategy.exec(RemoteSpawnStrategy.java:228)
at com.google.devtools.build.lib.analysis.actions.SpawnAction.internalExecute(SpawnAction.java:265)
at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:273)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:783)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:723)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$800(SkyframeActionExecutor.java:102)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:613)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:575)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:380)
at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:444)
at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:197)
at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:373)
... 4 more
java.lang.RuntimeException: Unrecoverable error while evaluating node 'ACTION_EXECUTION:action 'CreateImage third_party/docker/image-layer.tar' (CreateImage[[Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image.config.sha256, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image.config, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image.layer.sha256, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image.layer, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image.metadata-name.sha256, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image.metadata, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/genfiles]third_party/docker/java8.tar, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/host/bin]external/bazel_tools/tools/build_defs/docker/create_image, Artifact:[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1[source]]external/bazel_tools/tools/build_defs/docker/create_image.py, Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/host/internal]_middlemen/external_Sbazel_Utools_Stools_Sbuild_Udefs_Sdocker_Screate_Uimage-runfiles] -> [Artifact:[[/home/preston/.cache/bazel/_bazel_preston/7fcd27cc7c3bb4755396ba74e5927be1/execroot/workspace]bazel-out/local-fastbuild/bin]third_party/docker/image-layer.tar]])' (requested by nodes 'ARTIFACT:third_party/docker/image-layer.tar //third_party/docker:image a1aa5b17a64171df0d4f40bcacc0492a (527716221 628672696)')
at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:438)
at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:501)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: gRPC terminated prematurely: java.lang.RuntimeException: java.lang.IllegalArgumentException
at com.google.devtools.build.lib.remote.GrpcActionCache.uploadChunks(GrpcActionCache.java:532)
at com.google.devtools.build.lib.remote.GrpcActionCache.uploadAllResults(GrpcActionCache.java:413)
at com.google.devtools.build.lib.remote.RemoteSpawnStrategy.execLocally(RemoteSpawnStrategy.java:121)
at com.google.devtools.build.lib.remote.RemoteSpawnStrategy.exec(RemoteSpawnStrategy.java:228)
at com.google.devtools.build.lib.analysis.actions.SpawnAction.internalExecute(SpawnAction.java:265)
at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:273)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:783)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:723)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$800(SkyframeActionExecutor.java:102)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:613)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:575)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:380)
at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:444)
at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:197)
at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:373)
... 4 more
Build worker logged exception:
Mar 14, 2017 9:10:50 PM com.google.devtools.build.remote.RemoteWorker$CasServer$1 onNext
WARNING: Request failed: java.lang.IllegalArgumentException
Mar 14, 2017 9:10:50 PM com.google.devtools.build.remote.RemoteWorker$CasServer$1 onNext
WARNING: Request failed: java.lang.IllegalArgumentException: Missing input chunk for digest <digest: ccb4e763bd55ce04d008fbefb09d9be4432fbbfd, size: 699234304 bytes>
Mar 14, 2017 9:10:50 PM com.google.devtools.build.remote.RemoteWorker$CasServer$1 onNext
WARNING: Request failed: java.lang.IllegalArgumentException: Missing input chunk for digest <digest: ccb4e763bd55ce04d008fbefb09d9be4432fbbfd, size: 699234304 bytes>
Mar 14, 2017 9:10:50 PM com.google.devtools.build.remote.RemoteWorker$CasServer$1 onError
WARNING: Request errored remotely: io.grpc.StatusException: CANCELLED
If possible, provide a minimal example to reproduce the problem:
Remote worker running as such: bazel-bin/src/tools/remote_worker/remote_worker --work_path /tmp --listen_port=8080
Building that docker_build above with build --spawn_strategy=remote --remote_cache=172.17.0.3:8080
in ~/.bazelrc
Environment info
-
Operating System: Ubuntu 16.04
-
Bazel version (output of
bazel info release
): release 0.4.4 -
If
bazel info release
returns “development version” or “(@non-git)”, please tell us what source tree you compiled Bazel from; git commit hash is appreciated (git rev-parse HEAD
): The worker is running from commit 2046bb480075a8f412cb51882e64e31324fc57de
Have you found anything relevant by searching the web? (e.g. GitHub issues, email threads in the bazel-discuss@googlegroups.com archive)
Nope.
Anything else, information or logs or outputs that would be helpful?
(If they are large, please upload as attachment or provide link).
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 18 (12 by maintainers)
Closing this since the gRPC-based protocol can handle large files now. I filed #3250 for the test cache.
I happened to notice this thread, and wanted to chime in (if only to hijack a bit).
You should consider using the new
bazelbuild/rules_docker
for pulling your image down (it supports pushing too).Are you just looking for a java8 base to overlay your deploy jar on?