jib: Jib fails with BLOB_UPLOAD_UNKNOWN error on Sonatype Nexus

Environment:

  • Jib version: 2.1.0
  • Build tool: gradle 6.2
  • OS: Linux/macOS

Description of the issue: Jib fails with BLOB_UPLOAD_UNKNOWN error. I have a gradle project with about 15 submodules and I want to push images in our private registry. When I run jib for all sumbodules in my gradle project, sometimes this error occurs. Jib doesn’t fail immediately, it’s often able to create and push image for about 12-13 submodules and only then it crashes. If I rerun jib after waiting a few seconds, all the remaining submodules are processed successfully (currently, this is my workaround). I asked people responsible for our registry and they said that this is probably jib’s problem, because they have not encountered this error using docker cli. They also suggested to disable parallel execution in gradle, which seemingly made jib more stable, but didn’t solve the problem completely.

Execution failed for task ':submodule:jib'.

> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Tried to push BLOB for <remote image url> with digest sha256:277866581d95cdba21d952e55187c0bced4a1f7f0016d3fc2195616f0ab85744 but failed because: other: blob upload unknown to registry | If this is a bug, please file an issue at https://github.com/GoogleContainerTools/jib/issues/new

Expected behavior: Jib runs successfully regardless of number of modules/execution time without BLOB_UPLOAD_UNKNOWN error.

Steps to reproduce:

  1. Create a project with a lot of submodules
  2. Run jib
  3. Jib fails after processing significant amount of sumbodules

jib-gradle-plugin Configuration:

jib {
    from {
        image registryBaseImage
        auth {
            username registryUsername
            password registryPassword
        }
    }
    to {
        image registryRepository + project.name + ':' + project.version
        auth {
            username registryUsername
            password registryPassword
        }
    }
}

Log output:

FAILURE: Build failed with an exception.



* What went wrong:

Execution failed for task ':submodule:jib'.

> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Tried to push BLOB for <remote image url> with digest sha256:277866581d95cdba21d952e55187c0bced4a1f7f0016d3fc2195616f0ab85744 but failed because: other: blob upload unknown to registry | If this is a bug, please file an issue at https://github.com/GoogleContainerTools/jib/issues/new



* Try:

Run with --info or --debug option to get more log output. Run with --scan to get full insights.



* Exception is:

org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':submodule:jib'.

	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:205)

	at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:263)

	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:203)

	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:184)

	at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:114)

	at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)

	at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:62)

	at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)

	at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)

	at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)

	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)

	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)

	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)

	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:416)

	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:406)

	at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)

	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)

	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)

	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:102)

	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)

	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)

	at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:41)

	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:372)

	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:359)

	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:352)

	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:338)

	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:127)

	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:191)

	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:182)

	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:124)

	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)

	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)

	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)

Caused by: org.gradle.internal.UncheckedException: com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Tried to push BLOB for <remote image url> with digest sha256:277866581d95cdba21d952e55187c0bced4a1f7f0016d3fc2195616f0ab85744 but failed because: other: blob upload unknown to registry | If this is a bug, please file an issue at https://github.com/GoogleContainerTools/jib/issues/new

	at org.gradle.internal.UncheckedException.throwAsUncheckedException(UncheckedException.java:67)

	at org.gradle.internal.UncheckedException.throwAsUncheckedException(UncheckedException.java:41)

	at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:107)

	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:49)

	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:42)

	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:28)

	at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:727)

	at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:694)

	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$3.run(ExecuteActionsTaskExecuter.java:568)

	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:402)

	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:394)

	at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)

	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)

	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)

	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:92)

	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)

	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:553)

	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:536)

	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.access$300(ExecuteActionsTaskExecuter.java:109)

	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.executeWithPreviousOutputFiles(ExecuteActionsTaskExecuter.java:276)

	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.execute(ExecuteActionsTaskExecuter.java:265)

	at org.gradle.internal.execution.steps.ExecuteStep.lambda$execute$1(ExecuteStep.java:33)

	at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:33)

	at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:26)

	at org.gradle.internal.execution.steps.CleanupOutputsStep.execute(CleanupOutputsStep.java:67)

	at org.gradle.internal.execution.steps.CleanupOutputsStep.execute(CleanupOutputsStep.java:36)

	at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:49)

	at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:34)

	at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:43)

	at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:73)

	at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:54)

	at org.gradle.internal.execution.steps.CatchExceptionStep.execute(CatchExceptionStep.java:34)

	at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:44)

	at org.gradle.internal.execution.steps.SnapshotOutputsStep.execute(SnapshotOutputsStep.java:54)

	at org.gradle.internal.execution.steps.SnapshotOutputsStep.execute(SnapshotOutputsStep.java:38)

	at org.gradle.internal.execution.steps.BroadcastChangingOutputsStep.execute(BroadcastChangingOutputsStep.java:49)

	at org.gradle.internal.execution.steps.CacheStep.executeWithoutCache(CacheStep.java:153)

	at org.gradle.internal.execution.steps.CacheStep.execute(CacheStep.java:67)

	at org.gradle.internal.execution.steps.CacheStep.execute(CacheStep.java:41)

	at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:44)

	at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:33)

	at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:38)

	at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:24)

	at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:92)

	at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$0(SkipUpToDateStep.java:85)

	at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:55)

	at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:39)

	at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:76)

	at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:37)

	at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:36)

	at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:26)

	at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:94)

	at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:49)

	at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:79)

	at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:53)

	at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:74)

	at org.gradle.internal.execution.steps.SkipEmptyWorkStep.lambda$execute$2(SkipEmptyWorkStep.java:78)

	at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:78)

	at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:34)

	at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:39)

	at org.gradle.internal.execution.steps.LoadExecutionStateStep.execute(LoadExecutionStateStep.java:40)

	at org.gradle.internal.execution.steps.LoadExecutionStateStep.execute(LoadExecutionStateStep.java:28)

	at org.gradle.internal.execution.impl.DefaultWorkExecutor.execute(DefaultWorkExecutor.java:33)

	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:192)

	... 30 more

Caused by: com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Tried to push BLOB for <remote image url> with digest sha256:277866581d95cdba21d952e55187c0bced4a1f7f0016d3fc2195616f0ab85744 but failed because: other: blob upload unknown to registry | If this is a bug, please file an issue at https://github.com/GoogleContainerTools/jib/issues/new

	at com.google.cloud.tools.jib.plugins.common.JibBuildRunner.runBuild(JibBuildRunner.java:278)

	at com.google.cloud.tools.jib.gradle.BuildImageTask.buildImage(BuildImageTask.java:110)

	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

	at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)

	... 91 more

Caused by: com.google.cloud.tools.jib.registry.RegistryErrorException: Tried to push BLOB for <remote image url> with digest sha256:277866581d95cdba21d952e55187c0bced4a1f7f0016d3fc2195616f0ab85744 but failed because: other: blob upload unknown to registry | If this is a bug, please file an issue at https://github.com/GoogleContainerTools/jib/issues/new

	at com.google.cloud.tools.jib.registry.RegistryErrorExceptionBuilder.build(RegistryErrorExceptionBuilder.java:105)

	at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.newRegistryErrorException(RegistryEndpointCaller.java:207)

	at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:154)

	at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:115)

	at com.google.cloud.tools.jib.registry.RegistryClient.callRegistryEndpoint(RegistryClient.java:603)

	at com.google.cloud.tools.jib.registry.RegistryClient.pushBlob(RegistryClient.java:545)

	at com.google.cloud.tools.jib.builder.steps.PushBlobStep.call(PushBlobStep.java:89)

	at com.google.cloud.tools.jib.builder.steps.PushLayerStep.call(PushLayerStep.java:93)

	at com.google.cloud.tools.jib.builder.steps.PushLayerStep.call(PushLayerStep.java:33)

	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)

	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)

	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)

Caused by: com.google.cloud.tools.jib.http.ResponseException: 404 Not Found

{"errors":[{"code":"BLOB_UPLOAD_UNKNOWN","message":"blob upload unknown to registry","detail":"Missing upload with uuid: a9c1d102-00d7-4957-8fd0-a0b85a164072"}]}

	at com.google.cloud.tools.jib.http.FailoverHttpClient.call(FailoverHttpClient.java:323)

	at com.google.cloud.tools.jib.http.FailoverHttpClient.call(FailoverHttpClient.java:244)

	at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:139)

	... 9 more

Caused by: com.google.api.client.http.HttpResponseException: 404 Not Found

{"errors":[{"code":"BLOB_UPLOAD_UNKNOWN","message":"blob upload unknown to registry","detail":"Missing upload with uuid: a9c1d102-00d7-4957-8fd0-a0b85a164072"}]}

	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1113)

	at com.google.cloud.tools.jib.http.FailoverHttpClient.call(FailoverHttpClient.java:317)

	... 11 more

Additional Information:

I asked people responsible for our registry and they said that this is probably jib’s problem, because they have not encountered this error using docker cli. They also suggested to disable parallel execution in gradle, which seemingly made jib more stable, but didn’t solve the problem completely.

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 20 (9 by maintainers)

Commits related to this issue

Most upvoted comments

Adding that this seems to affect ghcr.io (Github Container Registry) as well. Adding -Djib.serialize=true to ./gradlew jib allowed it to succeed.

I have raised an issue to the github support team regarding this issue we experienced with ghcr.io and got a quite thorough response with investigation from their engineering team, I’ll paraphrase:

  • This issue seems to appear on multiple platforms which leads them to believe it is a issue with Jib

also from my support ticket: “one theory being that Jib may be retrying an upload before the previous upload has actually failed. This could also explain the inconsistency of failure you are seeing, as different uploads would take different amounts of time, and thus some would fit within the retry window (thus not failing), while a few might exceed the retry window and thus fail.”

Since this issue is already closed we probably need to open a new one if we want anything further to happen.