jib: Unrecognized field "LayerSources" error when processing an image in Windows Containers

Jib version: 2.0.1-SNAPSHOT (84933d528b4e4d445d88686e7c7db3945d6c896f)
Build tool: Gradle 5.6.2
OS: Windows 10

Description of the issue: There is following error when trying to build an image on Windows Containers:

Execution failed for task ':jibDockerBuild'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Unrecognized field "LayerSources" (class com.google.cloud.tools.jib.docker.json.DockerManifestEntryTemplate), not marked as ignorable (3 known properties: "config", "repoTags", "layers"])
   at [Source: (sun.nio.ch.ChannelInputStream); line: 1, column: 1015] (through reference chain: java.lang.Object[][0]->com.google.cloud.tools.jib.docker.json.DockerManifestEntryTemplate["LayerSources"])

Expected behavior: No error.

Steps to reproduce: 0. Be on Windows machine with Docker installed, and switched to Windows Containers mode.

Build jib from master as decribed here
docker pull openjdk:11.0.1-windowsservercore-ltsc2016 (to load the image to the cache)
Use build.gradle as shown below.
Execute jibDockerBuild.

jib-gradle-plugin Configuration:

jib {
  from {
        image = 'docker://openjdk:11.0.1-windowsservercore-ltsc2016'
  }
}

Log output:

...
Executing tasks:
[===========                   ] 37.5% complete
> saving base image openjdk:11.0.1-windowsserve...


Executing tasks:
[===========                   ] 37.5% complete
> processing base image openjdk:11.0.1-windowss...


> Task :jibDockerBuild FAILED
7 actionable tasks: 1 executed, 6 up-to-date

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':jibDockerBuild'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Unrecognized field "LayerSources" (class com.google.cloud.tools.jib.docker.json.DockerManifestEntryTemplate), not marked as ignorable (3 known properties: "config", "repoTags", "layers"])
   at [Source: (sun.nio.ch.ChannelInputStream); line: 1, column: 1015] (through reference chain: java.lang.Object[][0]->com.google.cloud.tools.jib.docker.json.DockerManifestEntryTemplate["LayerSources"])

* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':jibDockerBuild'.
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$3.accept(ExecuteActionsTaskExecuter.java:166)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$3.accept(ExecuteActionsTaskExecuter.java:163)
	at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:191)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:156)
	at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:62)
	at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:108)
	at org.gradle.api.internal.tasks.execution.ResolveBeforeExecutionOutputsTaskExecuter.execute(ResolveBeforeExecutionOutputsTaskExecuter.java:67)
	at org.gradle.api.internal.tasks.execution.ResolveAfterPreviousExecutionStateTaskExecuter.execute(ResolveAfterPreviousExecutionStateTaskExecuter.java:46)
	at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:94)
	at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
	at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:95)
	at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
	at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
	at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:416)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:406)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:102)
	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
	at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:43)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:355)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:343)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:336)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:322)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker$1.execute(DefaultPlanExecutor.java:134)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker$1.execute(DefaultPlanExecutor.java:129)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:202)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:193)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:129)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
Caused by: org.gradle.internal.UncheckedException: com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Unrecognized field "LayerSources" (class com.google.cloud.tools.jib.docker.json.DockerManifestEntryTemplate), not marked as ignorable (3 known properties: "config", "repoTags", "layers"])
 at [Source: (sun.nio.ch.ChannelInputStream); line: 1, column: 1015] (through reference chain: java.lang.Object[][0]->com.google.cloud.tools.jib.docker.json.DockerManifestEntryTemplate["LayerSources"])
	at org.gradle.internal.UncheckedException.throwAsUncheckedException(UncheckedException.java:67)
	at org.gradle.internal.UncheckedException.throwAsUncheckedException(UncheckedException.java:41)
	at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:106)
	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:49)
	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:42)
	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:28)
	at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:717)
	at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:684)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$5.run(ExecuteActionsTaskExecuter.java:476)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:402)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:394)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:92)
	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:461)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:444)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.access$200(ExecuteActionsTaskExecuter.java:93)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.execute(ExecuteActionsTaskExecuter.java:237)
	at org.gradle.internal.execution.steps.ExecuteStep.lambda$execute$1(ExecuteStep.java:33)
	at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:33)
	at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:26)
	at org.gradle.internal.execution.steps.CleanupOutputsStep.execute(CleanupOutputsStep.java:58)
	at org.gradle.internal.execution.steps.CleanupOutputsStep.execute(CleanupOutputsStep.java:35)
	at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:48)
	at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:33)
	at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:39)
	at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:73)
	at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:54)
	at org.gradle.internal.execution.steps.CatchExceptionStep.execute(CatchExceptionStep.java:35)
	at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:51)
	at org.gradle.internal.execution.steps.SnapshotOutputsStep.execute(SnapshotOutputsStep.java:45)
	at org.gradle.internal.execution.steps.SnapshotOutputsStep.execute(SnapshotOutputsStep.java:31)
	at org.gradle.internal.execution.steps.CacheStep.executeWithoutCache(CacheStep.java:208)
	at org.gradle.internal.execution.steps.CacheStep.execute(CacheStep.java:70)
	at org.gradle.internal.execution.steps.CacheStep.execute(CacheStep.java:45)
	at org.gradle.internal.execution.steps.BroadcastChangingOutputsStep.execute(BroadcastChangingOutputsStep.java:49)
	at org.gradle.internal.execution.steps.StoreSnapshotsStep.execute(StoreSnapshotsStep.java:43)
	at org.gradle.internal.execution.steps.StoreSnapshotsStep.execute(StoreSnapshotsStep.java:32)
	at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:38)
	at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:24)
	at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:96)
	at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$0(SkipUpToDateStep.java:89)
	at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:54)
	at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:38)
	at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:76)
	at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:37)
	at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:36)
	at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:26)
	at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:90)
	at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:48)
	at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:69)
	at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:47)
	at org.gradle.internal.execution.impl.DefaultWorkExecutor.execute(DefaultWorkExecutor.java:33)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:140)
	... 34 more
Caused by: com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Unrecognized field "LayerSources" (class com.google.cloud.tools.jib.docker.json.DockerManifestEntryTemplate), not marked as ignorable (3 known properties: "config", "repoTags", "layers"])
 at [Source: (sun.nio.ch.ChannelInputStream); line: 1, column: 1015] (through reference chain: java.lang.Object[][0]->com.google.cloud.tools.jib.docker.json.DockerManifestEntryTemplate["LayerSources"])
	at com.google.cloud.tools.jib.plugins.common.JibBuildRunner.runBuild(JibBuildRunner.java:283)
	at com.google.cloud.tools.jib.gradle.BuildDockerTask.buildDocker(BuildDockerTask.java:105)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:103)
	... 87 more
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "LayerSources" (class com.google.cloud.tools.jib.docker.json.DockerManifestEntryTemplate), not marked as ignorable (3 known properties: "config", "repoTags", "layers"])
 at [Source: (sun.nio.ch.ChannelInputStream); line: 1, column: 1015] (through reference chain: java.lang.Object[][0]->com.google.cloud.tools.jib.docker.json.DockerManifestEntryTemplate["LayerSources"])
	at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
	at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:823)
	at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1153)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1589)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1567)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:294)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
	at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:195)
	at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:21)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4014)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3071)
	at com.google.cloud.tools.jib.builder.steps.LocalBaseImageSteps.cacheDockerImageTar(LocalBaseImageSteps.java:209)
	at com.google.cloud.tools.jib.builder.steps.LocalBaseImageSteps.lambda$retrieveDockerDaemonLayersStep$0(LocalBaseImageSteps.java:126)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)

Additional Information: Please note that I’m specifying the from Docker image with the docker:// prefix, so it’s loaded from the Docker daemon cache.

It’s also related to #2215 when it is described that it is required to use the Docker daemon cache in order to make it working with Windows Containers (#1568 will allow to use Windows Containers in the standard way).

This issue follows #2270 that solved different problem with Windows Containers.

About this issue

Original URL
State: open
Created 4 years ago
Comments: 19 (10 by maintainers)

Most upvoted comments

Never mind. I managed to install Docker on Windows on GCE and checked these myself. For testing purposes, I think we can just ignore LayerSources and keep moving forward. These layers are a subset of of Layers. The exported tar has all the layer directories and tarballs.

chanseokoh on Feb 12, 2020

it needs them to start a container based on that image.

Ah, of course. I guess downloading and storing foreign layers is OK. I think we just don’t push foreign layers to remote registries (or export to a tar) by default; according to what you said, this seems like the default behavior. Then we could have a Jib config option to allow pushing foreign layers. We will need to generate different manifests in these two cases.

This does seems like a lot of work to do. Not sure if we will prioritize this, given that this is mostly an issue when trying to build Windows images. But if we can confirm at the current state that Windows images built by Jib more or less work, then I can imagine this will suddenly become a low-hanging fruit and get prioritized.

I will try to think some way to easily check this. Doesn’t seem easy though.

chanseokoh on Feb 12, 2020

So I guess that support for these fields in the parser could move us further.

I think we need to check a few things first. Now it seems possible that Docker entirely avoids downloading these “foreign” layers, probably for legal reasons. (That is, these layers don’t exist in the local Docker daemon cache.) If that’s the case, it’s possible that the tar archive from docker save lacks these foreign layers too (although it might attempt to download these on-the-fly when doing docker save). And on top of what @briandealwis said, it’s possible that Docker avoids pushing these foreign layers to remote registries. If that is the case, we should follow the same.

chanseokoh on Feb 12, 2020

It feels like we’re playing whackamole. What do these layer descriptors mean and how should Jib propagate them when pushing an image to a remote registry? What other changes were required in Docker and others?

briandealwis on Feb 12, 2020