kaniko: Image built with Kaniko claims to be OCI but in reality is not

Actual behavior

Coming from https://github.com/containers/buildah/issues/3668

I am using kaniko to build an image based on an OCI-image.

The base image has the following manifest: ( notice mediaType: application/vnd.oci.image.layer.v1.tar+gzip )

>>> skopeo inspect --raw docker://${BASE_IMAGE} | jq .

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:61679cc9cfe1e3c757bfe2ff01222e25a4e0349ff70739f7c982df4e9484d5a4",
    "size": 419
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:3bc51580eb8a78b645a88f9c89c0779be50944543b917c682af93035002c2d99",
      "size": 79650768
    }
  ]
}

If I use this image as a base for another image built with Kaniko, I get the following resulting image:

{
  "schemaVersion": 2,
  "mediaType": "",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "size": 906,
    "digest": "sha256:da250df73fc4c57e758739f562f7c5bf77703f0547951a254a6043719ccb35a6"
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 79650768,
      "digest": "sha256:3bc51580eb8a78b645a88f9c89c0779be50944543b917c682af93035002c2d99"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 222,
      "digest": "sha256:8962e548f920af01274257418f3414570b5d0761524773a86df7835344e467f7"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 198,
      "digest": "sha256:2b36e6ccfa17b538ef83c8aff96ffe823966883f90b4517b35346b57cc642c46"
    }
  ]
}

Which claims to be application/vnd.oci.image.config.v1+json but indeed has Docker application/vnd.docker.image.rootfs.diff.tar.gzip layers.

This shows as an error when going to use the child image as a base image in podman build, which shows (again see issue https://github.com/containers/buildah/issues/3668):

Error: error creating build container: error preparing image configuration: error converting image 
"containers-storage:[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@04ee292f7a5549e765c99205acc567738a09eb084409cd71f6600facd3743c51" 
from "application/vnd.oci.image.manifest.v1+json" to "application/vnd.docker.distribution.manifest.v2+json": 
Unknown media type during manifest conversion: "application/vnd.docker.image.rootfs.diff.tar.gzip"

Expected behavior

As @vrothberg suggests, the layers should be converted to OCI ones during build or when pushing to the registry.

To Reproduce

Steps to reproduce the behavior:

  • Have a base image with "mediaType": "application/vnd.oci.image.config.v1+json",
  • Use the base image to build another (multistage) image with Kaniko.

Additional Information

  • Dockerfile

Unfortunately it is quite difficult to find a public image that has "mediaType": "application/vnd.oci.image.config.v1+json", but to build one the following can be achieved with podman:

>> cat Containerfile
FROM docker.io/alpine
RUN touch file.txt
RUN echo "hello world"

>> podman build -t base-image -f Containerfile .
>> podman push base-image path/to/remote/repo/base-image

Dockerfile for child image:

FROM ubuntu:20.04 as installer
ADD installer.sh .
RUN bash installer.sh
######################
FROM path/to/remote/repo/base-image
COPY --from=installer /opt/application /opt/application
RUN ln -s /opt/application/1.0.0 /opt/application/stable
CMD ["/bin/bash"]
  • Build Context Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands)
>>> cat installer.sh
mkdir -p /opt/application/1.0.0
touch /opt/application/1.0.0/file.txt
touch /opt/application/1.0.0/file2.txt
touch /opt/application/1.0.0/file3.txt
  • Kaniko Image (fully qualified with digest)

Triage Notes for the Maintainers

Description Yes/No
Please check if this a new feature you are proposing
  • - [ ]
Please check if the build works in docker but not in kaniko
  • - [X]
Please check if this error is seen when you use --cache flag
  • - [ ]
Please check if your dockerfile is a multistage dockerfile
  • - [X]

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 16
  • Comments: 30 (7 by maintainers)

Commits related to this issue

Most upvoted comments

Still a problem, some progress on this issue would be greatly appreciated 😃

@BronzeDeer Yes, I’ve wrapped up my house move and its very hot outside so maybe I’ll chill and attempt to write test code in Go

from my poking around it looks like gcr’s “tarball.LayerFromFile” is being used to convert the snapshots taken at every RUN to the layers: https://github.com/GoogleContainerTools/kaniko/blob/61312a95ae1a305f5154f8d88dc58ee1e3259ae8/pkg/executor/build.go#L520

this method is “deprecated” in that lib but apparently is still there. there was an abandoned PR to switch to its preferred replacement: https://github.com/GoogleContainerTools/kaniko/pull/449

anyways, it seems like what’s needed here is for kaniko to pass in WithMediaType => types.OCILayer whenever its working FROM a known-OCI image, because the default is types.DockerLayer: https://github.com/google/go-containerregistry/blob/cd7761563a00fb38bb6d2126fc0f262fb6e64db1/pkg/v1/tarball/layer.go#L237

I just pinned all my Dockerfiles’ references to ubuntu:* images back to the -20221130 tags because that’s the last one published in the docker format and still works properly with both Kaniko and RedHat tools (podman, buildah, quay, etc)

Is this getting any traction? Otherwise we’re going to have to move away from kaniko

I say “tests running” because they appear to be a bit busted. for OCI specifically by GoogleContainerTools/container-diff#390, fixed as a part of #2425 I’m hoping.

I think I’m going to have to wait on that stuff before attempting to PR this.

As soon as https://github.com/GoogleContainerTools/container-diff/pull/390 is merged, I’ll change #2425 to use that commit and have green tests again, if you want to test in the meantime you can rebase on top of #2425, install the fixed container-diff and run the integration tests locally or in the CI of your fork (again, note that currently #2425 still loads the old broken container-diff, you’d need to amend that in hte make file until that fix is merged)

With the Ubuntu 22.04 images now in OCI format, this has become a pressing issue for us. I found a workaround: use --no-push and have Kaniko just write to a tar file (--tarPath). Then docker image load that, and let Docker do the push; that seems to result in a valid manifest.

Same approach appears to work with Podman.

Thanks for investigating! We have images built with Kaniko that can be run by Podman 3.3.1 on CentOS Stream 8 but not pushed to another registry. The error message is slightly different, however it shows it can’t handle the format.

$ skopeo inspect --raw docker://<kaniko built image> | jq .
{
  "schemaVersion": 2,
  "mediaType": "",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "size": 2229,
    "digest": "sha256:1d9384a1e8cf5636c4c525b02d3eb2c5e2a6300987717cf00b798db54aabd955"
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 79848180,
      "digest": "sha256:78846bc60c09f099fe07532fd402aff11d9704b96310c7bcf0a7ee20085774a1"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 334659526,
      "digest": "sha256:e8430b5e4f32476399aedbebd2bce8e31406d9f53e61bce5149b0f77fc3e11df"
    },
...

When trying to push:

$ podman push <kaniko built image>
Getting image source signatures
Copying blob 59a6252e61ff [--------------------------------------] 0.0b / 0.0b
Error: creating an updated image manifest: preparing updated manifest,
layer "sha256:b208550f9bbbc279a3ea5d174d86f9aadd31da9ae83f3496e3e92d2ea864cef6":
unsupported MIME type for compression: application/vnd.docker.image.rootfs.diff.tar.gzip