kaniko: Image built with Kaniko claims to be OCI but in reality is not
Actual behavior
Coming from https://github.com/containers/buildah/issues/3668
I am using kaniko to build an image based on an OCI-image.
The base image has the following manifest: ( notice mediaType: application/vnd.oci.image.layer.v1.tar+gzip )
>>> skopeo inspect --raw docker://${BASE_IMAGE} | jq .
{
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.oci.image.config.v1+json",
"digest": "sha256:61679cc9cfe1e3c757bfe2ff01222e25a4e0349ff70739f7c982df4e9484d5a4",
"size": 419
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:3bc51580eb8a78b645a88f9c89c0779be50944543b917c682af93035002c2d99",
"size": 79650768
}
]
}
If I use this image as a base for another image built with Kaniko, I get the following resulting image:
{
"schemaVersion": 2,
"mediaType": "",
"config": {
"mediaType": "application/vnd.oci.image.config.v1+json",
"size": 906,
"digest": "sha256:da250df73fc4c57e758739f562f7c5bf77703f0547951a254a6043719ccb35a6"
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"size": 79650768,
"digest": "sha256:3bc51580eb8a78b645a88f9c89c0779be50944543b917c682af93035002c2d99"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 222,
"digest": "sha256:8962e548f920af01274257418f3414570b5d0761524773a86df7835344e467f7"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 198,
"digest": "sha256:2b36e6ccfa17b538ef83c8aff96ffe823966883f90b4517b35346b57cc642c46"
}
]
}
Which claims to be application/vnd.oci.image.config.v1+json but indeed has Docker application/vnd.docker.image.rootfs.diff.tar.gzip layers.
This shows as an error when going to use the child image as a base image in podman build, which shows (again see issue https://github.com/containers/buildah/issues/3668):
Error: error creating build container: error preparing image configuration: error converting image
"containers-storage:[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@04ee292f7a5549e765c99205acc567738a09eb084409cd71f6600facd3743c51"
from "application/vnd.oci.image.manifest.v1+json" to "application/vnd.docker.distribution.manifest.v2+json":
Unknown media type during manifest conversion: "application/vnd.docker.image.rootfs.diff.tar.gzip"
Expected behavior
As @vrothberg suggests, the layers should be converted to OCI ones during build or when pushing to the registry.
To Reproduce
Steps to reproduce the behavior:
- Have a base image with
"mediaType": "application/vnd.oci.image.config.v1+json", - Use the base image to build another (multistage) image with Kaniko.
Additional Information
- Dockerfile
Unfortunately it is quite difficult to find a public image that has "mediaType": "application/vnd.oci.image.config.v1+json", but to build one the following can be achieved with podman:
>> cat Containerfile
FROM docker.io/alpine
RUN touch file.txt
RUN echo "hello world"
>> podman build -t base-image -f Containerfile .
>> podman push base-image path/to/remote/repo/base-image
Dockerfile for child image:
FROM ubuntu:20.04 as installer
ADD installer.sh .
RUN bash installer.sh
######################
FROM path/to/remote/repo/base-image
COPY --from=installer /opt/application /opt/application
RUN ln -s /opt/application/1.0.0 /opt/application/stable
CMD ["/bin/bash"]
- Build Context Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands)
>>> cat installer.sh
mkdir -p /opt/application/1.0.0
touch /opt/application/1.0.0/file.txt
touch /opt/application/1.0.0/file2.txt
touch /opt/application/1.0.0/file3.txt
- Kaniko Image (fully qualified with digest)
Triage Notes for the Maintainers
| Description | Yes/No |
|---|---|
| Please check if this a new feature you are proposing |
|
| Please check if the build works in docker but not in kaniko |
|
Please check if this error is seen when you use --cache flag |
|
| Please check if your dockerfile is a multistage dockerfile |
|
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 16
- Comments: 30 (7 by maintainers)
Commits related to this issue
- fix(executor): export snapshotted layers of OCI images as OCI mediaTypes quick and dirrty fix for #1836 ? — committed to kfix/kaniko by kfix a year ago
- fix(executor): export snapshotted layers of OCI images as OCI mediaTypes quick and dirrty fix for #1836 ? — committed to kfix/kaniko by kfix a year ago
- fix(executor): export snapshotted layers of OCI images as OCI mediaTypes quick and dirrty fix for #1836 ? — committed to kfix/kaniko by kfix a year ago
Still a problem, some progress on this issue would be greatly appreciated đ
@BronzeDeer Yes, Iâve wrapped up my house move and its very hot outside so maybe Iâll chill and attempt to write test code in Go
from my poking around it looks like gcrâs âtarball.LayerFromFileâ is being used to convert the snapshots taken at every RUN to the layers: https://github.com/GoogleContainerTools/kaniko/blob/61312a95ae1a305f5154f8d88dc58ee1e3259ae8/pkg/executor/build.go#L520
this method is âdeprecatedâ in that lib but apparently is still there. there was an abandoned PR to switch to its preferred replacement: https://github.com/GoogleContainerTools/kaniko/pull/449
anyways, it seems like whatâs needed here is for kaniko to pass in WithMediaType => types.OCILayer whenever its working FROM a known-OCI image, because the default is types.DockerLayer: https://github.com/google/go-containerregistry/blob/cd7761563a00fb38bb6d2126fc0f262fb6e64db1/pkg/v1/tarball/layer.go#L237
I just pinned all my Dockerfilesâ references to ubuntu:* images back to the -20221130 tags because thatâs the last one published in the docker format and still works properly with both Kaniko and RedHat tools (podman, buildah, quay, etc)
Is this getting any traction? Otherwise weâre going to have to move away from kaniko
As soon as https://github.com/GoogleContainerTools/container-diff/pull/390 is merged, Iâll change #2425 to use that commit and have green tests again, if you want to test in the meantime you can rebase on top of #2425, install the fixed container-diff and run the integration tests locally or in the CI of your fork (again, note that currently #2425 still loads the old broken container-diff, youâd need to amend that in hte make file until that fix is merged)
With the Ubuntu 22.04 images now in OCI format, this has become a pressing issue for us. I found a workaround: use
--no-pushand have Kaniko just write to a tar file (--tarPath). Thendocker image loadthat, and let Docker do the push; that seems to result in a valid manifest.Same approach appears to work with Podman.
Thanks for investigating! We have images built with Kaniko that can be run by Podman 3.3.1 on CentOS Stream 8 but not pushed to another registry. The error message is slightly different, however it shows it canât handle the format.
When trying to push: