kaniko: Layer fails to extract when creating multistage image

@dsalaza4 I’ve been facing the same issue, but here’s how i worked around it.

For my use-case, i’m kind of doing the same thing, but not with GCR, but rather with AWS ECR.

I noticed that this usually happens with the config.json file where kaniko somehow saves this file with a long recursive filename (which i think is probably a bug, but not so sure). To avoid that, i tried copying the config.json file separately from the base /kaniko image and other relevant files separately instead of copying the whole /kaniko folder (which i think causes this issue) so that these files are not affected when executor builds and make changes to the filesystem.

In my case (with ECR), i’m doing this with my custom config.json. For you, i guess something like this would work as well (you just need to use google api credentials for your use-case):

# This image builds alpine with bash, git and kaniko executor

FROM gcr.io/kaniko-project/executor:debug as kaniko

# Do this if you have a custom config.json to auth with docker registry
COPY config.json /kaniko/.docker/


FROM alpine:latest

ENV DOCKER_CONFIG /kaniko/.docker/

RUN apk update && \
  apk upgrade && \
  apk add --no-cache \
    bash \
    git

# Copying complete /kaniko folder generates long filenames which 
# fails in pulling the docker image after it's built. So [WARNING] don't do this.
# COPY --from=kaniko /kaniko /kaniko

# Copy relevant files from /kaniko separately instead
COPY --from=kaniko /kaniko/executor /kaniko/
COPY --from=kaniko /kaniko/docker-credential-ecr-login /kaniko/
COPY --from=kaniko /kaniko/.docker/config.json /kaniko/.docker/
ENV PATH=/kaniko:$PATH

Build and run this image to check if this works:

>> docker build -t test .
>> docker run -it -v /path/to/workspace:/workspace --entrypoint="" test /bin/sh 
/workspace # executor --dockerfile=./Dockerfile --context=dir://`pwd` --force --no-push

INFO[0000] Resolved base name gcr.io/kaniko-project/executor:debug to gcr.io/kaniko-project/executor:debug
INFO[0000] Resolved base name alpine:latest to alpine:latest
INFO[0000] Resolved base name gcr.io/kaniko-project/executor:debug to gcr.io/kaniko-project/executor:debug
INFO[0000] Resolved base name alpine:latest to alpine:latest
INFO[0000] Downloading base image gcr.io/kaniko-project/executor:debug
INFO[0001] Error while retrieving image from cache: getting file info: stat /cache/sha256:a54d167d7c4b7ce0c7a622f17dcf473c652b29341b321ca507425c8fa3525842: no such file or directory
INFO[0001] Downloading base image gcr.io/kaniko-project/executor:debug
INFO[0002] Downloading base image alpine:latest
INFO[0004] Error while retrieving image from cache: getting file info: stat /cache/sha256:57334c50959f26ce1ee025d08f136c2292c128f84e7b229d1b0da5dac89e9866: no such file or directory
INFO[0004] Downloading base image alpine:latest
INFO[0005] Built cross stage deps: map[0:[/kaniko/executor /kaniko/docker-credential-ecr-login /kaniko/.docker/config.json]]
INFO[0005] Downloading base image gcr.io/kaniko-project/executor:debug
INFO[0006] Error while retrieving image from cache: getting file info: stat /cache/sha256:a54d167d7c4b7ce0c7a622f17dcf473c652b29341b321ca507425c8fa3525842: no such file or directory
INFO[0006] Downloading base image gcr.io/kaniko-project/executor:debug
INFO[0014] Taking snapshot of full filesystem...
INFO[0015] Using files from context: [/workspace/config.json]
INFO[0015] COPY config.json /kaniko/.docker/
INFO[0015] Taking snapshot of files...
INFO[0015] Saving file /kaniko/executor for later use.
INFO[0015] Saving file /kaniko/docker-credential-ecr-login for later use.
INFO[0015] Saving file /kaniko/.docker/config.json for later use.
INFO[0015] Deleting filesystem...
INFO[0016] Downloading base image alpine:latest
INFO[0017] Error while retrieving image from cache: getting file info: stat /cache/sha256:57334c50959f26ce1ee025d08f136c2292c128f84e7b229d1b0da5dac89e9866: no such file or directory
INFO[0017] Downloading base image alpine:latest
INFO[0018] Unpacking rootfs as cmd RUN apk update &&   apk upgrade &&   apk add --no-cache     bash     git requires it.
INFO[0018] Taking snapshot of full filesystem...
INFO[0019] ENV DOCKER_CONFIG /kaniko/.docker/
INFO[0019] RUN apk update &&   apk upgrade &&   apk add --no-cache     bash     git
INFO[0019] cmd: /bin/sh
INFO[0019] args: [-c apk update &&   apk upgrade &&   apk add --no-cache     bash     git]
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
v3.10.1-96-g031621e2cf [http://dl-cdn.alpinelinux.org/alpine/v3.10/main]
v3.10.1-99-gcf78a82040 [http://dl-cdn.alpinelinux.org/alpine/v3.10/community]
OK: 10337 distinct packages available
(1/2) Upgrading musl (1.1.22-r2 -> 1.1.22-r3)
(2/2) Upgrading musl-utils (1.1.22-r2 -> 1.1.22-r3)
Executing busybox-1.30.1-r2.trigger
OK: 6 MiB in 14 packages
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/11) Installing ncurses-terminfo-base (6.1_p20190518-r0)
(2/11) Installing ncurses-terminfo (6.1_p20190518-r0)
(3/11) Installing ncurses-libs (6.1_p20190518-r0)
(4/11) Installing readline (8.0.0-r0)
(5/11) Installing bash (5.0.0-r0)
Executing bash-5.0.0-r0.post-install
(6/11) Installing ca-certificates (20190108-r0)
(7/11) Installing nghttp2-libs (1.38.0-r0)
(8/11) Installing libcurl (7.65.1-r0)
(9/11) Installing expat (2.2.7-r0)
(10/11) Installing pcre2 (10.33-r0)
(11/11) Installing git (2.22.0-r0)
Executing busybox-1.30.1-r2.trigger
Executing ca-certificates-20190108-r0.trigger
OK: 30 MiB in 25 packages
INFO[0023] Taking snapshot of full filesystem...
INFO[0025] COPY --from=kaniko /kaniko/executor /kaniko/
INFO[0025] Taking snapshot of files...
INFO[0026] COPY --from=kaniko /kaniko/docker-credential-ecr-login /kaniko/
INFO[0026] Taking snapshot of files...
INFO[0026] COPY --from=kaniko /kaniko/.docker/config.json /kaniko/.docker/
INFO[0026] Taking snapshot of files...
INFO[0026] ENV PATH=/kaniko:$PATH
INFO[0026] WORKDIR /workspace
INFO[0026] cmd: workdir
INFO[0026] Changed working directory to /workspace
INFO[0026] Skipping push to container registry due to --no-push flag

/workspace # find / -name config.json

/workspace/config.json
/kaniko/.docker/config.json
/kaniko/0/kaniko/.docker/config.json

However if you copy the entire folder of /kaniko during build, the find command would reveal the following results.

/workspace # find / -name config.json

/workspace/config.json
/kaniko/.docker/config.json
/kaniko/0/kaniko/.docker/config.json
/kaniko/0/kaniko/0/kaniko/.docker/config.json
/kaniko/0/kaniko/0/kaniko/0/kaniko/.docker/config.json
/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/.docker/config.json
/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/.docker/config.json
/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/.docker/config.json
/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/.docker/config.json
/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/.docker/config.json
/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/.docker/config.json
/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/.docker/config.json
/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/.docker/config.json
/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/.docker/config.json
/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/.docker/config.json
...
...
...

You can read this more to see what more you need to copy w.r.t GCR and Google APIs.

Hopefully it works for you as well since it works for me.

As a note; using kaniko in a non-official image or copying the binaries to a non-official image is not supported. There may be a way to make it work, but YMMV

The problem is that the official kaniko images are FROM scratch, so if you want to use e.g. Git to extract useful building/tagging info from revision control, you’re SOL. kaniko:debug just adds Busybox, and at least in the case of Git there are no official static binaries made available that could easily be added to the kaniko image.

TBH it would be much better if there were e.g. a kaniko:alpine image available.

Okay, let’s step back a little and try to figure out what we were trying to solve in the second place. i.e. your credentials were being saved and visible in the resulting docker image. Prior to that, with the solution i proposed before, it worked and you were able to build a docker image.

From what i understand, although you don’t require the docker credentials at build time, you still require those credentials at runtime i.e. in the live kaniko executor container from the resulting image i.e. when you run executor to build and push images. The good thing is, you don’t need to save the credentials to the image but you can inject it in a running executor container with the echo blah.

The problem with this approach is that, the config where the docker credentials were stored got wiped out when the executor deleted the filesystem on building the image. I said that it had a default whitelist directory that prevents these directories from being deleted, however i was wrong. By default it doesn’t whitelist the /home/jenkins/agent directory, but you can instruct kaniko to whitelist it by setting a VOLUME directive in the Dockerfile as in (let’s go with /build now instead of /home/jenkins/agent for simplicity):

VOLUME /build

So can you try with the following Dockerfile:

FROM gcr.io/kaniko-project/executor:v0.10.0 as kaniko

FROM alpine:latest

RUN apk update && \
  apk upgrade && \
  apk add --no-cache \
    bash \
    git

# Should be put after above installations
COPY --from=kaniko /kaniko/executor /kaniko/
ENV PATH=/kaniko:$PATH

# Path to docker credentials
ENV DOCKER_CONFIG /build/.docker

# Do this to whitelist directory, store your docker credentials here
VOLUME /build

# Optional
WORKDIR /build

Then you can echo your credentials into the /build/.docker/config.json file.

The way i’m doing it doing my builds is the following:

>> docker build -f Dockerfile.test -t my-registry/my-repo:pr179 .

>> docker push my-registry/my-repo:pr179 .

Now i use my-registry/my-repo:pr179 inside my CI pipeline to build images with kaniko going forward. This can also be used to build the same image recursively with kaniko. So think of this image as the seed image built by docker instead of kaniko and then from here on, all images would be built inside this container with kaniko executor as in the following:

>> mkdir -p /build/.docker

>> echo "{\"auths\":{\"${DOCKER_HUB_URL}\":\
  {\"username\":\"${DOCKER_HUB_USER}\",\
  \"password\":\"${DOCKER_HUB_PASS}\"}}}" > "/build/.docker/config.json"

>> /kaniko/executor \
  --cleanup \
  --context "dockerfiles/public/$1/" \
  --dockerfile "dockerfiles/public/$1/Dockerfile" \
  --destination "fluidattacks/$1:$2" \
  --snapshotMode time

The reason it worked for me before was because i’ve been using the jenkins/jnlp-slave:alpine base image instead of the pure alpine image and that already had the VOLUME directive set to /home/jenkins/agent which made kaniko automatically whitelist the directory. So in this case, the /build directory should now be whitelisted and your credentials shouldn’t be wiped out when building and pushing the image.

I’ve tried it on my end, hopefully it works for you too.

I’m reopening this issue as I just found out that although copying the executor and other needed files fixes the initial problem, it seems like we’re still having some serious issues when it comes to what we’re actually copying.

Let me explain in more detail.

When executing:

COPY --from=kaniko /kaniko/executor /kaniko/
COPY --from=kaniko /kaniko/.docker/config.json /kaniko/.docker/

one might expect that such files would come from the specified stage:

FROM gcr.io/kaniko-project/executor:debug as kaniko

That is not the case with Kaniko. If the container you’re using for building the image is also gcr.io/kaniko-project/executor:debug, what happens is that the files get copied over from the container that is building the image, instead of the stage included in the Dockerfile. I know it’s a little confusing, I will show the particular example that made me realize of this.

Right after I built the container with kaniko, I went in to check the files within the /kaniko folder. There, I found out that my config.json file looked like this:

{"auths":{"index.docker.io":  {"username":"MY_USER",  "password":"MY_PASS"}}}

where both MY_USER and MY_PASS were visible. This was due to the fact that when building the image, in order to be able to push it to the registry, I logged in first:

echo "{\"auths\":{\"${DOCKER_HUB_URL}\":\
  {\"username\":\"${DOCKER_HUB_USER}\",\
  \"password\":\"${DOCKER_HUB_PASS}\"}}}" > /kaniko/.docker/config.json

/kaniko/executor \
  --cleanup \
  --context "dockerfiles/public/$1/" \
  --dockerfile "dockerfiles/public/$1/Dockerfile" \
  --destination "fluidattacks/$1:$2" \
  --snapshotMode time

I tried in many ways to reference the files from the stage instead of the files from the building container with no success.

I also tried building the image with docker, and voila, it worked as expected. My config.json looked like this (the default kaniko config.json):

{
        "auths": {},
        "credHelpers": {
                "asia.gcr.io": "gcr",
                "eu.gcr.io": "gcr",
                "gcr.io": "gcr",
                "staging-k8s.gcr.io": "gcr",
                "us.gcr.io": "gcr"
        }
}

What a bummer 😞