kaniko: error building image: error building stage GCC

With kaniko 0.24.0 this problem still persists.

First run without cache is successful, but on the second run, with the cached layer, it fails:

INFO[0013] Found cached layer, extracting to filesystem 
error building image: error building stage: failed to execute command: extracting fs from image: error removing /usr/local/lib/python2.7/site-packages/.wh.setuptools-41.4.0.dist-info to make way for new file.: fstatat /usr/local/lib/python2.7/site-packages/.wh.setuptools-41.4.0.dist-info: operation not permitted

Is there a workaround?

I’ve changed the docker storage driver from the default aufs to overlay2 as mentioned in this comment.

As a note if you use a gitlab-runner with docker executor: I needed to change it within /etc/docker/daemon.json for the whole docker process. The configuration in /etc/gitlab-runner/config.toml using environment= ["DOCKER_DRIVER=overlay2"] or within the CI was not sufficient (as suggested in the gitlab documentation)

This has fixed my issues I faced currently.

I’ve experienced the same issue while using warmer. It occurs when the executor is creating a .tag.gz file when running under Docker.

error building image: error building stage: failed to execute command: extracting fs from image: unexpected EOF

Edit: I’m using the latest debug image.

We are also on gitlab/gitlab-runner + k8s.

I can reproduce this ~sometimes~ always (if cache is present) with this simple Dockerfile:

FROM nginx:1.17

RUN echo "server {listen 80; root /html/; \
          location /api/ {return 500;} \
          location /static/ {} \
          location / {try_files \$uri /index.html;}}" > /etc/nginx/conf.d/default.conf

COPY ./build /html

with this command:

/kaniko/executor
      --context "./"
      --dockerfile Dockerfile
      --destination "$REGISTRY/$GROUP/$PROJECT/frontend:$TAG" # maybe this is important?
      --cache=true

inside the image “gcr.io/kaniko-project/executor:debug” (sha256:025bd79d3e0699b5f59142b03f7e66916980bd0e32653b9c7e21b561d4e538c3) and also “gcr.io/kaniko-project/executor:debug-v0.16.0”

When the cache is cleared, the build works fine.