kaniko: error building image: error building stage: lstat /tmp/apt-key-gpghome.VoPBz66R2g/gnupg_spawn_agent_sentinel.lock: no such file or directory

Actual behavior I am trying to build a Dockerfile which is built correctly with docker daemon. I have this error :

Setting up google-chrome-stable (77.0.3865.75-1) ...
update-alternatives: using /usr/bin/google-chrome-stable to provide /usr/bin/x-www-browser (x-www-browser) in auto mode
update-alternatives: using /usr/bin/google-chrome-stable to provide /usr/bin/gnome-www-browser (gnome-www-browser) in auto mode
update-alternatives: using /usr/bin/google-chrome-stable to provide /usr/bin/google-chrome (google-chrome) in auto mode
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Processing triggers for libgdk-pixbuf2.0-0:amd64 (2.36.5-2+deb9u2) ...
INFO[0278] Taking snapshot of full filesystem...        
INFO[0279] Adding whiteout for /var/lib/apt/lists       
error building image: error building stage: lstat /tmp/apt-key-gpghome.7n7iliD9iR/gnupg_spawn_agent_sentinel.lock: no such file or directory

Expected behavior As it works with docker daemon, I expect that the build is also OK with kaniko without changing the dockerfile.

To Reproduce Use this Dockerfile in a builder-node directory

FROM google/cloud-sdk:262.0.0-slim

RUN apt-get update && apt-get install --yes curl && \
    curl -sL https://deb.nodesource.com/setup_10.x | bash - && \
    apt-get install -y nodejs
 
RUN echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/chrome.list && \
    curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg |  apt-key add - && \
    echo "deb https://dl.yarnpkg.com/debian/ stable main" |  tee /etc/apt/sources.list.d/yarn.list && \
    apt-get update && \
    apt-get install --no-install-recommends -y software-properties-common && \
    # installation des paquets via apt
    apt-get install --no-install-recommends -y --allow-unauthenticated unzip google-chrome-stable yarn build-essential && \
    apt-get clean && \
    rm -rf /etc/apt/sources.list.d/chrome.list /var/lib/apt/lists

Run it :

run_in_docker.sh Dockerfile $(pwd)/builder-node gcr.io/myproject/builder-node:kaniko false

Additional Information

  • no other file in build context than the Dockerfile
ls -l builder-node 
total 4
-rw-r--r-- 1 olivier olivier 2354 15 sept. 23:26 Dockerfile
  • Kaniko Image built with make image from commit d67a8225031f2809f0d6c8c4f1825f233d8574a1 :
gcr.io/kaniko-project/executor                  latest              af14b1090d2e        2 hours ago         49.1MB

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Reactions: 4
  • Comments: 49 (14 by maintainers)

Most upvoted comments

I’m experiencing a similar issue with a gradle container:

INFO[0091] Taking snapshot of full filesystem...        
error building image: error building stage: Failed to get file info for /root/.kotlin/daemon/kotlin-daemon.2019-10-29T08-34-55.719Z.7338a1b69672dd00d0aa900c1e9f04a7.17450.run: lstat /root/.kotlin/daemon/kotlin-daemon.2019-10-29T08-34-55.719Z.7338a1b69672dd00d0aa900c1e9f04a7.17450.run: no such file or directory

When I enabled debug logging the issue disappeared. I suspect it might be a race condition but I have no experience to further investigate.

+13
nielsvanvelzen on Oct 29, 2019

@nielsvanvelzen did you ever manage to solve the kotlin daemon issue? I keep having that same error undeterministically in random commits that don’t change Cloudbuild, Kaniko, Gradle or Docker configurations at all…

A possible fix may be to disable Kotlin daemon with ENV GRADLE_OPTS -Dkotlin.compiler.execution.strategy="in-process" (in Dockerfile) but I can’t yet say as the issue might still arise again later… it is undeterministic 😒

+12
elnygren on Mar 8, 2020

Looks like the whole issue is just about kaniko design problems. I can’t realise that the issue is closed while it still active and described problem still reproducing.

Build tool that requires all the filesystem outside the build context to be consistent looks awful for me. Every time this tool will meets race conditions, because the reality is different. And this is looks like a real design problem for kaniko

Probably the better alternative is already existing, i hope you will find something for you https://blog.alexellis.io/building-containers-without-docker/

+9
nezed on Nov 19, 2020

Faced the similar issue with v0.24.0

COPY . .

RUN gradle :applications:backoffice:bootJar --no-daemon

BUILD SUCCESSFUL in 3m 43s
44 actionable tasks: 44 executed
INFO[0282] Taking snapshot of full filesystem...
error building image: error building stage: failed to take snapshot: Failed to get file info for /root/.kotlin/daemon/kotlin-daemon.2020-07-14T05-44-03.523Z.5f6154794ffdd6c37d85778697d6da61.17180.run: lstat /root/.kotlin/daemon/kotlin-daemon.2020-07-14T05-44-03.523Z.5f6154794ffdd6c37d85778697d6da61.17180.run: no such file or directory
+6
aca on Jul 14, 2020

To fix both errors reported by @caseycs and @nielsvanvelzen

error building image: error building stage: lstat /tmp/hsperfdata_root/6383: no such file or directory
error building image: error building stage: Failed to get file info for /root/.kotlin/daemon/kotlin-daemon.2019-10-29T08-34-55.719Z.7338a1b69672dd00d0aa900c1e9f04a7.17450.run: lstat /root/.kotlin/daemon/kotlin-daemon.2019-10-29T08-34-55.719Z.7338a1b69672dd00d0aa900c1e9f04a7.17450.run: no such file or directory

that are common for many java/kotlin projects, just add both following env variables to your Dockerfile:

ENV GRADLE_OPTS -Dkotlin.compiler.execution.strategy="in-process"
ENV JAVA_OPTS -XX:-UsePerfData
+4
nezed on Nov 20, 2020

i will take a look at this tomorrow.

+4
tejal29 on Nov 14, 2019

Getting this error on executor:debug on GitLab CI:

INFO[0020] Taking snapshot of full filesystem...        
INFO[0027] RUN yum clean all                            
INFO[0027] cmd: /bin/sh                                 
INFO[0027] args: [-c yum clean all]                     
INFO[0027] Running: [/bin/sh -c yum clean all]          
Loaded plugins: ovl, priorities
Cleaning repos: amzn2-core
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
INFO[0027] Taking snapshot of full filesystem...        
error building image: error building stage: failed to get files used from context: failed to get fileinfo for /workspace/requirements.txt: lstat /workspace/requirements.txt: no such file or directory

Dockerfile to reproduce:

FROM amazonlinux:latest

# Installing Python3
RUN yum install -y python3

# Clean yum cache
RUN yum clean all

# copy requirements.txt to tmp folder
ADD requirements.txt /tmp

# copy script to system
ADD script.py /usr/bin/

# install script's dependencies
RUN pip3 install -r /tmp/requirements.txt -q
+3
beanaroo on Oct 16, 2020

Getting same kind of error here:

update-alternatives: using /usr/bin/google-chrome-stable to provide /usr/bin/x-www-browser (x-www-browser) in auto mode
update-alternatives: using /usr/bin/google-chrome-stable to provide /usr/bin/gnome-www-browser (gnome-www-browser) in auto mode
update-alternatives: using /usr/bin/google-chrome-stable to provide /usr/bin/google-chrome (google-chrome) in auto mode
Processing triggers for libc-bin (2.24-11+deb9u4) ...
INFO[0254] Taking snapshot of full filesystem...        
error building image: error building stage: lstat /tmp/apt-key-gpghome.4lNiMJ5oLl/pubring.kbx: no such file or directory

Any solution?

+3
fabn on Sep 26, 2019

@EppO and @drshrey i pushed an image without the debug_. Thanks!

docker tag gcr.io/kaniko-project/executor:debug_769 gcr.io/kaniko-project/debug:769
docker push gcr.io/kaniko-project/debug:769
+2
tejal29 on Jan 30, 2020

Exactly the same thing while building huge (3gb) container with gradle&android sdk:

INFO[0454] Taking snapshot of full filesystem...        
error building image: error building stage: lstat /tmp/hsperfdata_root/6383: no such file or directory
+2
caseycs on Nov 27, 2019

Issue closed, but the bug is still there.

INFO[2023-12-06T03:43:03Z] Taking snapshot of full filesystem...        
error building image: error building stage: failed to take snapshot: Failed to get file info for /tmp/ssh-XXXXXXHNLnGp: lstat /tmp/ssh-XXXXXXHNLnGp: no such file or directory

It is very intermittent. Sometimes just one retry is enough, and the next execution is successful. Today I had a case where we retried multiple times, and several hours later it eventually started working again. The mitigation factors that we applied were removed, but it kept succeeding, which shows we did not fix it, it just fixed itself after some hours.

+1
jrkessl on Dec 6, 2023

Still encountering this fairly often with Kotlin. If kaniko fails to snapshot a file because it doesn’t exist, it should just ignore this, as the file no longer exists, so no longer needs snapshotting… That said, my intuition would be that the issue is caused by kaniko not waiting for all processes in the RUN to be fully stopped. It should probably ensure these are properly signalled to stop then waited upon. A quick look at the source code would suggest to me that the gradle daemon, kotlin daemon, etc are changing their process group, meaning that https://github.com/GoogleContainerTools/kaniko/blob/main/pkg/commands/run.go#L124C23-L135 doesn’t kill them. Update, yep, the gradle daemon changes its pgid: 26006 (java) S 4275 26006 26006 0 -1 4194304 126429 0 13 0 2378 72 0 0 20 0 53 0 2861752 5986787328 120556 18446744073709551615 94751120973824 94751120974905 140721562476352 0 0 0 4 0 16800975 0 0 0 17 2 0 0 0 0 0 94751120985408 94751120986128 94751136600064 140721562480465 140721562481353 140721562481353 140721562484665 0 (the gradle daemon /proc/pid/stat) shows that the PGID is 26006, the same as the PID. So to fix this issue, kaniko should kill all children of the launched process, not just those who remain in the PG. I don’t know how to accomplish that, though.

+1
penn5 on Jun 21, 2023 
args:
          - "--dockerfile=Dockerfile"
          - "--context=dir://./"

Worked for me

+1
xi2817-aajgaonkar on Dec 10, 2021

Hey folks, Release v0.17.0 is now up! Please use the latest image and let us know if you still see this issue!

Thank you for your patience!

+1
tejal29 on Feb 3, 2020

Great! This fix will available in next release in about an hour!

+1
tejal29 on Jan 31, 2020

Hey @olivierboudet, @jandillmann, @HerrmannHinz , @drshrey, @EppO, @Neonox31, @fabn , @nielsdenissen Sorry this took a lot of time. I have a fix for this and verified your dockerfile on my branch. https://github.com/GoogleContainerTools/kaniko/pull/1000

Thanks a again! Please let me know if you want an image to verify this works on your side.

+1
tejal29 on Jan 25, 2020

Same error here, I’m trying to build a danlynn/ember-cli image on Gitlab CI…

+1
jandillmann on Dec 19, 2019

any progress on this one? can’t use kaniko atm for building images from cicd. using kaniko:debug latest image from gcr.io

+1
HerrmannHinz on Dec 16, 2019

I can confirm it’s about the chrome install. When I just remove Chrome (google-chrome-unstable in my case) from the list of packages installed by apt-get, kaniko is able to build the image

+1
EppO on Oct 15, 2019

@Neonox31 that was my first thought but same dockerfile builds in native docker with no issues

+1
fabn on Oct 7, 2019

Thanks @fabn, i will take a look at this.

+1
tejal29 on Oct 4, 2019
Read more comments on GitHub
← kaniko: error building image: error building stage: gzip: invalid header
kaniko: error checking push permissions when pushing to insecure registry →