kaniko: Kaniko builds fail in Cloud Build without `--additional-whitelist=/var/run`

Actual behavior Kaniko builds fail in Cloud Build, in standard configuration.

Expected behavior Kaniko builds work just fine, without the extra --additional-whitelist=/var/run flag.

To Reproduce Steps to reproduce the behavior:

  1. Configure a Cloud Build job in standard debug configuration:
steps:
  - name: gcr.io/kaniko-project/executor:debug
    args: ["--dockerfile=<path to Dockerfile within the build context>",
           "--context=dir://<path to build context>",
           "--destination=<gcr.io/$PROJECT/$IMAGE:$TAG>"]
  1. Observe that the build fails with
error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/docker.sock: device or resource busy
  1. Add --additional-whitelist=/var/run to the build step, and it succeeds.

Additional Information Any Dockerfile (unverified).

Triage Notes for the Maintainers

Initially reported in https://github.com/GoogleContainerTools/kaniko/issues/903#issuecomment-578362533

Description Yes/No
Please check if this a new feature you are proposing
  • - [ ]
Please check if the build works in docker but not in kaniko
  • - [x]
Please check if this error is seen when you use --cache flag
  • - [x]
Please check if your dockerfile is a multistage dockerfile
  • - [x]

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 30
  • Comments: 36 (14 by maintainers)

Commits related to this issue

Most upvoted comments

hey folks, PR in progress. https://github.com/GoogleContainerTools/kaniko/pull/1025

Patch fix coming soon.

+6
tejal29 on Feb 4, 2020

This is broken when using the Kaniko debug image in GitLab CI.

+5
stevehipwell on Feb 4, 2020

@tejal29 can confirm now it works with both v0.17.1 and debug-v0.17.1 without needing to add any additional flag (like it was before). Thanks!

+4
ejose19 on Feb 4, 2020

Just ran into this as well. Do you plan to roll out a fix soon, or should we roll back?

+4
CAFxX on Feb 4, 2020

To re-iterate, what we’d like to be able to see (ideally) is that Kaniko auto-detects Cloud Build environment, and automatically whitelists /var/run. If that’s not possible, at least there should be (ideally) an option that preserves the old behavior by default (i.e. no need to whitelist /var/run explicitly), and then for those users who don’t need to whitelist it (i.e. non-Cloud Build users), they will be able to --whitelist-var-run=false to disable this whitelisting explicitly…

Otherwise, the current behavior breaks all Cloud Build jobs that are using executor:latest or executor:debug images, by requiring DevOps teams to adjust every build manually (to add the new argument). In addition, the current behavior does not seem to work at all in Cloud Build, with or without the new option (and both for true or false), so the only workaround atm is to use a pinned version <=0.16.0 (which incidentally, also requires us to manually go and adjust all builds…)

+4
dinvlad on Feb 4, 2020

@dinvlad, @keylowgee, @artichaulo, @atikhono, @wintersolutions, The gcr.io/kaniko-project/executor:debug now points to gcr.io/kaniko-project/executor:debug-v0.16.0

tejaldesai@@kaniko (remove_debug_trigger)$ docker tag gcr.io/kaniko-project/executor:debug-v0.16.0 gcr.io/kaniko-project/executor:debug
tejaldesai@@kaniko (remove_debug_trigger)$ docker push gcr.io/kaniko-project/executor:debug

We will also add a fix to retain the default behavior which is whitelisting /var/run so that the current users are not affected.

+4
tejal29 on Jan 27, 2020

@tejal29

From a GitLab CI build:

DEBU[0000] Copying file /builds/base/ubuntu-base/Dockerfile to /kaniko/Dockerfile 
DEBU[0000] Skip resolving path /kaniko/Dockerfile       
DEBU[0000] Skip resolving path /builds/base/ubuntu-base 
DEBU[0000] Skip resolving path /cache                   
DEBU[0000] Skip resolving path                          
DEBU[0000] Skip resolving path                          
DEBU[0000] Skip resolving path                          
DEBU[0002] No file found for cache key sha256:bc025862c3e8ec4a8754ea4756e33da6c41cba38330d7e324abd25c8e0b93300 stat /cache/sha256:bc025862c3e8ec4a8754ea4756e33da6c41cba38330d7e324abd25c8e0b93300: no such file or directory 
DEBU[0002] Image ubuntu:18.04 not found in cache        
DEBU[0004] No file found for cache key sha256:bc025862c3e8ec4a8754ea4756e33da6c41cba38330d7e324abd25c8e0b93300 stat /cache/sha256:bc025862c3e8ec4a8754ea4756e33da6c41cba38330d7e324abd25c8e0b93300: no such file or directory 
DEBU[0004] Image ubuntu:18.04 not found in cache        
DEBU[0005] Mounted directories: [{/kaniko false} {/etc/mtab false} {/tmp/apt-key-gpghome true} {/proc false} {/dev false} {/dev/pts false} {/sys false} {/sys/fs/cgroup false} {/sys/fs/cgroup/systemd false} {/sys/fs/cgroup/net_prio,net_cls false} {/sys/fs/cgroup/cpuset false} {/sys/fs/cgroup/pids false} {/sys/fs/cgroup/blkio false} {/sys/fs/cgroup/memory false} {/sys/fs/cgroup/cpuacct,cpu false} {/sys/fs/cgroup/hugetlb false} {/sys/fs/cgroup/freezer false} {/sys/fs/cgroup/devices false} {/sys/fs/cgroup/perf_event false} {/dev/mqueue false} {/builds false} {/busybox false} {/dev/termination-log false} {/etc/resolv.conf false} {/etc/hostname false} {/etc/hosts false} {/dev/shm false} {/run/secrets false} {/var/run/secrets/kubernetes.io/serviceaccount false} {/proc/bus false} {/proc/fs false} {/proc/irq false} {/proc/sys false} {/proc/sysrq-trigger false} {/proc/acpi false} {/proc/kcore false} {/proc/keys false} {/proc/timer_list false} {/proc/timer_stats false} {/proc/sched_debug false} {/proc/scsi false} {/sys/firmware false}] 
DEBU[0006] Not adding /dev because it is whitelisted    
DEBU[0006] Not adding /etc/hostname because it is whitelisted 
DEBU[0006] Not adding /etc/hosts because it is whitelisted 
DEBU[0006] Not adding /etc/resolv.conf because it is whitelisted 
DEBU[0006] Not adding /proc because it is whitelisted   
DEBU[0006] Not adding /sys because it is whitelisted    
error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_02_04_18_36_43.168969777: read-only file system
+1
claudioweiler on Feb 4, 2020

@tejal29 here it is

Cloudbuild:

steps:
  - name: "gcr.io/kaniko-project/executor:debug"
    args:
      ["--dockerfile=Dockerfile", "--destination=<url>", "--verbosity=debug"]

Dockerfile:

FROM alpine
RUN apk add nano

Logs:

BUILD
Pulling image: gcr.io/kaniko-project/executor:debug
debug: Pulling from kaniko-project/executor
bfb70510d7c5: Pulling fs layer
dc2057c58a5b: Pulling fs layer
1fcacdcafaa9: Pulling fs layer
d06d96ef79d9: Pulling fs layer
a7090596b381: Pulling fs layer
e6f4337a185f: Pulling fs layer
4536006be0b7: Pulling fs layer
d06d96ef79d9: Waiting
a7090596b381: Waiting
e6f4337a185f: Waiting
4536006be0b7: Waiting
dc2057c58a5b: Verifying Checksum
dc2057c58a5b: Download complete
1fcacdcafaa9: Verifying Checksum
1fcacdcafaa9: Download complete
a7090596b381: Verifying Checksum
a7090596b381: Download complete
d06d96ef79d9: Verifying Checksum
d06d96ef79d9: Download complete
e6f4337a185f: Verifying Checksum
e6f4337a185f: Download complete
4536006be0b7: Verifying Checksum
4536006be0b7: Download complete
bfb70510d7c5: Verifying Checksum
bfb70510d7c5: Download complete
bfb70510d7c5: Pull complete
dc2057c58a5b: Pull complete
1fcacdcafaa9: Pull complete
d06d96ef79d9: Pull complete
a7090596b381: Pull complete
e6f4337a185f: Pull complete
4536006be0b7: Pull complete
Digest: sha256:53bf8a6d56fed34914676e8d930fd96c3969914d96082de08dc99bf31f09c636
Status: Downloaded newer image for gcr.io/kaniko-project/executor:debug
gcr.io/kaniko-project/executor:debug
DEBU[0000] Copying file /workspace/Dockerfile to /kaniko/Dockerfile 
DEBU[0000] Skip resolving path /kaniko/Dockerfile       
DEBU[0000] Skip resolving path /workspace/              
DEBU[0000] Skip resolving path /cache                   
DEBU[0000] Skip resolving path                          
DEBU[0000] Skip resolving path                          
DEBU[0000] Skip resolving path                          
INFO[0000] Resolved base name alpine to alpine          
INFO[0000] Resolved base name alpine to alpine          
INFO[0000] Retrieving image manifest alpine             
DEBU[0001] No file found for cache key sha256:ddba4d27a7ffc3f86dd6c2f92041af252a1f23a8e742c90e6e1297bfa1bc0c45 stat /cache/sha256:ddba4d27a7ffc3f86dd6c2f92041af252a1f23a8e742c90e6e1297bfa1bc0c45: no such file or directory 
DEBU[0001] Image alpine not found in cache              
INFO[0001] Retrieving image manifest alpine             
INFO[0002] Built cross stage deps: map[]                
INFO[0002] Retrieving image manifest alpine             
DEBU[0003] No file found for cache key sha256:ddba4d27a7ffc3f86dd6c2f92041af252a1f23a8e742c90e6e1297bfa1bc0c45 stat /cache/sha256:ddba4d27a7ffc3f86dd6c2f92041af252a1f23a8e742c90e6e1297bfa1bc0c45: no such file or directory 
DEBU[0003] Image alpine not found in cache              
INFO[0003] Retrieving image manifest alpine             
INFO[0004] Unpacking rootfs as cmd RUN apk add nano requires it. 
DEBU[0004] Mounted directories: [{/kaniko false} {/etc/mtab false} {/tmp/apt-key-gpghome true} {/proc false} {/dev false} {/dev/pts false} {/sys false} {/sys/fs/cgroup false} {/sys/fs/cgroup/systemd false} {/sys/fs/cgroup/hugetlb false} {/sys/fs/cgroup/rdma false} {/sys/fs/cgroup/net_cls,net_prio false} {/sys/fs/cgroup/blkio false} {/sys/fs/cgroup/memory false} {/sys/fs/cgroup/cpu,cpuacct false} {/sys/fs/cgroup/pids false} {/sys/fs/cgroup/devices false} {/sys/fs/cgroup/freezer false} {/sys/fs/cgroup/cpuset false} {/sys/fs/cgroup/perf_event false} {/dev/mqueue false} {/workspace false} {/busybox false} {/builder/home false} {/builder/outputs false} {/root/tokencache false} {/etc/resolv.conf false} {/etc/hostname false} {/etc/hosts false} {/dev/shm false} {/var/run/docker.sock false}] 
DEBU[0004] Not adding /dev because it is whitelisted    
DEBU[0004] Not adding /etc/hostname because it is whitelisted 
DEBU[0004] Not adding /etc/hosts because it is whitelisted 
DEBU[0004] Not adding /etc/mtab because it is whitelisted 
DEBU[0004] Not adding /proc because it is whitelisted   
DEBU[0004] Not adding /sys because it is whitelisted    
error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/docker.sock: device or resource busy
ERROR
ERROR: build step 0 "gcr.io/kaniko-project/executor:debug" failed: exit status 1
+1
ejose19 on Feb 4, 2020

Yes, and we also got that error in Cloud Build! I suspect it’s because CB has those images cached (because 0.17.0 was re-tagged). Maybe we could try the full sha256-based image tag…

Would it be possible to detect Cloud Build environment and set this option automatically instead, so that DevOps don’t have to apply this fix retroactively? Thanks

+1
dinvlad on Feb 3, 2020

Could you add a note that we should use it in Cloud Build by default from now on? I.e. the default Cloud Build config suggested in README.md should be

steps:
  - name: gcr.io/kaniko-project/executor:latest
    args: ["--dockerfile=<path to Dockerfile within the build context>",
           "--context=dir://<path to build context>",
           "--destination=<gcr.io/$PROJECT/$IMAGE:$TAG>",
           "--whitelist-var-run=false"]

Otherwise, it seems like this solution is not any different from where it started…

+1
dinvlad on Feb 3, 2020

Hey folks, our v0.17.0 release is out!

Please use --whitelist-var-run=false to include /var/run in your destination image.

+1
tejal29 on Feb 3, 2020

Now it might fail with --additional-whitelist due to #1006

Yeah, I could never get this flag to actually work. I guess I’ll revert to 0.15.0 while this gets resolved in some way.

+1
jdurzi on Jan 28, 2020
Read more comments on GitHub
← kaniko: Kaniko build fails on apt-get reporting "unable to fetch libraries"
kaniko: Kaniko builds sometimes produces broken image (doesn't happen on first build, only building w/ cache?) →