kaniko: when replacing a folder by symlink : i get corrupt images when pulling : failed to mknod("somefile", S_IFCHR, 0): no such file or directory
Actual behavior
When replacing a folder by symlink i get corrupt images , docker pull will fail with : Failed to register layer: Error processing tar file(exit status 1): failed to mknod(“/etc/fail2ban/action.d”, S_IFCHR, 0): no such file or directory
Expected behavior Be able to pull the image
To Reproduce
The error doesn’t seems t happen with every files/folder. I tried disabling all caches to be sure
The build/pull works when using the dockerfile directly with docker build command The build works with kaniko but appears corrupt as i can t pull it
Additional Information
- Dockerfile
FROM quay.io/centos/centos:stream9 RUN dnf -y update && dnf -y install systemd RUN /usr/bin/systemctl mask systemd-logind && /usr/bin/systemctl mask systemd-hostnamed && /usr/bin/systemctl mask dbus- RUN ln -s /data/git_conf/conf/systemd/journald.conf.d /etc/systemd/journald.conf.d RUN dnf install -y rsyslog && mv -f /etc/rsyslog.d /etc/rsyslog.d.orig && ln -s /data/git_conf/conf/rsyslog/rsyslog.d /etc/rsyslog.d RUN systemctl enable rsyslog RUN dnf -y install procps net-tools STOPSIGNAL SIGRTMIN+3 CMD [ “/sbin/init” ]
This one build and pull fine even with the the broken symlink that will lead to a mounted volume set in docker compose
FROM previousdockerimage RUN dnf -y update && dnf install -y epel-release && dnf -y install fail2ban && systemctl enable fail2ban && dnf clean all RUN rm -rf /etc/fail2ban && ln -s /data/git_conf/conf/fail2ban /etc/fail2ban STOPSIGNAL SIGRTMIN+3 EXPOSE 22 CMD [ “/sbin/init” ]
When pulling this one errors with Error processing tar file(exit status 1): failed to mknod(“/etc/fail2ban/action.d”, S_IFCHR, 0): no such file or directory The folder action.d isn’t supposed to be there anymore.
- Build Context
Nothing special, here is the gitlab ci/cd, but i get the same error running it manually
make_fail2ban: stage: make_fail2ban image: name: gcr.io/kaniko-project/executor:v1.9.1-debug entrypoint: [“”] script: - /kaniko/executor -c /workspace --dockerfile https://mygitlab/mydockerfile --destination “${CI_REGISTRY_IMAGE}/fail2ban:${CI_COMMIT_TAG}”
- Kaniko Image (fully qualified with digest)
gcr.io/kaniko-project/executor:v1.9.1-debug sha256:ac169723b2076f9d5804f4bc05c98397e286da6fdcdd5a09fdc179f06ccb3be1
Triage Notes for the Maintainers
| Description | Yes/No |
|---|---|
| Please check if this a new feature you are proposing |
|
| Please check if the build works in docker but not in kaniko |
|
Please check if this error is seen when you use --cache flag |
|
| Please check if your dockerfile is a multistage dockerfile |
|
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 22 (13 by maintainers)
Commits related to this issue
- Don't write whiteout files if parent path got replaced with link If a non-empty directory gets replaced with a link, the files in that directory also get deleted. However, there should not be any whi... — committed to andreasf/kaniko by andreasf a year ago
- Don't write whiteout files to directories that were replaced with files or links If a non-empty directory gets replaced with something other than a directory (e.g. file or symlink), the files in that... — committed to andreasf/kaniko by andreasf a year ago
- Don't write whiteout files to directories that were replaced with files or links If a non-empty directory gets replaced with something other than a directory (e.g. file or symlink), the files in that... — committed to andreasf/kaniko by andreasf a year ago
- Don't write whiteout files to directories that were replaced with files or links If a non-empty directory gets replaced with something other than a directory (e.g. file or symlink), the files in that... — committed to andreasf/kaniko by andreasf a year ago
- Don't write whiteout files to directories that were replaced with files or links (#2590) If a non-empty directory gets replaced with something other than a directory (e.g. file or symlink), the file... — committed to GoogleContainerTools/kaniko by andreasf a year ago
- Don't write whiteout files to directories that were replaced with files or links (#2590) If a non-empty directory gets replaced with something other than a directory (e.g. file or symlink), the file... — committed to coder/kaniko by andreasf a year ago
Oopsie. I think I know what the problem is and I’ll prepare a fix
Hmm not sure about, I refactored the strange whiteout files once to be clearer, there was lots of old stuff laying around, but I think that commit came afterwards? My refactor was about to track correctly deleted and new files for each layer and act on these two lists later in the process when the tar for the layer is created. #2066 I dont have time this week, maybe I can have a look at it next week.
Info: the line
RUN rm -Rfv /a && ln -sf /b /ashould create a whiteout (thats a deletion of a file) file and at the same time in the next command create a normal symlink a -> /b and I suspect these two things do not play nice with each other. I suspect that maybe there is no creation of a new fileadetected because of symlink following and the whiteout forais not discarded, resulting in a whiteout and a writing of the symlink into the tar because the tar mechanism is kind of different (???), something along these lines@andreasf Found some time to test your fixed image (
docker.io/andreasfleig/kaniko-executor:debug) with the Dockerfile which lead me here and it fixes the issue for me 🎉I specified a different registry for cache and it works with the cache as well. So i guess a bad layer was still in cache. (in gitlab registry deletion is not instant).
@andreasf : Sorry I was mistaken, everything works fine,
/asymlink and/bare added to the add list in the last layer, all seams correct.Minimal reproducer:
Dockerfile
Build with:
The resulting tar file cannot be loaded into podman/docker: