kaniko: Missing symlink in the images built using Kaniko

Actual behavior

I am attempting to migrate the image build process for our micro services from docker to kaniko and have realized that the kaniko built images fails to create files/symlinks generated by the OS packages.

Expected behavior

The image built from a Dockerfile using kaniko should the same as the ones built using Docker, it must include all files including symlinks.

To Reproduce Steps to reproduce the behavior:

Use the following Dockerfile and build an image using kaniko and another one using docker, push both images to repo.

FROM python:3.10.2-slim

RUN apt-get update && apt-get install -y \
  libglib2.0-0 \
  libgl1-mesa-glx \
  && rm -rf /var/lib/apt/lists/*

Try running the images to experience the difference. The package libxdmcp6:amd64 generates a symlink /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 pointing to /usr/lib/x86_64-linux-gnu/libXdmcp.so.6.0.0 which is missing in the image built using Kaniko and it does exist in the image built using Docker as shown below.

Kaniko

docker run --rm -it prasadzone/pythonapp:kaniko bash
root@3f1505cd4cee:/# dpkg -L libxdmcp6:amd64
/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libXdmcp.so.6.0.0
/usr/share
/usr/share/doc
/usr/share/doc/libxdmcp6
/usr/share/doc/libxdmcp6/changelog.Debian.gz
/usr/share/doc/libxdmcp6/changelog.gz
/usr/share/doc/libxdmcp6/copyright
/usr/lib/x86_64-linux-gnu/libXdmcp.so.6
root@3f1505cd4cee:/# ls -la /usr/lib/x86_64-linux-gnu | grep libXdmcp
-rw-r--r--  1 root root    22728 Mar  2  2017 libXdmcp.so.6.0.0
root@3f1505cd4cee:/#

Docker

$ docker run --rm -it prasadzone/pythonapp:docker bash
root@864bf398734b:/# dpkg -L libxdmcp6:amd64
/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libXdmcp.so.6.0.0
/usr/share
/usr/share/doc
/usr/share/doc/libxdmcp6
/usr/share/doc/libxdmcp6/changelog.Debian.gz
/usr/share/doc/libxdmcp6/changelog.gz
/usr/share/doc/libxdmcp6/copyright
/usr/lib/x86_64-linux-gnu/libXdmcp.so.6
root@864bf398734b:/# ls -la /usr/lib/x86_64-linux-gnu | grep libXdmcp
lrwxrwxrwx  1 root root       17 Mar  2  2017 libXdmcp.so.6 -> libXdmcp.so.6.0.0
-rw-r--r--  1 root root    22728 Mar  2  2017 libXdmcp.so.6.0.0

This deviation from docker image is a show stopper as the applications depending on this library will fail to start if we use kaniko based images (will get errors similar to below), the app works fine if we use the image built with docker.

  File "/usr/local/lib/python3.10/site-packages/cv2/__init__.py", line 8, in <module>
    from .cv2 import *
ImportError: libXdmcp.so.6: cannot open shared object file: No such file or directory

Additional Information

Dockerfile (Please refer to the reproduce steps)

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [x]
Please check if this error is seen when you use `--cache` flag	- [ ]
Please check if your dockerfile is a multistage dockerfile	- [ ]

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 4
Comments: 15 (10 by maintainers)

Commits related to this issue

hasher: hash security.capability attributes (#1994) In Dockerfile, if there is something like: ``` RUN setcap cap_net_raw=+ep /path/to/binary ``` kaniko won't detect that there is a change on... — committed to GoogleContainerTools/kaniko by zhouhaibing089 2 years ago

Most upvoted comments

I think that is super important, to get fixed, since this holds people back from using kaniko…

gabyx on Mar 17, 2022

I have the following output which shows the problem over two stages

INFO[0504] Taking snapshot of full filesystem...        
INFO[0513] RUN ls -al /usr/lib/x86_64-linux-gnu/libbsd.so.0 
INFO[0513] cmd: /bin/sh                                 
INFO[0513] args: [-c ls -al /usr/lib/x86_64-linux-gnu/libbsd.so.0] 
INFO[0513] util.Lookup returned: &{Uid:1000 Gid:1000 Username:developer Name: HomeDir:/home/developer} 
INFO[0513] performing slow lookup of group ids for developer 
INFO[0513] Running: [/bin/sh -c ls -al /usr/lib/x86_64-linux-gnu/libbsd.so.0] 
lrwxrwxrwx 1 root root 16 Aug  8  2019 /usr/lib/x86_64-linux-gnu/libbsd.so.0 -> libbsd.so.0.10.0
INFO[0513] Taking snapshot of full filesystem...        
INFO[0515] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0515] WORKDIR "/home/$USER_NAME"                   
INFO[0515] cmd: workdir                                 
INFO[0515] Changed working directory to /home/developer 
INFO[0515] No files changed in this command, skipping snapshotting. 
INFO[0515] Storing source image from stage 0 at path /kaniko/stages/0 
INFO[0532] Deleting filesystem...                       

----------------------------------- HERE IS THE FIRST STAGE FINISHED (all good)

INFO[0540] Base image from previous stage 0 found, using saved tar at path /kaniko/stages/0 
INFO[0540] Executing 0 build triggers                   
INFO[0543] Unpacking rootfs as cmd RUN ls -al /usr/lib/x86_64-linux-gnu/libbsd.so.0 requires it. 
INFO[0572] ARG SECRETS_DIR                              
INFO[0572] ARG BUILD_DIR                                
INFO[0572] ARG USER_NAME                                
INFO[0572] ARG PYTHON_ENV                               
INFO[0572] ARG BUILD_VERSION                            
INFO[0572] ENV CONTAINER_BUILD_VERSION=$BUILD_VERSION   
INFO[0572] RUN ls -al /usr/lib/x86_64-linux-gnu/libbsd.so.0 
INFO[0572] Taking snapshot of full filesystem...        
INFO[0580] cmd: /bin/sh                                 
INFO[0580] args: [-c ls -al /usr/lib/x86_64-linux-gnu/libbsd.so.0] 
INFO[0580] util.Lookup returned: &{Uid:1000 Gid:1000 Username:developer Name: HomeDir:/home/developer} 
INFO[0580] performing slow lookup of group ids for developer 
INFO[0580] Running: [/bin/sh -c ls -al /usr/lib/x86_64-linux-gnu/libbsd.so.0] 
ls: cannot access '/usr/lib/x86_64-linux-gnu/libbsd.so.0': No such file or directory

----------------------------------- Now suddenly the symlink is missing.

It seems that the next initial state does not unpack symlinks properly (???) This happens over the following stage from AAAA to BBBB

### FIRST STAGE
FROM ... as AAAA

RUN ls -al /usr/lib/x86_64-linux-gnu/libbsd.so.0

WORKDIR "/home/$USER_NAME"

### SECOND STAGE *until here all good*
FROM AAAA as BBBB
ARG SECRETS_DIR
ARG BUILD_DIR
ARG USER_NAME
ARG PYTHON_ENV
ENV CONTAINER_ID="7fff7106-af66-4e23-bba8-2278408b982c""

RUN ls -al /usr/lib/x86_64-linux-gnu/libbsd.so.0

Somehow I cannot find an MWE so far which shows the problem…

gabyx on Mar 17, 2022