kaniko: Missing files in Kaniko-built image

Actual behavior When building from Red Hat’s UBI 8 and installing nginx a library is missing in the resulting image preventing docker from pulling it.

$ docker pull registry.example.com/nginx:v0.2
v0.2: Pulling from it/docker-images/nginx
03e56b46bf0b: Already exists
3a13cc2f5d65: Already exists
d42cbf90e64a: Pull complete
b4427cf0b409: Extracting [==================================================>] 
 22.93MB/22.93MB
97d00bd6afb4: Download complete
failed to register layer: Error processing tar file(exit status 1): open /lib64/libtirpc.so.3.0.0: no such file or directory

Expected behavior The files should be present

To Reproduce

$ docker run --rm -v $(pwd):/workspace gcr.io/kaniko-project/executor:debug

Additional Information

$ docker run --rm gcr.io/kaniko-project/executor:debug version  
Kaniko version :  v0.17.1

$ docker version
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea838
 Built:             Wed Nov 13 07:26:10 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea838
  Built:            Wed Nov 13 07:24:42 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Build Context:

  • Dockerfile

    FROM registry.access.redhat.com/ubi8/ubi-minimal
    
    # Copy nginx repo
    COPY nginx.repo /etc/yum.repos.d/nginx.repo
    
    # TODO: Add --nodocs or --setopt=tsflags=nodocs once
    # https://bugzilla.redhat.com/show_bug.cgi?id=1769831
    # is closed
    RUN microdnf install nginx \
      && microdnf clean all
    
    EXPOSE 8080/tcp
    
    VOLUME /usr/share/nginx/html
    
    # Run nginx
    CMD ["/usr/sbin/nginx", "-g", "daemon off;"]
    
  • nginx.repo

    [nginx-stable]
    name=nginx stable repo
    baseurl=http://nginx.org/packages/rhel/$releasever/$basearch/
    gpgcheck=1
    enabled=1
    gpgkey=https://nginx.org/keys/nginx_signing.key
    module_hotfixes=true
    
  • Kaniko Image (fully qualified with digest)
    gcr.io/kaniko-project/executor:debug@sha256:025bd79d3e0699b5f59142b03f7e66916980bd0e32653b9c7e21b561d4e538c3

Triage Notes for the Maintainers

Description Yes/No
Please check if this a new feature you are proposing
  • - [ ]
Please check if the build works in docker but not in kaniko
  • - [x]
Please check if this error is seen when you use --cache flag
  • - [ ]
Please check if your dockerfile is a multistage dockerfile
  • - [ ]

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 5
  • Comments: 18 (5 by maintainers)

Commits related to this issue

Most upvoted comments

Thanks @HaehnleinMar Sorry for the regression. Please use v0.16.0 untill we fix this issue.

@mareksuscak That is to be expected, see the –ignore-var-run option, it defaults to true.

The problem has to do with symbolic links, as has been written here and elsewhere a couple of times. In ubi8 there are 4 such links:

[root@c17cc86c6c20 /]# ls -l /
lrwxrwxrwx   1 root root    7 Apr 23  2020 bin -> usr/bin
lrwxrwxrwx   1 root root    7 Apr 23  2020 lib -> usr/lib
lrwxrwxrwx   1 root root    9 Apr 23  2020 lib64 -> usr/lib64
lrwxrwxrwx   1 root root    8 Apr 23  2020 sbin -> usr/sbin

Inside the lib folders are links like

[root@c17cc86c6c20 /]# ls -l usr/lib64/*ssl*
lrwxrwxrwx 1 root root     16 Mar 25 16:49 usr/lib64/libssl.so.1.1 -> libssl.so.1.1.1g
-rwxr-xr-x 1 root root 615576 Mar 25 16:49 usr/lib64/libssl.so.1.1.1g

Inside kaniko they look like this, however:

lrwxrwxrwx  1 root root     23 Apr 14 10:25 libssl.so.1.1 -> ../../lib/libssl.so.1.1

And do not work.

A simple workaround is, to add the following line at the top of the Dockerfile for ubi8 derived images:

FROM quay.io/keycloak/keycloak:11.0.3
# Fix resolving path when building with kaniko:
ENV LD_LIBRARY_PATH=/usr/lib64:/usr/lib

With this, the kaniko build succeeds!

Similar problem with postgres package, kaniko 0.17.1 produces invalid image. 0.18.0 is OK 🎉

@binnythomas-1989 please use the tag v0.16.0-debug