kaniko: Copying rootfs fails

Dockerfile to reproduce:

FROM debian:buster-slim AS importer

RUN apt-get update && apt-get install -y libguestfs-tools wget xz-utils
RUN wget -nv https://raspi.debian.net/verified/20210718_raspi_4_buster.img.xz && xz -d 20210718_raspi_4_buster.img.xz && mkdir /rootfs && guestfish --ro -a 20210718_raspi_4_buster.img -m /dev/sda2 copy-out / /rootfs/ && rm 20210718_raspi_4_buster.img

FROM scratch
COPY --from=importer /rootfs/ /

For v1.6.0 this hangs at “Saving file rootfs for later use”. This is also described in #960 and the PR #1724 improves the situation. However, it now stops in the copy step reporting

error building image: error building stage: failed to execute command: copying dir: chown /sys: read-only file system

The workaround from #1007 seems to work though.

About this issue

Original URL
State: open
Created 3 years ago
Reactions: 1
Comments: 15 (2 by maintainers)

Most upvoted comments

Oh that’s bad, I just bought a new Macbook with M1 chip, and get confused by amount of architectures 😅

Images are fixed now:
ghcr.io/kvaps/kaniko-executor:v1.7.0@sha256:63c8fddec65e5b48ab41657da537f9f2360d6fbc179ecc096c414597d739143a
ghcr.io/kvaps/kaniko-executor:v1.7.0-debug@sha256:2124a5418d126d8684ed03e539b08d1b4bc5f1cc23751abeff182802cb7906ad
ghcr.io/kvaps/kaniko-warmer:v1.7.0@sha256:a3f95ed186b37cf792f5dd0f648d8b2ba4907ddb8dffee3ab0cee97a01dbeb45
The same images are used in my kubectl-build plugin

Unfortunately it doesn’t work either.

I tried this example:

FROM registry.access.redhat.com/ubi9 AS ubi-micro-build
RUN mkdir -p /mnt/rootfs
RUN dnf install --installroot /mnt/rootfs curl --releasever 9 --setopt install_weak_deps=false --nodocs -y; dnf --installroot /mnt/rootfs clean all

FROM quay.io/keycloak/keycloak
COPY --from=ubi-micro-build /mnt/rootfs /

mav96 on May 12, 2023

Maybe try like this? It’s a bit verbose but should work around the symlink issue. I did notice that our Kaniko version is wildly outdated, might need to look into upgrading that 😅 gcr.io/kaniko-project/executor:debug-v0.22.0

FROM quay.io/keycloak/keycloak:22.0.2

COPY --from=ubi-micro-build /mnt/rootfs/usr/bin/curl /usr/bin/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcurl.so.4 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcurl.so.4.7.0 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libnghttp2.so.14 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libidn2.so.0 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libssh.so.4 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libpsl.so.5 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libssl.so.3 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcrypto.so.3 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libgssapi_krb5.so.2 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libkrb5.so.3 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libk5crypto.so.3 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcom_err.so.2 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libldap.so.2 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/liblber.so.2 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libbrotlidec.so.1 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libunistring.so.2 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libkrb5support.so.0 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libkeyutils.so.1 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libevent-2.1.so.7 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libsasl2.so.3 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libbrotlicommon.so.1 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcrypt.so.2 /usr/lib64/

Depending on versions of stuff, you might need to copy over more or less of these .so files. I found the list for my instance like this

$ docker run -it kc '--&& ldd /usr/bin/curl'                                                                                                                                                                                                                  
        linux-vdso.so.1 (0x00007ffc9fbdd000)
        libcurl.so.4 => /lib64/libcurl.so.4 (0x00007fbd74599000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fbd7457f000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fbd74376000)
        libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007fbd7434c000)
        libidn2.so.0 => /lib64/libidn2.so.0 (0x00007fbd7432b000)
        libssh.so.4 => /lib64/libssh.so.4 (0x00007fbd742b7000)
        libpsl.so.5 => /lib64/libpsl.so.5 (0x00007fbd742a1000)
        libssl.so.3 => /lib64/libssl.so.3 (0x00007fbd741fb000)
        libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007fbd73dca000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fbd73d73000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fbd73c98000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fbd73c7f000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fbd73c76000)
        libldap.so.2 => /lib64/libldap.so.2 (0x00007fbd73c0f000)
        liblber.so.2 => /lib64/liblber.so.2 (0x00007fbd73bfd000)
        libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007fbd73bef000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fbd7467d000)
        libunistring.so.2 => /lib64/libunistring.so.2 (0x00007fbd73a6a000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fbd73a57000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fbd73a50000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fbd73a3c000)
        libevent-2.1.so.7 => /lib64/libevent-2.1.so.7 (0x00007fbd739e3000)
        libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007fbd739c3000)
        libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007fbd7399e000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fbd73971000)
        libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007fbd73937000)
        libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007fbd7389b000)
        ```

Hi @niekcandaele,

Thank you a lot for support. We are using keycloak operator to generate keycloak instance. I just updated keycloak docker image and it requires to update keycloak operator to 22.0.4 as well (I forgot to update this one) and it is using http-get to do health check (https://www.keycloak.org/server/health#_kubernetes), then no need install curl anymore and no need to deal with install stuff currently. Thanks!

chary1112004 on Oct 24, 2023

FROM quay.io/keycloak/keycloak:22.0.2

COPY --from=ubi-micro-build /mnt/rootfs/usr/bin/curl /usr/bin/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcurl.so.4 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcurl.so.4.7.0 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libnghttp2.so.14 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libidn2.so.0 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libssh.so.4 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libpsl.so.5 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libssl.so.3 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcrypto.so.3 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libgssapi_krb5.so.2 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libkrb5.so.3 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libk5crypto.so.3 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcom_err.so.2 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libldap.so.2 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/liblber.so.2 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libbrotlidec.so.1 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libunistring.so.2 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libkrb5support.so.0 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libkeyutils.so.1 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libevent-2.1.so.7 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libsasl2.so.3 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libbrotlicommon.so.1 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcrypt.so.2 /usr/lib64/

Depending on versions of stuff, you might need to copy over more or less of these .so files. I found the list for my instance like this

$ docker run -it kc '--&& ldd /usr/bin/curl'                                                                                                                                                                                                                  
        linux-vdso.so.1 (0x00007ffc9fbdd000)
        libcurl.so.4 => /lib64/libcurl.so.4 (0x00007fbd74599000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fbd7457f000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fbd74376000)
        libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007fbd7434c000)
        libidn2.so.0 => /lib64/libidn2.so.0 (0x00007fbd7432b000)
        libssh.so.4 => /lib64/libssh.so.4 (0x00007fbd742b7000)
        libpsl.so.5 => /lib64/libpsl.so.5 (0x00007fbd742a1000)
        libssl.so.3 => /lib64/libssl.so.3 (0x00007fbd741fb000)
        libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007fbd73dca000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fbd73d73000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fbd73c98000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fbd73c7f000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fbd73c76000)
        libldap.so.2 => /lib64/libldap.so.2 (0x00007fbd73c0f000)
        liblber.so.2 => /lib64/liblber.so.2 (0x00007fbd73bfd000)
        libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007fbd73bef000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fbd7467d000)
        libunistring.so.2 => /lib64/libunistring.so.2 (0x00007fbd73a6a000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fbd73a57000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fbd73a50000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fbd73a3c000)
        libevent-2.1.so.7 => /lib64/libevent-2.1.so.7 (0x00007fbd739e3000)
        libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007fbd739c3000)
        libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007fbd7399e000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fbd73971000)
        libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007fbd73937000)
        libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007fbd7389b000)
        ```

niekcandaele on Oct 24, 2023

Could you please share which version of kaniko that you are using?

I am using version gcr.io/kaniko-project/executor:v1.9.0-debug which is also not working.

I am building the Dockerfile inside of a AWS EKS (Kubernetes) cluster on Fargate (AWS Serverless compute) pods. They are running without privileged mode.

sanderjochems-whyellow on Oct 24, 2023

Here’s my Dockerfile (with some unrelated things stripped out). Hope this helps

FROM registry.access.redhat.com/ubi9 AS ubi-micro-build
RUN mkdir -p /mnt/rootfs
RUN dnf install --installroot /mnt/rootfs curl --releasever 9 --setopt install_weak_deps=false --nodocs -y; dnf --installroot /mnt/rootfs clean all


FROM quay.io/keycloak/keycloak:22.0.2 as builder

WORKDIR /opt/keycloak

COPY --from=providergenerator /providers/bin/ /opt/keycloak/providers/

RUN /opt/keycloak/bin/kc.sh build

FROM quay.io/keycloak/keycloak:22.0.2

COPY --from=ubi-micro-build /mnt/rootfs/usr/bin/curl /usr/bin/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/* /usr/lib64/
COPY --from=builder /opt/keycloak /opt/keycloak

ENTRYPOINT ["/opt/keycloak/bin/kc.sh", "start"]

niekcandaele on Oct 23, 2023

Heya, I made an oopsie when copying my suggestion, my bad! 🙈

Give the following a try:

COPY --from=ubi-micro-build /mnt/rootfs/usr/bin/curl /usr/bin/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/* /usr/lib64/

Tested like this

catalysm:px/ (develop✗) $ docker run -it kc '--&& bash'                                                                                                                                                                                                                      
bash-5.1$ curl --help
Usage: curl [options...] <url>
 -d, --data <data>   HTTP POST data
 -f, --fail          Fail silently (no output at all) on HTTP errors
 -h, --help <category>  Get help for commands
 -i, --include       Include protocol response headers in the output
 -o, --output <file>  Write to file instead of stdout
 -O, --remote-name   Write output to a file named as the remote file
 -s, --silent        Silent mode
 -T, --upload-file <file>  Transfer local FILE to destination
 -u, --user <user:password>  Server user and password
 -A, --user-agent <name>  Send User-Agent <name> to server
 -v, --verbose       Make the operation more talkative
 -V, --version       Show version number and quit

This is not the full help, this menu is stripped into categories.
Use "--help category" to get an overview of all categories.
For all options use the manual or "--help all".
bash-5.1$ curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

niekcandaele on Oct 23, 2023

I struggeled using your image in our Gitlab CI/CD pipeline. I digged a little deeper and it turns out that multiple achitectures are mixed up there. When pulling from x64 I obtain an arm64 image where for example /busybox/sh is indeed arm64 but the executor is x64:

uname -m
# x86_64
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
mkdir -p /tmp/123
docker run --rm -it -v /tmp/123:/context --entrypoint /busybox/sh ghcr.io/kvaps/kaniko-executor:v1.7.0-debug
# WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested
# cp /busybox/sh /context/
# cp /kaniko/executor /context/
# exit
file /tmp/123/sh
# sh: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, stripped
file /tmp/123/executor
# executor: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=7BkQGRwXsNnIqlKjVroQ/0m_j6BzaX-FhkPpGcJaR/-9sTmtlaXltjhcGsg2mS/HMbpOMzbZstMMB7jfvs1, stripped

ghost on Nov 9, 2021