kaniko: Multistage build is causing deletion of files from kaniko run environment

Actual behavior Before executor is called in the runtime of kaniko the /root/.docker/config.json file is created to allow kaniko access to private registries. When multistage build is executed before the image of next stage is downloaded kaniko make deletes the filesystem. After the deletion of the file system also the /root/.docker/config.json file is deleted. As consequence the image of the third stage cant be downloaded form private registry anymore. In the 2 stage builds, after both stages are processed the resulting image cant be pushed to private registry because of missing credentials.

This problem affects #407 and was not completely fixed with !192 MR https://github.com/GoogleContainerTools/kaniko/pull/192

Expected behavior Deleting of filesystem is not deleting files from kaniko runtime environment at all.

To Reproduce Steps to reproduce the behavior:

Create dockerfile with 3 stage build and images from private registry
Run kaniko in the container and before executor is called create /root/.docker/config.json file
Call executor and wait till the third stage is executed. The third image cant be pulled from private registry

Additional Information Dockerfile:

FROM $CI_REGISTRY_IMAGE:test2 as pg FROM $CI_REGISTRY_IMAGE:test1 as go FROM $CI_REGISTRY_IMAGE:latest

Build with gitlab ci pipeline:

build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG
  only:
    - tags

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 1
Comments: 18 (2 by maintainers)

Most upvoted comments

Problem still exists.

Askill on Jul 18, 2022

The way that Kaniko seems to work is to use it’s local filesystem, not a chroot (or similar) filesystem, and that’s why it destroys the system. Apps written in C or C++ will fail to find the dynamic linker file (located at /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2), so you’ll have to put it in one of the directories that are not destroyed, and then edit the ELF files to point to it correctly.

Go binaries (like terraform and it’s plugins, which are separate executables, not dynamic libraries) are not dynamic ELF files by default (they are statically linked by default), so they still work. C, C++, and Rust binaries can be configured to by static binaries as well: With C, you’ll probably want to use MUSL for your build tools, and there’s just a line that you add to a config file, and recompile for Rust based build tools that you might want to use.

This sort of solution (Kaniko destroying everything instead of restoring the original state on the second image) is less than optimal.

Type1J on Jun 7, 2021

@devopsinthecloud If you put things in the /kaniko directory, then when Kaniko deletes things, it will leave those thing alone.

Type1J on Aug 17, 2022

@qalinn Yes, and that requires having a very specific structure to your images, and it must be used in every stage. If this could be hidden from the perspective of the Dockerfile, then that might work, but if it’s not transparent (meaning that the same Dockerfile could be used in both Docker Engine and Kaniko), then it would force all parties (devs, QA and production environments) using a Dockerfile to exclusively use Kaniko, which can’t be forced on them.

Type1J on Apr 25, 2022