kaniko: multistage build in same container fails because cross stage deps are not cleaned up ( symlinks )

Actual behavior I want to set up image building for our project as part of CI pipeline using GitLab CI capabilities. Following https://docs.gitlab.com/13.2/ee/ci/docker/using_kaniko.html#building-a-docker-image-with-kaniko I done CI configuration and it works perfect if you build one image per job. It is not GitLab issue, just bear with me.

We have a multi stage Dockerfile to build our images. So if you try build multiple targets inside same ( and this is crucial ) container it will fail with:

error building image: could not save file: symlink ../chi-teck/drupal-code-generator/bin/dcg /kaniko/0/app/vendor/bin/dcg: file exists

Expected behavior

Two (in my case) images built.

To Reproduce Output of commands that successfully ran is omitted:

$ docker run --rm  --interactive  --tty  --volume $PWD:/app  --user $(id -u):$(id -g)  composer:1  create-project --ignore-platform-reqs drupal/recommended-project kaniko-test
$ cd kaniko-test
$ docker run --rm  --interactive  --tty  --volume $PWD:/app  --user $(id -u):$(id -g)  composer:1  require --ignore-platform-reqs drush/drush:^10
$ cat <<EOF >> Dockerfile
FROM composer:1 AS full-code-base
WORKDIR /app
COPY composer.json composer.lock /app/
RUN composer install --ignore-platform-reqs --no-dev --working-dir=/app
COPY web /app/web
RUN composer dump-autoload --optimize --working-dir=/app

FROM php:7.4-fpm-buster AS project-php
COPY --from=full-code-base /app /app

FROM nginx:1 AS project-nginx
COPY --from=full-code-base /app /app

EOF
$ docker run --network=host -v $(pwd):/workspace  --entrypoint '' --rm -it gcr.io/kaniko-project/executor:debug sh
inside container $ executor --target project-php --destination kanico-test-image:php-latest --no-push
inside container $ executor --target project-nginx --destination kanico-test-image:nginx-latest --no-push
INFO[0000] Resolved base name composer:1 to full-code-base
INFO[0000] Resolved base name php:7.4-fpm-buster to project-php
INFO[0000] Resolved base name nginx:1 to project-nginx
INFO[0000] Retrieving image manifest composer:1
INFO[0000] Retrieving image composer:1
INFO[0003] Retrieving image manifest composer:1
INFO[0003] Retrieving image composer:1
INFO[0006] Retrieving image manifest php:7.4-fpm-buster
INFO[0006] Retrieving image php:7.4-fpm-buster
INFO[0009] Retrieving image manifest php:7.4-fpm-buster
INFO[0009] Retrieving image php:7.4-fpm-buster
INFO[0012] Retrieving image manifest nginx:1
INFO[0012] Retrieving image nginx:1
INFO[0014] Retrieving image manifest nginx:1
INFO[0014] Retrieving image nginx:1
INFO[0017] Built cross stage deps: map[0:[/app /app]]
INFO[0017] Retrieving image manifest composer:1
INFO[0017] Retrieving image composer:1
INFO[0019] Retrieving image manifest composer:1
INFO[0019] Retrieving image composer:1
INFO[0022] Executing 0 build triggers
INFO[0022] Unpacking rootfs as cmd COPY composer.json composer.lock /app/ requires it.
INFO[0033] WORKDIR /app
INFO[0033] cmd: workdir
INFO[0033] Changed working directory to /app
INFO[0033] No files changed in this command, skipping snapshotting.
INFO[0033] COPY composer.json composer.lock /app/
INFO[0033] Taking snapshot of files...
INFO[0033] RUN composer install --ignore-platform-reqs --no-dev --working-dir=/app
INFO[0033] Taking snapshot of full filesystem...
INFO[0036] cmd: /bin/sh
INFO[0036] args: [-c composer install --ignore-platform-reqs --no-dev --working-dir=/app]
INFO[0036] Running: [/bin/sh -c composer install --ignore-platform-reqs --no-dev --working-dir=/app]
Loading composer repositories with package information
Installing dependencies from lock file
Nothing to install or update
Generating autoload files
INFO[0036] Taking snapshot of full filesystem...
INFO[0037] Taking snapshot of files...
INFO[0037] COPY web /app/web
INFO[0039] Taking snapshot of files...
INFO[0041] RUN composer dump-autoload --optimize --working-dir=/app
INFO[0041] cmd: /bin/sh
INFO[0041] args: [-c composer dump-autoload --optimize --working-dir=/app]
INFO[0041] Running: [/bin/sh -c composer dump-autoload --optimize --working-dir=/app]
Generating optimized autoload files
Generated optimized autoload files containing 4906 classes
INFO[0042] Taking snapshot of full filesystem...
INFO[0046] Saving file app for later use
error building image: could not save file: symlink ../chi-teck/drupal-code-generator/bin/dcg /kaniko/0/app/vendor/bin/dcg: file exists

I’ve tried to raise verbosity level to debug - nothing useful. With trace it shows way too much to digest.

Directory content of vendor/bin is:

$ ll vendor/bin/
total 8
drwxr-xr-x  2 kirill kirill 4096 Aug 31 16:45 ./
drwxr-xr-x 31 kirill kirill 4096 Aug 31 16:45 ../
lrwxrwxrwx  1 kirill kirill   41 Aug 31 16:45 dcg -> ../chi-teck/drupal-code-generator/bin/dcg*
lrwxrwxrwx  1 kirill kirill   20 Aug 31 16:45 drush -> ../drush/drush/drush*
lrwxrwxrwx  1 kirill kirill   33 Aug 31 16:45 php-parse -> ../nikic/php-parser/bin/php-parse*
lrwxrwxrwx  1 kirill kirill   22 Aug 31 16:45 psysh -> ../psy/psysh/bin/psysh*
lrwxrwxrwx  1 kirill kirill   44 Aug 31 16:45 release -> ../consolidation/self-update/scripts/release*
lrwxrwxrwx  1 kirill kirill   26 Aug 31 16:45 robo -> ../consolidation/robo/robo*
lrwxrwxrwx  1 kirill kirill   51 Aug 31 16:41 var-dump-server -> ../symfony/var-dumper/Resources/bin/var-dump-server*

Additional Information

• Dockerfile Included in the steps to reproduce
• Build Context Included in the steps to reproduce
• Kaniko Image (fully qualified with digest)

$ docker inspect gcr.io/kaniko-project/executor:debug
[
    {
        "Id": "sha256:b0070f18add278df20229ce34172fc16a4c76392fc28d33df7837396a2b882c0",
        "RepoTags": [
            "gcr.io/kaniko-project/executor:debug"
        ],
        "RepoDigests": [
            "gcr.io/kaniko-project/executor@sha256:0f27b0674797b56db08010dff799c8926c4e9816454ca56cc7844df228c53485"
        ],
        "Created": "2020-08-18T02:40:08.570969026Z",
        "DockerVersion": "19.03.8",
        "Architecture": "amd64",
        "Os": "linux",
}

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [x]
Please check if the build works in docker but not in kaniko	- [x]
Please check if this error is seen when you use --cache flag	- [x]
Please check if your dockerfile is a multistage dockerfile	- [x]