buildkit: bug: buildkit does not consider ONBUILD COPY --from

Just chiming in here - we do the same as @astorath at our company. It’s been a godsend for enabling extremely efficient / modular builds and dockerfiles

For context: We discovered this issue after trying to use (new) build secrets (with buildkit)

If you are hitting this I strongly also advise you to look if this is actually a correct solution for your problem

I think this solution is born out of the need to create Dockerfiles for a lot of similarly structured projects combined with an attempt to follow best practices. Let’s say you have a setup where:

• build stage takes your source and produces some binaries
• runtime stage copies/installs the binaries into the production image

This is, of course, trivial to achieve with multi-stage and that’s the recommended approach (keeping build and runtime environments separate). However, this doesn’t really scale well, as you’d need to repeat this same Dockerfile for 10, 20 or even 50 projects (not uncommon for systems powered by microservices).

Before BuildKit, (arguably) the neatest way to achieve this was with multi-stage ONBUILD files. It enables developers to write extremely modular/reusable Dockerfiles, whilst keeping Docker workflows native (docker/docker-compose build just works).

Given BuildKit feature set, I agree that the solution no longer seems like a great fit. @tonistiigi is the vision to fill this gap with custom BuildKit frontends?

If the path forward is via BuildKit custom frontends, it’d be great to make dockerfile.Build more easily extensible. Creating a custom frontend is quite an undertaking today, as it requires a lot of specialist knowledge (LLB) and boiler plate code (not to mention additional duplication if the end goal is to use Dockerfile syntax). It’d be awesome if there was a simpler API for this to match the learning curve of multi-stage Dockerfiles. For example, what I can imagine doing is:

• Have a streamlined Dockerfile similar to below:

# syntax = dockerfiles/java-microservice
# Any additional (project-specific) customisations can be specified via familiar Dockerfile syntax
COPY ... 
RUN ... 

• Have a custom frontend that uses some default pre-baked actions, making the effective Dockerfile look something like this:

FROM jdk as build
COPY . /src
RUN mvn ... # Build the application in a streamlined fasion

FROM jre
COPY --from=build /src/build /bin

# Any additional (project-specific) customisations specified in source
COPY ... 
RUN ... 

• If there’s an easy way to load files from build context, this is even more powerful; You’d now be able to have a generic frontend that can combine Java/Python/etc functionality in a single streamlined entrypoint:

# syntax = dockerfiles/microservice
# Any additional (project-specific) customisations can be specified via familiar Dockerfile syntax
COPY ... 
RUN ... 

What are you thoughts on this?

Is there any update on a solution for this? I’ve just discovered this limitation while building out some reusable images for a large project. I have multi-stage images which declare instructions such as ONBUILD COPY --from=other-image /foo /bar and also use BuiltKit secrets mechanism in ONBUILD instructions: ONBUILD RUN --mount=type=secret,id=my-secret,uid=101 source /run/secrets/my-secret && install-dependencies. The instructions are not being run in downstream image builds, which is a serious limitation. If it worked it would allow me to reduce the amount of boiler plate and have a set of reusable layers for application images across a large project.

EDIT: As I wrote the above I realized that I only use BuildKit for the secrets mechanism. If I can find an alternative I may drop BuildKit due to this limitation.

Here is the switch https://github.com/companieshouse/ch.gov.uk/pull/192

Just set environment arg DOCKER_BUILDKIT=0 before docker build

Hey folks 👋 Docker Desktop 2.4.0.0 enables BuildKit by default now, which will potentially break any Dockerfiles dependent on this functionality. I’m happy to work with maintainers to get this addressed if capacity is an issue here (will possibly need some pointers on where to start though). It looks like @tonistiigi’s suggestion could be a way forward:

• BuildKit starts by pulling the all external FROMs
• BuildKit checks for stage additional dependencies in ONBUILD
• BuildKit performs stage builds in parallel where possible (as before, but now accounting for --from)

@tonistiigi

This is a very weird (and inventive) way to use COPY --from.

Well, this is not my invention, https://engineering.busbud.com/2017/05/21/going-further-docker-multi-stage-builds/ - this is one of the first google search results. So I don’t think I’m alone here.

@thaJeztah

It’s not deprecated, but I think the official images stopped creating unbuild variants, due to the behaviour being confusing.

Maybe this is confusing for official images, but for multistage internal images designed for that purpose - ONBUILD is a revelation…

Down the link above is a nice solution to the problem: if we could use something like:

ONBUILD FROM runtime:v1

we won’t need this strange hack.

Hi, so there going to be some fix or you’re just breaking backward compatibility by literally saying “you always do it wrong way, our way is better! Just do refactoring of all your dockerfiles and ci/cd”? )

Ditto, when is this getting fixed?

Quote from https://engineering.busbud.com/2017/05/21/going-further-docker-multi-stage-builds/:

I’m not sure if it’s a bug, a feature, or a undefined behavior

As the author of the multi-stage PR I can confirm I was not clever enough to see it as a possible feature.

Just for the sake of adding another usage example of this (now missing) feature: https://github.com/r2d2bzh/docker-build-nodejs

This project started internally 3 years ago. At this time it was designed this way after reading the article pointed by @astorath in a previous comment and it also closely relates to the situation described by @EricHripko.

Many of us are still not using buildkit. This means that, for now, we have to support both the pre-buildkit and post-buildkit environments. The only way to do that ATM is to provide two different build methods, one for non-buildkit users and another for buildkit users.

What is also clearly odd for someone using the docker compose plugin is that these errors start popping between versions 2.3.3 and 2.3.4 of the plugin. It took me at least an hour to relate these behaviors to this issue, this is OK but perhaps having a warning on this particular subject in the Dockerfile reference (or somewhere else) might be helpful to spot or avoid the issue more easily. I can help documenting this, provided the proper guidance.

This is a very weird (and inventive) way to use COPY --from.

• Can I turn off these optimizations to run builds sequentially?
• Is there a way to mark these stages as dependent explicitly?

No, we don’t want to add any special behavior for this. Either we consider this as bug and just fix the image resolution or we document that this is invalid usage (and make it error with a proper message).

A problem with implementing this is that we can’t start to process all stages in parallel like we do now because we only know about the dependant images after we have pulled the config of the base. Still doable, just adds complexity. Once we have determined the correct dependencies of the stages it would build in regular concurrent manner.

@AkihiroSuda @tiborvass @ijc @thaJeztah get your votes in

Any update? As we now see:

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

This will kill all our builds.

@softworm This does not solve the problem. Turning off BuiltKit means you cannot use the secrets mechanism, which is the only decent approach for passing secrets into a build.

any update? same issue on multi-stage ONBUILD copy.

For anyone that loves to work on this issue, I’ve included steps to reproduce the issue in moby/moby#42845 and kept those as simple as possible.

Could you provide a working fix based on my minimal example ?

Note that the use of images instead of stage names in COPY --from is not documented, so you should use probably stage aliases as shown.

./template/Dockerfile:

FROM php:7.4.2-fpm-buster
ONBUILD COPY --from=template /usr/bin/composer /usr/bin/composer

./project/Dockerfile:

FROM composer:1.9.2 AS template
RUN ["touch", "/tmp/dummy"]

FROM bug_template
COPY --from=template /tmp/dummy /tmp/dummy

DOCKER_BUILDKIT=1 docker build ./template --tag bug_template
DOCKER_BUILDKIT=1 docker build ./project