buildkit: Cannot give a local image to FROM when using docker-container

If you want your builds to access Docker images you need to use the Docker driver

Understood. But isn’t the point of buildkit and the various drivers so that we can open new capabilities? buildkit does this phenomenally well, but some of those capabilities are not yet retrofitted into the docker driver. So you have to choose between:

• images in the docker cache to use as FROM
• new buildkit capabilities (like multi-arch and OCI format storage and remote builders and etc. etc.)

I think we are trying to bridge this gap: if docker driver could do everything buildkit can (which it eventually will, I understand), and buildkit containerized could do everything docker could (again, within reason), then the choice between features Docker and features buildkit (and if you need both, you are out of luck) wouldn’t be a problem.

As the docs explain in most cases this isn’t even needed and you should just use multi-stage builds. This old pattern was invented before multi-stage builds support was added and is only needed for cases where you have a requirement to build multiple projects that depend on each other and can’t be merged(and you can’t access any registry).

Sure, and I built tons of images that way in the early days. And when I could move to multistage after it existed, I did.

But is it a fair assumption that every (or even a majority) of builds are simple linear “start at point A, go through some intermediates, get to point Z” and thus multistage candidates?

It is very common to have a base image (A), build some “golden base” (B), then some intermediates (C, D, E, F), then some finals (Q, R, S, T, U, V, W, X, Y, Z). This “tree” isn’t a single build that can go in a multistage Dockerfile, and the intermediates aren’t stored in a remote registry: they might be private, they might be part of a testing process and aren’t valid to be stored until the finals (Q, R, S, T, U, V, W, X, Y, Z) are generated and fully tested (if at all). The local cache holds all of those interim images, and the later steps absolutely are not a single multistage build, simply cannot be. They cannot even be run at the same time (like bake with a bake.hcl).

1. Build B from A.
2. Build C, D, E, F from B
3. Build Q, R, S, T, U, V, W, X, Y, Z from C, D, E, F

These are separate build processes, might be running at separate times. This isn’t a single multi-stage Dockerfile (imagine what a nightmare that single file would be), and isn’t run as a single step or stage in a CI process.

Let’s be practical: someone has been doing this for years with docker build. It works great (it really does, hats off). They see buildkit, love what it can do, need those features that do not yet exist in the docker driver, so they try to go containerized.

And their entire build process breaks. Because step 1 stored output B, and step 2 cannot find B for C,D,E,F (repeat for step 3).

If you use named context you only have one command with no extra arguments that takes care of all of this.

That’s just it. The above very common use case describes how “one command” isn’t practical here. It has to be multiple commands. Yet docker build combined with FROM and local caches made those multiple commands dead simple to grasp and use. That was a key part of docker’s adoption.

The part I am having a really hard time figuring out is, why the strong objection to proper caching inside buildkit container? You already store blobs, you already have the OCI layout, the number of additional steps to get to parity with default docker behaviour is so small (and people have offered to help build it). Why the resistance?

While explaining this issue to some people, I realized the most basic use case why it matters, and why multi-staged docker files don’t solve it, why bake and build contexts only partially do it.

Let’s say that I am developing an image that normally is public, call it foo/myimage. Part of my build and test process is that there are downstream images, completely independent, that depend on foo/myimage, call them bar/image and bar/other.

A normal, sane, build process has me do the following:

1. Do work on the source of foo/myimage
2. docker build -t foo/myimage:testout .
3. Do not push foo/myimage:testout, since I have absolutely no idea if that is the finally version until I work up the downstream dependencies
4. Modify the Dockerfiles for the source of bar/image and bar/other
5. docker build -t bar/image:something and docker build -t bar/other:else
6. If all works, docker push foo/myimage:testout

The above is a great, simple chaining process, very Docker-ish, and 100% depends on foo/myimage:testout being available in some local read-through cache.

Even with build-contexts, how would I do my normal process? What can I give to docker build -t bar/image:something that will point it at local output? buildkit container-drive only understands a networked registry derived from the FROM image ref, and build contexts require that I explicitly give it the name (meaning yet another part of my build process I need to edit and manage, in addition to the Dockerfile), but it also has no “source” that I can point at the output of the first.

In theory, I could do a docker save on my first image, and then untar it, or maybe build the output with -o to a local file, but my process got much more complex, and if my testing succeeds, I need to rebuild it again in order to push it. Caching will make it more efficient, but yet another step to run.

Am I doing a decent enough job explaining how this breaks normal processes?

As far as I can tell - please do correct me - the buildx bake context stuff allows you to, essentially, “alias” a FROM (or --from=) in a Dockerfile or other builder to one of a local directory, an image in a registry, or possibly the results of a previous stage (based on this).

It looks pretty hesitant to use that target reference:

Please note that in most cases you should just use a single multi-stage Dockerfile with multiple targets for similar behavior. This case is recommended when you have multiple Dockerfiles that can’t be easily merged into one.

I don’t understand how this manages to store the output of one to the other, especially with containerized builder. Is it just because it is building both at once, so the builder knows about all of the outputs and can “hold onto” them?

More important UX question: Does this mean that, in order to do the simple “use a a local cached image” that we have been used to for docker since day 1, where I do docker build -t somelocalimage:foo -f Dockerfile1 . and then consume it in another with FROM somelocalimage:foo, I now need to:

1. have a bake.hcl in addition to my Dockerfiles
2. learn the bake syntax
3. actually have both of them there
4. build them both at the same time so the bake file can reference them
5. call a different command

all to replicate a “docker-simple” ™ functionality from the existing flow?

What options are available to me here?

If you switch to the default builder (docker buildx use default), both docker buildx build and docker build should uses the same storage, in which case you can refer to the image.

If you are on a recent version of Docker Desktop or the Docker Engine, you can enable the containerd image store integration. That provides a multi-arch store without requiring a docker-container builder. It’s still in preview, but work is progressing and will be moved out of “experimental” in the not too distant future; see https://docs.docker.com/storage/containerd/

Make sure to switch the builder instance back to the default (not the container-builder, otherwise it continues to use the docker-container driver)

and then run docker buildx build --alias myimage:1.2.3=driver://image where driver is different drivers:

You don’t need a new flag. This is already supported. --build-context myimage:1.2.3=docker-image://image. The source can be a local directory, git repository, docker image, URL to tarball, or another build definition in bake for chaining. Adding another source that is path to OCI file layout does not conflict with the buildkit design.

@felipecrs Bake has a different security model indeed. Atm. for local bake files it is allowed for them to point to any local path. For remote files, we have disabled parent access. In future releases, the plan is to move to https://github.com/docker/buildx/issues/179 to control what is allowed and what is not.

FWIW, this is exactly what we raised in #2210 . builder-container has a regression in functionality compared to docker builder, at least as far as cache is concerned.

I had looked into the containerized builder works, and was surprised that it actually was missing only a few things to make it work. If I recall correctly:

• when building store manifests and indexes in blobs (remove the logic to store only binary blobs)
• when building store image ref to root blob (manifest or index), either in something like the containerd db or simply in index.json (like we do in linuxkit’s cache)
• change the FROM lookup to look in local cache before going to remote registry

You could do other things, like load and save equivalents, which would be helpful, but not, I believe, critically necessary.

I still know of a few people willing to help with it a bit, but it has been 9 months since we opened the issue.

https://github.com/docker/buildx/blob/v0.8.0/docs/reference/buildx_build.md#build-context https://github.com/docker/buildx/blob/v0.8.0/docs/reference/buildx_bake.md#defining-additional-build-contexts-and-linking-targets