buildx: Docker buildkit fails to build .net project in contrary to successful builds on depracted builder

Description

I found that when you have more than one dotnet publish in dockerfile, the docker build with buildkit fails with some strange, nondescriptive error: ERROR: failed to solve: failed to prepare 6boxvrjdjur378egamsa297vp as lnddt61dq57lwjio5fkmhme9e: invalid argument. When there is only one dotnet publish line, it works with buildkit.

This failing behavior does not exists when i turn off buildkit via DOCKER_BUILDKIT=0 docker build. It successfully builds image, despite having more than one dotnet publish command.

I attached minimal repro repository, and repro steps. Hope we can clarify of what is going on here.

Reproduce

Download minimal repro repository https://github.com/incloudss/testbuildkit
Go to testbuildkit directory and run docker build .
The build fails.
Run once again build, now with DOCKER_BUILDKIT=0 docker build ..
Build succeeds.

Expected behavior

No response

docker version

Client: Docker Engine - Community
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 20:35:35 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 20:35:35 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 24.0.5
 Storage Driver: vfs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.4.225-200.el7.x86_64
 Operating System: Debian GNU/Linux 12 (bookworm) (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 31.36GiB
 Name: connect-build-agent-karas-5d84b54566-hkz8d
 ID: d76c3467-6972-423b-8208-7a5f12201c2b
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

About this issue

Original URL
State: open
Created 10 months ago
Reactions: 2
Comments: 21 (5 by maintainers)

Most upvoted comments

I wonder if Storage Driver: vfs is related to your issue.

I would definitely NOT recommend using the vfs storage driver for any other purpose than debugging. The vfs storage driver creates a full copy of the image for every layer that’s created, and for every container that’s run (which is both very slow, and may cause a lot of disk space to be used).

There have been some cases where vfs may not be able to write extended attributes (see https://github.com/moby/moby/issues/45535, https://github.com/moby/moby/issues/45417), and older version of docker were silently discarding those errors; docker 25.0 will produce an error in this conditions (added in https://github.com/moby/moby/pull/45464), but other version may either discard that error (or perhaps BuildKit may not).

thaJeztah on Oct 11, 2023

I have the same error but only in CI where I’m using DinD. It never throws when I build locally.

I don’t have multiple publish commands but I the error occurs if I have multiple run statements that call dotnet.

I’m using the latest version of docker (24.0.6)

Docker throws an error with this file

FROM mcr.microsoft.com/dotnet/sdk:7.0 AS build-env
WORKDIR /source

# Copy everything
COPY src/Project .

# Restore as distinct layers
RUN dotnet restore

# Build and publish a release
RUN dotnet publish ./Project.csproj -c Release --no-restore -o /app

# Build runtime image
FROM mcr.microsoft.com/dotnet/aspnet:7.0
WORKDIR /app
COPY --from=build-env /app .
EXPOSE 80
ENTRYPOINT ["./Project"]

But this file works without any issues

FROM mcr.microsoft.com/dotnet/sdk:7.0 AS build-env
WORKDIR /source

# Copy everything
COPY src/Project .

# Build and publish a release
RUN dotnet publish ./Project.csproj -c Release -o /app

# Build runtime image
FROM mcr.microsoft.com/dotnet/aspnet:7.0
WORKDIR /app
COPY --from=build-env /app .
EXPOSE 80
ENTRYPOINT ["./Project"]

grimurd on Oct 5, 2023