testcontainers-dotnet: [Bug]: Cannot run kafka container

Testcontainers version

4.6.0

Using the latest Testcontainers version?

Yes

Host OS

macOS

Host arch

ARM

.NET version

9.0.203

Docker version

Client:
 Version:           27.4.0
 API version:       1.47
 Go version:        go1.22.10
 Git commit:        bde2b89
 Built:             Sat Dec  7 10:35:43 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.37.1 (178610)
 Engine:
  Version:          27.4.0
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.10
  Git commit:       92a8393
  Built:            Sat Dec  7 10:38:33 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.21
  GitCommit:        472731909fa34bd7bc9c087e4c27943f9835f111
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Version:    27.4.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.5.1
    Path:     /Users/george/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.2-desktop.1
    Path:     /Users/george/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.31.0-desktop.2
    Path:     /Users/george/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.37
    Path:     /Users/george/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Beta) (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/george/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/george/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     /Users/george/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/george/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/george/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/george/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.15.1
    Path:     /Users/george/.docker/cli-plugins/docker-scout

Server:
 Containers: 8
  Running: 6
  Paused: 0
  Stopped: 2
 Images: 15
 Server Version: 27.4.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.14-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 3
 Total Memory: 3.828GiB
 Name: docker-desktop
 ID: 39675418-d776-4f5a-8775-8a7ab970d5cf
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/george/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

What happened?

  1. .Kafka package uses outdated kafka version (not present in docker registry)
  2. when switching to latest, container failes to start zookeeper (can’t find script to start it)

Relevant log output

2025-06-24 20:34:36 /testcontainers.sh: line 5: zookeeper-server-start: command not found
2025-06-24 20:34:37 KAFKA_PROCESS_ROLES is required.
2025-06-24 20:34:37 Command [/usr/local/bin/dub ensure KAFKA_PROCESS_ROLES] FAILED !
2025-06-24 20:34:36 ===> User
2025-06-24 20:34:36 uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
2025-06-24 20:34:36 ===> Configuring ...

Additional information

No response

About this issue

Most upvoted comments

We somehow need to pass the platform information to this method:

https://github.com/testcontainers/testcontainers-dotnet/blob/63f67dc257809ba6d206f2cf8797955f09c54e0b/src/Testcontainers/Clients/TestcontainersClient.cs#L333-L337

and set the platform property here:

https://github.com/testcontainers/testcontainers-dotnet/blob/63f67dc257809ba6d206f2cf8797955f09c54e0b/src/Testcontainers/Clients/DockerImageOperations.cs#L60-L63

Unfortunately, I’m not yet sure how we can implement this properly. Because of the parameter modifier callback, which gives us access to the low-level Docker Engine APIs, we don’t have the platform information “available” while pulling.

seems to me that fixes in Docker.DotNet are rather simple, I wouldn’t mind looking into that, if you accept contributions 👀

Yes, please. That would be great and much appreciated! Just a heads-up. I’ll be on vacation until next week, so my responses might be a bit slow (be aware and choose the right repo when creating the PR, be default GH suggest the legacy and outdated repo).

For Docker.DotNet, we’ll probably need to check the HTTP response somewhere around here.

pulling image prior to running test container solves the problem though, but it is rather a hack then a solution

Thanks for checking! OC this is just a workaround and not a proper fix, but this was the information and feedback I was looking for.

i was incorrect about pull step not being present, looked closely and found it

Neither the Docker Engine API (runtime) nor Docker.DotNet pulls images when creating a container. Testcontainers pulls the images in advance (we pull them here if they don’t already exist).

Right now, we’re not passing the platform information when pulling the image. That’s basically the issue and something we need to improve.

My assumption was that if Rosetta (emulation) is enabled, the runtime would automatically handle it and pull the amd64 image (if arm64 is not available).

And the second bug is probably that the is no proper handling of pull failures

Ah, I see. That’s unfortunate. Since the HTTP status code is 200, no exception is thrown. Docker.DotNet probably should throw one in this case.

I am not quite familiar with the internals of the library, but it seems for me that there are a couple of bug (?) present

first (maybe not that important to this exact issue, but nevertheless spotted) the Docker.DotNet library only checks if Name is present when creating the container to build a query string, while Platform value should also go in there

https://github.com/testcontainers/Docker.DotNet/blob/bcf0c7317b6eb7e28fa9dd0afdbbcddc9d5e1f33/src/Docker.DotNet/Endpoints/ContainerOperations.cs#L48-L51

it is not that relevant to the case because (according to docker API docs) when creating the container, docker looks up only the local images with matching platform (and falls back to virtualisation if image with host arch not found):

https://docs.docker.com/reference/api/engine/version/v1.50/#tag/Container/operation/ContainerCreate

When specified, the daemon checks if the requested image is present in the local image cache with the given OS and Architecture, and otherwise returns a 404 status.

And the second bug is probably that the is no proper handling of pull failures (i was incorrect about pull step not being present, looked closely and found it). because docker API returns this when pulling image :

{"status":"Pulling from confluentinc/cp-kafka","id":"6.1.9"}
{"errorDetail":{"message":"no matching manifest for linux/arm64/v8 in the manifest list entries"},"error":"no matching manifest for linux/arm64/v8 in the manifest list entries"}

code 200 though))

But unfortunately, we’re not passing that option to the pull command.

Not sure about that, I mean clearly in the permalink you site there is no platform passed, but I dont think that is the code being executed when I call Container.StartAsync. I have a breakpoint at the line below (in a container client that being called there to be precise), and the exception occurs inside that call to the client. So i believe that there was no separate pulling step.

https://github.com/testcontainers/testcontainers-dotnet/blob/63f67dc257809ba6d206f2cf8797955f09c54e0b/src/Testcontainers/Clients/DockerContainerOperations.cs#L225

(pulling image prior to running test container solves the problem though, but it is rather a hack then a solution)