testcontainers-dotnet: [Bug]: Cannot run kafka container
Testcontainers version
4.6.0
Using the latest Testcontainers version?
Yes
Host OS
macOS
Host arch
ARM
.NET version
9.0.203
Docker version
Client:
Version: 27.4.0
API version: 1.47
Go version: go1.22.10
Git commit: bde2b89
Built: Sat Dec 7 10:35:43 2024
OS/Arch: darwin/arm64
Context: desktop-linux
Server: Docker Desktop 4.37.1 (178610)
Engine:
Version: 27.4.0
API version: 1.47 (minimum version 1.24)
Go version: go1.22.10
Git commit: 92a8393
Built: Sat Dec 7 10:38:33 2024
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.7.21
GitCommit: 472731909fa34bd7bc9c087e4c27943f9835f111
runc:
Version: 1.1.13
GitCommit: v1.1.13-0-g58aa920
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Docker info
Client:
Version: 27.4.0
Context: desktop-linux
Debug Mode: false
Plugins:
ai: Ask Gordon - Docker Agent (Docker Inc.)
Version: v0.5.1
Path: /Users/george/.docker/cli-plugins/docker-ai
buildx: Docker Buildx (Docker Inc.)
Version: v0.19.2-desktop.1
Path: /Users/george/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.31.0-desktop.2
Path: /Users/george/.docker/cli-plugins/docker-compose
debug: Get a shell into any image or container (Docker Inc.)
Version: 0.0.37
Path: /Users/george/.docker/cli-plugins/docker-debug
desktop: Docker Desktop commands (Beta) (Docker Inc.)
Version: v0.1.0
Path: /Users/george/.docker/cli-plugins/docker-desktop
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.2
Path: /Users/george/.docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.27
Path: /Users/george/.docker/cli-plugins/docker-extension
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: v1.0.5
Path: /Users/george/.docker/cli-plugins/docker-feedback
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.4.0
Path: /Users/george/.docker/cli-plugins/docker-init
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /Users/george/.docker/cli-plugins/docker-sbom
scout: Docker Scout (Docker Inc.)
Version: v1.15.1
Path: /Users/george/.docker/cli-plugins/docker-scout
Server:
Containers: 8
Running: 6
Paused: 0
Stopped: 2
Images: 15
Server Version: 27.4.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
runc version: v1.1.13-0-g58aa920
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 6.10.14-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 3
Total Memory: 3.828GiB
Name: docker-desktop
ID: 39675418-d776-4f5a-8775-8a7ab970d5cf
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Labels:
com.docker.desktop.address=unix:///Users/george/Library/Containers/com.docker.docker/Data/docker-cli.sock
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false
What happened?
- .Kafka package uses outdated kafka version (not present in docker registry)
- when switching to latest, container failes to start zookeeper (can’t find script to start it)
Relevant log output
2025-06-24 20:34:36 /testcontainers.sh: line 5: zookeeper-server-start: command not found
2025-06-24 20:34:37 KAFKA_PROCESS_ROLES is required.
2025-06-24 20:34:37 Command [/usr/local/bin/dub ensure KAFKA_PROCESS_ROLES] FAILED !
2025-06-24 20:34:36 ===> User
2025-06-24 20:34:36 uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
2025-06-24 20:34:36 ===> Configuring ...
Additional information
No response
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 15
We somehow need to pass the platform information to this method:
https://github.com/testcontainers/testcontainers-dotnet/blob/63f67dc257809ba6d206f2cf8797955f09c54e0b/src/Testcontainers/Clients/TestcontainersClient.cs#L333-L337
and set the platform property here:
https://github.com/testcontainers/testcontainers-dotnet/blob/63f67dc257809ba6d206f2cf8797955f09c54e0b/src/Testcontainers/Clients/DockerImageOperations.cs#L60-L63
Unfortunately, I’m not yet sure how we can implement this properly. Because of the parameter modifier callback, which gives us access to the low-level Docker Engine APIs, we don’t have the platform information “available” while pulling.
Yes, please. That would be great and much appreciated! Just a heads-up. I’ll be on vacation until next week, so my responses might be a bit slow (be aware and choose the right repo when creating the PR, be default GH suggest the legacy and outdated repo).
For Docker.DotNet, we’ll probably need to check the HTTP response somewhere around here.
Thanks for checking! OC this is just a workaround and not a proper fix, but this was the information and feedback I was looking for.
Neither the Docker Engine API (runtime) nor Docker.DotNet pulls images when creating a container. Testcontainers pulls the images in advance (we pull them here if they don’t already exist).
Right now, we’re not passing the platform information when pulling the image. That’s basically the issue and something we need to improve.
My assumption was that if Rosetta (emulation) is enabled, the runtime would automatically handle it and pull the
amd64image (ifarm64is not available).Ah, I see. That’s unfortunate. Since the HTTP status code is 200, no exception is thrown. Docker.DotNet probably should throw one in this case.
I am not quite familiar with the internals of the library, but it seems for me that there are a couple of bug (?) present
first (maybe not that important to this exact issue, but nevertheless spotted) the Docker.DotNet library only checks if Name is present when creating the container to build a query string, while Platform value should also go in there
https://github.com/testcontainers/Docker.DotNet/blob/bcf0c7317b6eb7e28fa9dd0afdbbcddc9d5e1f33/src/Docker.DotNet/Endpoints/ContainerOperations.cs#L48-L51
it is not that relevant to the case because (according to docker API docs) when creating the container, docker looks up only the local images with matching platform (and falls back to virtualisation if image with host arch not found):
https://docs.docker.com/reference/api/engine/version/v1.50/#tag/Container/operation/ContainerCreate
And the second bug is probably that the is no proper handling of pull failures (i was incorrect about pull step not being present, looked closely and found it). because docker API returns this when pulling image :
code 200 though))
Not sure about that, I mean clearly in the permalink you site there is no platform passed, but I dont think that is the code being executed when I callContainer.StartAsync. I have a breakpoint at the line below (in a container client that being called there to be precise), and the exception occurs inside that call to the client. So i believe that there was no separate pulling step.https://github.com/testcontainers/testcontainers-dotnet/blob/63f67dc257809ba6d206f2cf8797955f09c54e0b/src/Testcontainers/Clients/DockerContainerOperations.cs#L225
(pulling image prior to running test container solves the problem though, but it is rather a hack then a solution)