testcontainers-go: [Bug]: intermittent "port not found" error preventing Kafka and generic containers from starting
Testcontainers version
0.32.0
Using the latest Testcontainers version?
Yes
Host OS
MacOS
Host arch
ARM
Go version
1.21.12
Docker version
Client:
Version: 27.1.1
API version: 1.46
Go version: go1.21.12
Git commit: 6312585
Built: Tue Jul 23 19:54:12 2024
OS/Arch: darwin/arm64
Context: desktop-linux
Server: Docker Desktop 4.33.0 (160616)
Engine:
Version: 27.1.1
API version: 1.46 (minimum version 1.24)
Go version: go1.21.12
Git commit: cc13f95
Built: Tue Jul 23 19:57:14 2024
OS/Arch: linux/arm64
Experimental: true
containerd:
Version: 1.7.19
GitCommit: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
runc:
Version: 1.7.19
GitCommit: v1.1.13-0-g58aa920
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Docker info
Client:
Version: 27.1.1
Context: desktop-linux
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.16.1-desktop.1
Path: /Users/aakso/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.29.1-desktop.1
Path: /Users/aakso/.docker/cli-plugins/docker-compose
debug: Get a shell into any image or container (Docker Inc.)
Version: 0.0.34
Path: /Users/aakso/.docker/cli-plugins/docker-debug
desktop: Docker Desktop commands (Alpha) (Docker Inc.)
Version: v0.0.14
Path: /Users/aakso/.docker/cli-plugins/docker-desktop
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.2
Path: /Users/aakso/.docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.25
Path: /Users/aakso/.docker/cli-plugins/docker-extension
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: v1.0.5
Path: /Users/aakso/.docker/cli-plugins/docker-feedback
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.3.0
Path: /Users/aakso/.docker/cli-plugins/docker-init
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /Users/aakso/.docker/cli-plugins/docker-sbom
scout: Docker Scout (Docker Inc.)
Version: v1.11.0
Path: /Users/aakso/.docker/cli-plugins/docker-scout
Server:
Containers: 63
Running: 28
Paused: 0
Stopped: 35
Images: 267
Server Version: 27.1.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
runc version: v1.1.13-0-g58aa920
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 6.10.0-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 10
Total Memory: 31.3GiB
Name: docker-desktop
ID: 02318c78-1ce1-45a9-85fe-81f705b23f70
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 289
Goroutines: 274
System Time: 2024-07-26T11:25:07.547170422Z
EventsListeners: 16
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Labels:
com.docker.desktop.address=unix:///Users/aakso/Library/Containers/com.docker.docker/Data/docker-cli.sock
Experimental: true
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false
Default Address Pools:
Base: 192.168.128.0/17, Size: 24
WARNING: daemon is not using the default seccomp profile
What happened?
related to #2543
Unfortunately, the “port not found” error still seems to be present with v0.32.0.
I see it mainly with the Kafka module but also occasionally with testcontainers.GenericContainer too.
The code snippet I’m using the start the Kafka test container:
container, err := kafka.Run(context.Background(), "confluentinc/confluent-local:7.5.0",
tckafka.WithClusterID("test-kafka"),
testcontainers.WithLogger(testcontainers.TestLogger(t)),
)
require.NoError(t, err, "RunContainer")
So pretty standard.
Relevant log output
## Using Kafka module
2024/07/26 13:34:14 github.com/testcontainers/testcontainers-go - Connected to docker:
Server Version: 27.1.1
API Version: 1.46
Operating System: Docker Desktop
Total Memory: 32046 MB
Testcontainers for Go Version: v0.32.0
Resolved Docker Host: unix:///var/run/docker.sock
Resolved Docker Socket Path: /var/run/docker.sock
Test SessionID: fdf2809a5158c9725548a7f2151f954e32b0a25616f182e596f60b7426bfae62
Test ProcessID: 3704bc3c-2bdf-4232-a3fb-e968a0995440
lifecycle.go:62: 🐳 Creating container for image testcontainers/ryuk:0.7.0
lifecycle.go:68: ✅ Container created: bffb711625a3
lifecycle.go:74: 🐳 Starting container: bffb711625a3
lifecycle.go:80: ✅ Container started: bffb711625a3
lifecycle.go:263: ⏳ Waiting for container id bffb711625a3 image: testcontainers/ryuk:0.7.0. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms}
lifecycle.go:86: 🔔 Container is ready: bffb711625a3
lifecycle.go:62: 🐳 Creating container for image confluentinc/confluent-local:7.5.0
lifecycle.go:68: ✅ Container created: a87a46618629
lifecycle.go:74: 🐳 Starting container: a87a46618629
lifecycle.go:80: ✅ Container started: a87a46618629
lifecycle.go:333: container logs (port not found
context deadline exceeded):
kafka_test.go:22:
Error Trace: /Users/aakso/git/upcloud/REDACTED/kafka_test.go:22
Error: Received unexpected error:
failed to start container: port not found
context deadline exceeded
Test: REDACTED
Messages: RunContainer
## Using GenericContainer
daemon_internal_test.go:57:
Error Trace: /Users/aakso/git/REDACTED/daemon_internal_test.go:57
Error: Received unexpected error:
port not found: creating reaper failed: failed to create container
Test: REDACTED
Messages: testcontainers.GenericContainer
Additional information
I’m able to mitigate the error for Kafka module by wrapping the PostStarts lifecycle hooks and retrying on error. In my case it’s the first hook that always fails. It seems to fail to c.MappedPort call. So far, a single retry has been enough to mitigate the problem.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 3
- Comments: 25 (20 by maintainers)
Commits related to this issue
- Workaround for https://github.com/testcontainers/testcontainers-go/issues/2670. — committed to mabrarov/testcontainers-go-kafka-2748 by mabrarov 2 years ago
- Workaround for https://github.com/testcontainers/testcontainers-go/issues/2670 — committed to mabrarov/testcontainers-go-kafka-2670 by mabrarov 2 years ago
- Workaround for https://github.com/testcontainers/testcontainers-go/issues/2670. — committed to mabrarov/testcontainers-go-kafka-2748 by mabrarov 2 years ago
Hi @mdelapenya,
Answering your question - please check the note at the end of README (I just added this note, it repeats the same as described in issue https://github.com/testcontainers/testcontainers-go/issues/2670). It can take up to 20 retries for me to reproduce this issue.
In my case (I don’t have hardware where I can run Windows 11 with 16 GB RAM + WSL + Docker Desktop) I use VMware Workstation VM (with nested virtualization enabled). This can be the root cause my Docker is slower and it is easier (takes less number of tries) to reproduce https://github.com/testcontainers/testcontainers-go/issues/2670.
My colleague (who faced https://github.com/testcontainers/testcontainers-go/issues/2670 immediately when he tried similar code) uses real hardware with Windows 11 + Docker Desktop (no WSL).
Updates:
I suspect that delay b/w completion of Docker container creation and port mapping happening is larger if Docker Desktop is used on non Linux OS (it looks like @aakso - reporter of this issue - used Docker Desktop on Mac), because in that case Docker Desktop uses virtualization. It makes this issue probability larger when Docker Desktop is used and smaller if Docker Engine (Docker CE) is used.
Thank you.
Hi,
Here is repro repo for this issue (item 1 from the list in my previous comment): https://github.com/mabrarov/testcontainers-go-kafka-2670.
Please note that issue happens intermittently (race condition nature) so multiple (5-20) tries can be required.
Thank you.
@stevenh I tested with pseudo module version of
v0.32.1-0.20240814110719-4545c292e7b8and it seems the issue is resolved.I tried with few batches of 10 and with v0.32.0, about 50% of container starts failed. With the new patch, I don’t see any failures.
Same issue, solved with:
Similar issue can be actual in other parts of Testcontainers for Go where
testcontainers.Container.MappedPortis used after start of container without loop waiting for port mapping to happen.Example where wait for port mapping is required but is missing:
Note that probability of this issue in
grafanalgtm.Runis small (I’d say zero, but it is wrong from math point of view) due to wait for specific log record which gives time for Docker to perform port mapping.Here is similar case with small probability, because there is wait for specific log record which gives time for Docker to perform port mapping:
Example where wait for port mapping is not required, because there is preceding wait checking same port being accessible outside of container:
Thanks for the repo, I’m running a test with it now, couple of initial issues:
testcontainers.CleanupContainermethod e.g.Hi,
I feel this issue root cause is similar to https://github.com/testcontainers/testcontainers-go/issues/2543 (https://github.com/testcontainers/testcontainers-go/issues/2543#issuecomment-2127084190).
https://github.com/testcontainers/testcontainers-go/pull/2606 has fixed caching issue, but it is not sufficient for https://github.com/testcontainers/testcontainers-go/issues/2543, because we still need to wait for port mapping to happen.
https://github.com/testcontainers/testcontainers-go/pull/2696 doesn’t seem to fix https://github.com/testcontainers/testcontainers-go/issues/2543 or/and https://github.com/testcontainers/testcontainers-go/issues/2670, because https://github.com/testcontainers/testcontainers-go/pull/2696 introduced another issue.
Here is a workaround I’m testing now (github.com/testcontainers/testcontainers-go@v0.37.0 + github.com/testcontainers/testcontainers-go/modules/kafka@v0.32.0)
Thank you.
Can you check if https://github.com/testcontainers/testcontainers-go/pull/2696 fixes this?