testcontainers-go: [Bug]: intermittent "port not found" error preventing Kafka and generic containers from starting

Testcontainers version

0.32.0

Using the latest Testcontainers version?

Yes

Host OS

MacOS

Host arch

ARM

Go version

1.21.12

Docker version

Client:
 Version:           27.1.1
 API version:       1.46
 Go version:        go1.21.12
 Git commit:        6312585
 Built:             Tue Jul 23 19:54:12 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.33.0 (160616)
 Engine:
  Version:          27.1.1
  API version:      1.46 (minimum version 1.24)
  Go version:       go1.21.12
  Git commit:       cc13f95
  Built:            Tue Jul 23 19:57:14 2024
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.7.19
  GitCommit:        2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc:
  Version:          1.7.19
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Version:    27.1.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.1-desktop.1
    Path:     /Users/aakso/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.1-desktop.1
    Path:     /Users/aakso/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.34
    Path:     /Users/aakso/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.14
    Path:     /Users/aakso/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/aakso/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     /Users/aakso/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/aakso/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     /Users/aakso/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/aakso/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.11.0
    Path:     /Users/aakso/.docker/cli-plugins/docker-scout

Server:
 Containers: 63
  Running: 28
  Paused: 0
  Stopped: 35
 Images: 267
 Server Version: 27.1.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.0-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 31.3GiB
 Name: docker-desktop
 ID: 02318c78-1ce1-45a9-85fe-81f705b23f70
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 289
  Goroutines: 274
  System Time: 2024-07-26T11:25:07.547170422Z
  EventsListeners: 16
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/aakso/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: true
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 192.168.128.0/17, Size: 24

WARNING: daemon is not using the default seccomp profile

What happened?

related to #2543

Unfortunately, the “port not found” error still seems to be present with v0.32.0.

I see it mainly with the Kafka module but also occasionally with testcontainers.GenericContainer too.

The code snippet I’m using the start the Kafka test container:

	container, err := kafka.Run(context.Background(), "confluentinc/confluent-local:7.5.0",
		tckafka.WithClusterID("test-kafka"),
		testcontainers.WithLogger(testcontainers.TestLogger(t)),
	)
	require.NoError(t, err, "RunContainer")

So pretty standard.

Relevant log output

## Using Kafka module

2024/07/26 13:34:14 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 27.1.1
  API Version: 1.46
  Operating System: Docker Desktop
  Total Memory: 32046 MB
  Testcontainers for Go Version: v0.32.0
  Resolved Docker Host: unix:///var/run/docker.sock
  Resolved Docker Socket Path: /var/run/docker.sock
  Test SessionID: fdf2809a5158c9725548a7f2151f954e32b0a25616f182e596f60b7426bfae62
  Test ProcessID: 3704bc3c-2bdf-4232-a3fb-e968a0995440
    lifecycle.go:62: 🐳 Creating container for image testcontainers/ryuk:0.7.0
    lifecycle.go:68: ✅ Container created: bffb711625a3
    lifecycle.go:74: 🐳 Starting container: bffb711625a3
    lifecycle.go:80: ✅ Container started: bffb711625a3
    lifecycle.go:263: ⏳ Waiting for container id bffb711625a3 image: testcontainers/ryuk:0.7.0. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms}
    lifecycle.go:86: 🔔 Container is ready: bffb711625a3
    lifecycle.go:62: 🐳 Creating container for image confluentinc/confluent-local:7.5.0
    lifecycle.go:68: ✅ Container created: a87a46618629
    lifecycle.go:74: 🐳 Starting container: a87a46618629
    lifecycle.go:80: ✅ Container started: a87a46618629
    lifecycle.go:333: container logs (port not found
        context deadline exceeded):
        
    kafka_test.go:22: 
                Error Trace:    /Users/aakso/git/upcloud/REDACTED/kafka_test.go:22
                Error:          Received unexpected error:
                                failed to start container: port not found
                                context deadline exceeded
                Test:           REDACTED
                Messages:       RunContainer

## Using GenericContainer
    daemon_internal_test.go:57: 
        	Error Trace:	/Users/aakso/git/REDACTED/daemon_internal_test.go:57
        	Error:      	Received unexpected error:
        	            	port not found: creating reaper failed: failed to create container
        	Test:       	REDACTED
        	Messages:   	testcontainers.GenericContainer

Additional information

I’m able to mitigate the error for Kafka module by wrapping the PostStarts lifecycle hooks and retrying on error. In my case it’s the first hook that always fails. It seems to fail to c.MappedPort call. So far, a single retry has been enough to mitigate the problem.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 3
  • Comments: 25 (20 by maintainers)

Commits related to this issue

Most upvoted comments

Hi @mdelapenya,

Answering your question - please check the note at the end of README (I just added this note, it repeats the same as described in issue https://github.com/testcontainers/testcontainers-go/issues/2670). It can take up to 20 retries for me to reproduce this issue.

In my case (I don’t have hardware where I can run Windows 11 with 16 GB RAM + WSL + Docker Desktop) I use VMware Workstation VM (with nested virtualization enabled). This can be the root cause my Docker is slower and it is easier (takes less number of tries) to reproduce https://github.com/testcontainers/testcontainers-go/issues/2670.

My colleague (who faced https://github.com/testcontainers/testcontainers-go/issues/2670 immediately when he tried similar code) uses real hardware with Windows 11 + Docker Desktop (no WSL).

Updates:

  1. Running multiple (10 by default) Kafka containers in parallel increased probability of this issue - refer to https://github.com/mabrarov/testcontainers-go-kafka-2670/pull/1.
  2. I run the same test on Linux against local Docker Engine (VMware Workstation VM with Rocky Linux 9.5 and Docker Engine 28.1.1) and failed to reproduce this issue.
  3. I run the same test against remote Docker Engine running on Linux (VMware Workstation VM with Rocky Linux 9.5 and Docker Engine 28.1.1) and failed to reproduce this issue.
  4. Workaround for this issue (applicable only for github.com/testcontainers/testcontainers-go/modules/kafka@v0.32.0 and previous versions, since github.com/testcontainers/testcontainers-go/modules/kafka@v0.33.0 this issue is not actual, but another issue takes place): https://github.com/mabrarov/testcontainers-go-kafka-2670/pull/2

I suspect that delay b/w completion of Docker container creation and port mapping happening is larger if Docker Desktop is used on non Linux OS (it looks like @aakso - reporter of this issue - used Docker Desktop on Mac), because in that case Docker Desktop uses virtualization. It makes this issue probability larger when Docker Desktop is used and smaller if Docker Engine (Docker CE) is used.

Thank you.

Hi,

Here is repro repo for this issue (item 1 from the list in my previous comment): https://github.com/mabrarov/testcontainers-go-kafka-2670.

Please note that issue happens intermittently (race condition nature) so multiple (5-20) tries can be required.

Thank you.

@stevenh I tested with pseudo module version of v0.32.1-0.20240814110719-4545c292e7b8 and it seems the issue is resolved.

I tried with few batches of 10 and with v0.32.0, about 50% of container starts failed. With the new patch, I don’t see any failures.

Same issue, solved with:

// HOTFIX: testcontainers-go v0.32.0
// https://github.com/testcontainers/testcontainers-go/issues/2670

type hotfix2670 struct{}

func (h hotfix2670) Customize(req *testcontainers.GenericContainerRequest) error {
	originalHook := req.LifecycleHooks[0].PostStarts[0]
	req.LifecycleHooks[0].PostStarts[0] = func(ctx context.Context, container testcontainers.Container) error {
		var err error
		for retry := 0; retry < 10; retry++ {
			err = originalHook(ctx, container)
			if err == nil {
				break
			}
			time.Sleep(time.Second)
		}
		return err
	}
	return nil
}

func setupKafka(t *testing.T, ctx context.Context) *kafka.KafkaContainer {
	t.Helper()

	testcontainers.Logger = log.New(io.Discard, "", 0)
	kafkaContainer, err := kafka.Run(ctx, "confluentinc/confluent-local:7.5.0",
		kafka.WithClusterID("test-cluster"),
		hotfix2670{},
	)
	require.NoError(t, err)

	t.Cleanup(func() { require.NoError(t, kafkaContainer.Terminate(ctx)) })
	return kafkaContainer
}

Similar issue can be actual in other parts of Testcontainers for Go where testcontainers.Container.MappedPort is used after start of container without loop waiting for port mapping to happen.

Example where wait for port mapping is required but is missing:

1. func grafanalgtm.Run(context.Context, string, ...testcontainers.ContainerCustomizer) (*grafanalgtm.GrafanaLGTMContainer, error)
  1. func (*grafanalgtm.GrafanaLGTMContainer) grafanalgtm.OtlpHttpEndpoint(context.Context) (string, error)
    1. func grafanalgtm.baseEndpoint(context.Context, *grafanalgtm.GrafanaLGTMContainer, nat.Port) (string, error)
      1. testcontainers.Container.MappedPort(context.Context, nat.Port) (nat.Port, error)

Note that probability of this issue in grafanalgtm.Run is small (I’d say zero, but it is wrong from math point of view) due to wait for specific log record which gives time for Docker to perform port mapping.

Here is similar case with small probability, because there is wait for specific log record which gives time for Docker to perform port mapping:

1. func dolt.Run(context.Context, string, ...testcontainers.ContainerCustomizer) (*dolt.DoltContainer, error)
  1. func (*dolt.DoltContainer) dolt.initialize(context.Context, bool) error
    1. func (*dolt.DoltContainer) dolt.initialConnectionString(context.Context) (string, error)
      1. testcontainers.Container.MappedPort(context.Context, nat.Port) (nat.Port, error) 

Example where wait for port mapping is not required, because there is preceding wait checking same port being accessible outside of container:

1. func couchbase.Run(context.Context, string, ...testcontainers.ContainerCustomizer) (*couchbase.CouchbaseContainer, error)
  1. func (*couchbase.CouchbaseContainer) couchbase.initCluster(context.Context) error
    1. func (*couchbase.CouchbaseContainer) couchbase.waitUntilNodeIsOnline(context.Context) error.
      1. func (*wait.HTTPStrategy) wait.WaitUntilReady(context.Context, StrategyTarget) error
        1. testcontainers.Container.MappedPort(context.Context, nat.Port) (nat.Port, error) <- called in wait loop with default 1 minute timeout (wait.defaultStartupTimeout) and checking couchbase.MGMT_PORT port
    2. func (*couchbase.CouchbaseContainer) couchbase.initializeIsEnterprise(context.Context)
      1. func (*couchbase.CouchbaseContainer) couchbase.doHTTPRequest(context.Context, string, string, string, map[string]string, bool) ([]byte, error)
        1. func (*couchbase.CouchbaseContainer) couchbase.getURL(context.Context, string, string) (string, error)`
          1. testcontainers.Container.MappedPort(context.Context, nat.Port) (nat.Port, error) <- no need to wait for couchbase.MGMT_PORT port mapping to happen, because it is done at call 1.1.1.1.1.

Thanks for the repo, I’m running a test with it now, couple of initial issues:

  1. You need to update the dependencies at was setup for a very old version of testcontainers-go
go get -u github.com/testcontainers/testcontainers-go/modules/kafka@main
  1. The test is using a feature from go 1.24 but was configured for 1.21
go mod edit -go=1.24
  1. Not an issue but you can simplify the test using testcontainers.CleanupContainer method e.g.
func TestKafkaContainerStart(t *testing.T) {
	image := *imageFlag
	t.Logf("starting container using image: %s", image)
	container, err := kafka.Run(t.Context(), image)
	testcontainers.CleanupContainer(t, container)
	require.NoError(t, err)
	t.Logf("successfully started container: %s", container.GetContainerID())
}

Hi,

I feel this issue root cause is similar to https://github.com/testcontainers/testcontainers-go/issues/2543 (https://github.com/testcontainers/testcontainers-go/issues/2543#issuecomment-2127084190).

https://github.com/testcontainers/testcontainers-go/pull/2606 has fixed caching issue, but it is not sufficient for https://github.com/testcontainers/testcontainers-go/issues/2543, because we still need to wait for port mapping to happen.

https://github.com/testcontainers/testcontainers-go/pull/2696 doesn’t seem to fix https://github.com/testcontainers/testcontainers-go/issues/2543 or/and https://github.com/testcontainers/testcontainers-go/issues/2670, because https://github.com/testcontainers/testcontainers-go/pull/2696 introduced another issue.

Here is a workaround I’m testing now (github.com/testcontainers/testcontainers-go@v0.37.0 + github.com/testcontainers/testcontainers-go/modules/kafka@v0.32.0)

	const port = 9093
	const mappedPortWaitDuration = time.Minute
	const mappedPortWaitInterval = time.Second
	containerPort, err := nat.NewPort("tcp", strconv.Itoa(port))
	...
	c, err := kafka.Run(ctx, "confluentinc/confluent-local:7.4.9", testcontainers.CustomizeRequestOption(
		func(req *testcontainers.GenericContainerRequest) error {
			req.LifecycleHooks = append([]testcontainers.ContainerLifecycleHooks{{
				PostStarts: []testcontainers.ContainerHook{
					func(ctx context.Context, c testcontainers.Container) error {
						portMapCtx, cancel := context.WithTimeout(ctx, mappedPortWaitDuration)
						defer cancel()
						_, portMapErr := c.MappedPort(portMapCtx, containerPort)
						for i := 0; portMapErr != nil; i++ {
							select {
							case <-portMapCtx.Done():
								return fmt.Errorf("mapped port: retries: %d, port: %s, last err: %w, ctx err: %w", i, containerPort, portMapErr, portMapCtx.Err())
							case <-time.After(mappedPortWaitInterval):
								_, portMapErr = c.MappedPort(portMapCtx, containerPort)
							}
						}
						return nil
					},
				},
			}}, req.LifecycleHooks...)
			return nil
		},
	))

Thank you.