testcontainers-go: [Bug]: Unable to connect to container

Testcontainers version

0.25.0

Using the latest Testcontainers version?

Yes

Host OS

Linux, MacOS

Host arch

x86, Arm

Go version

1.21

Docker version

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:28:49 2023
 OS/Arch:           darwin/arm64
 Context:           default

Docker info

Client:
 Version:    24.0.6
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.5
    Path:     /Users/itziklavon/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.22.0-desktop.2
    Path:     /Users/itziklavon/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/itziklavon/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/itziklavon/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.8
    Path:     /Users/itziklavon/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/itziklavon/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/itziklavon/.docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.0.7
    Path:     /Users/itziklavon/.docker/cli-plugins/docker-scout

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 83
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.4.16-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 11
 Total Memory: 7.765GiB
 Name: docker-desktop
 ID: 0c850adf-94c4-45e6-97d4-fd7d753d0141
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

What happened?

After upgrading to version 0.25.0 this error occurs:

[2023-10-18T20:52:10.894Z] {"level":"error","ts":"2023-10-18 20:52:09","caller":"component/mysql_init_ms.go:33","F":"bitbucket.org/gamescale-rnd/go_tests_utils/component.StartMSSql","msg":"Error response from daemon: No such container: d11e9589adc269f7720f307f7506385110c0e9ab5580a577c3c1f2568fe23333: creating reaper failed: failed to create container"}
[2023-10-18T20:52:10.894Z] panic: runtime error: invalid memory address or nil pointer dereference [recovered]
[2023-10-18T20:52:10.894Z] 	panic: runtime error: invalid memory address or nil pointer dereference
[2023-10-18T20:52:10.894Z] [signal SIGSEGV: segmentation violation code=0x1 addr=0x68 pc=0x1000581]

this happens when i have multiple tests starting multiple containers, when i run single test(2-3 containers), no issues when reverting back to version 0.23.0, all seems to be working well as expected

Relevant log output

[2023-10-18T20:52:04.113Z] ok  	withdrawal-go/tests/component_test_expanded_player_info	28.668s	coverage: [no statements]
[2023-10-18T20:52:04.113Z] ok  	withdrawal-go/tests/component_test_high_risk	27.613s	coverage: [no statements]
[2023-10-18T20:52:10.894Z] 2023/10/18 20:52:02 github.com/testcontainers/testcontainers-go - Connected to docker: 
[2023-10-18T20:52:10.894Z]   Server Version: 20.10.12
[2023-10-18T20:52:10.894Z]   API Version: 1.41
[2023-10-18T20:52:10.894Z]   Operating System: Ubuntu 20.04.4 LTS
[2023-10-18T20:52:10.894Z]   Total Memory: 7700 MB
[2023-10-18T20:52:10.894Z]   Resolved Docker Host: unix:///var/run/docker.sock
[2023-10-18T20:52:10.894Z]   Resolved Docker Socket Path: /var/run/docker.sock
[2023-10-18T20:52:10.894Z]   Test SessionID: 27f2a3d02835da8cca598bb5a0551abf0074cff82ea48d2670bc4c0829619808
[2023-10-18T20:52:10.894Z]   Test ProcessID: 3e61313b-c7af-465e-b3e9-d6841747cf04
[2023-10-18T20:52:10.894Z] {"level":"error","ts":"2023-10-18 20:52:09","caller":"component/mysql_init_ms.go:33","F":"bitbucket.org/gamescale-rnd/go_tests_utils/component.StartMSSql","msg":"Error response from daemon: No such container: d11e9589adc269f7720f307f7506385110c0e9ab5580a577c3c1f2568fe23333: creating reaper failed: failed to create container"}
[2023-10-18T20:52:10.894Z] panic: runtime error: invalid memory address or nil pointer dereference [recovered]
[2023-10-18T20:52:10.894Z] 	panic: runtime error: invalid memory address or nil pointer dereference
[2023-10-18T20:52:10.894Z] [signal SIGSEGV: segmentation violation code=0x1 addr=0x68 pc=0x1000581]
[2023-10-18T20:52:10.894Z] 
[2023-10-18T20:52:10.894Z] goroutine 1 [running]:
[2023-10-18T20:52:10.894Z] withdrawal-go/tests/component_test_irregular_play.TestMain.func1()
[2023-10-18T20:52:10.894Z] 	/var/lib/jenkins/workspace/nightly_go_withdrawal/tests/component_test_irregular_play/irregular_play_component_test.go:112 +0x57
[2023-10-18T20:52:10.894Z] panic({0x11e7780?, 0x1f66e30?})
[2023-10-18T20:52:10.894Z] 	/usr/local/go/src/runtime/panic.go:914 +0x21f
[2023-10-18T20:52:10.894Z] bitbucket.org/gamescale-rnd/go_tests_utils/component.StartMSSql({0x139d80d, 0xb}, {0x139d80d, 0xb}, {0x164e4c8, 0xc00006e6d8}, {0x164e4c8, 0xc00006e6e0})
[2023-10-18T20:52:10.894Z] 	/go/pkg/mod/bitbucket.org/gamescale-rnd/go_tests_utils@v1.1.87/component/mysql_init_ms.go:35 +0x321
[2023-10-18T20:52:10.894Z] withdrawal-go/tests/component_test_irregular_play.TestMain(0x1394c0a?)
[2023-10-18T20:52:10.894Z] 	/var/lib/jenkins/workspace/nightly_go_withdrawal/tests/component_test_irregular_play/irregular_play_component_test.go:118 +0x22e
[2023-10-18T20:52:10.894Z] main.main()
[2023-10-18T20:52:10.894Z] 	_testmain.go:183 +0x1e6
[2023-10-18T20:52:10.894Z] FAIL	withdrawal-go/tests/component_test_irregular_play	7.585s
[2023-10-18T20:52:10.894Z] 2023/10/18 20:52:02 github.com/testcontainers/testcontainers-go - Connected to docker: 
[2023-10-18T20:52:10.894Z]   Server Version: 20.10.12
[2023-10-18T20:52:10.894Z]   API Version: 1.41
[2023-10-18T20:52:10.894Z]   Operating System: Ubuntu 20.04.4 LTS
[2023-10-18T20:52:10.894Z]   Total Memory: 7700 MB
[2023-10-18T20:52:10.894Z]   Resolved Docker Host: unix:///var/run/docker.sock
[2023-10-18T20:52:10.894Z]   Resolved Docker Socket Path: /var/run/docker.sock
[2023-10-18T20:52:10.894Z]   Test SessionID: 27f2a3d02835da8cca598bb5a0551abf0074cff82ea48d2670bc4c0829619808
[2023-10-18T20:52:10.895Z]   Test ProcessID: 6b064694-4144-441c-8cdc-eae0a06507a8
[2023-10-18T20:52:10.895Z] {"level":"error","ts":"2023-10-18 20:52:09","caller":"component/mysql_init_ms.go:33","F":"bitbucket.org/gamescale-rnd/go_tests_utils/component.StartMSSql","msg":"Error response from daemon: No such container: d11e9589adc269f7720f307f7506385110c0e9ab5580a577c3c1f2568fe23333: creating reaper failed: failed to create container"}
[2023-10-18T20:52:10.895Z] panic: runtime error: invalid memory address or nil pointer dereference [recovered]
[2023-10-18T20:52:10.895Z] 	panic: runtime error: invalid memory address or nil pointer dereference
[2023-10-18T20:52:10.895Z] [signal SIGSEGV: segmentation violation code=0x1 addr=0x68 pc=0x1050b01]



### Additional information

This log is taken from jenkins job (Ubuntu)
getting same error when running locally (MacOS)

About this issue

Original URL
State: closed
Created 8 months ago
Comments: 30

Most upvoted comments

Hello @mdelapenya We see same error for 0.25.0 and 0.26.0 versions.

I tried using t.Setenv("TESTCONTAINERS_RYUK_RECONNECTION_TIMEOUT_DEFAULT", "10m") and t.Setenv("TESTCONTAINERS_RYUK_CONNECTION_TIMEOUT_DEFAULT", "10m"), but it didn’t help.

Also i can confirm - v0.23.0 works correctly.

We get error for 2 different containers. Requests:

	imageName := "redis:5.0"
	port := "6379/tcp"
	req := testcontainers.ContainerRequest{
		Name:         "redis",
		Image:        imageName,
		ExposedPorts: []string{port},
		SkipReaper:   true,
		AutoRemove:   true,
	}

	redis, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
		ContainerRequest: req,
		Started:          true,
	})

	req := testcontainers.ContainerRequest{
		Name:         "registry",
		Image:        registryImage,
		ExposedPorts: []string{registryPort},
		Env: map[string]string{
			"REGISTRY_HTTP_ADDR":                 "0.0.0.0:5443",
			"REGISTRY_HTTP_TLS_CERTIFICATE":      "/certs/cert.pem",
			"REGISTRY_HTTP_TLS_KEY":              "/certs/key.pem",
			"REGISTRY_AUTH":                      "token",
			"REGISTRY_AUTH_TOKEN_REALM":          fmt.Sprintf("%s/auth", authURL),
			"REGISTRY_AUTH_TOKEN_SERVICE":        "registry.docker.io",
			"REGISTRY_AUTH_TOKEN_ISSUER":         "Trivy auth server",
			"REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE": "/certs/cert.pem",
			"REGISTRY_AUTH_TOKEN_AUTOREDIRECT":   "false",
		},
		Mounts: testcontainers.Mounts(
			testcontainers.BindMount(filepath.Join(baseDir, "data", "certs"), "/certs"),
		),
		SkipReaper: true,
		AutoRemove: true,
		WaitingFor: wait.ForLog("listening on [::]:5443"),
	}

	registryC, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
		ContainerRequest: req,
		Started:          true,
	})

DmitriyLewen on Nov 1, 2023

Hi just encountered the issue again, even with timeout set to 2 minutes(tried also with 5 minutes, did not resolve, did not try more than that) currently reverted back to version 0.23.0

tried to upgrade to 0.26.0 also did not help

something broken after version 0.23.0 version 0.23.0 works well without any issue

itziklavon on Oct 29, 2023

@destel no worries at all… shit happens 🤷 and I’m more than happy that it resolved the error for you 👏

If everybody agree in this thread, I’m going to close this issue at the end of the day as resolved by #1904

mdelapenya on Dec 18, 2023

Somehow I confused myself and you. I did two builds one from the master and one from the #1094 branch. Master works just fine. Sorry for the confusion.

destel on Dec 18, 2023

I have the same issue running multiple tests, and creating multiple instances of the containers, locally works but fails on GitHub Actions, setting TESTCONTAINERS_RYUK_DISABLED: true solves it.

acevedomiguel on Dec 8, 2023

hello @mdelapenya Sorry for waiting.

I can reproduce error when using SkipReaper (if it make sense for you, because it is depriceted):

=== RUN   TestRegistry
    registry_test.go:124: 
                Error Trace:    /Users/work/work/aqua/trivy/integration/registry_test.go:124
                Error:          Received unexpected error:
                                Error response from daemon: No such container: 967ff0f05d434ce6c3aab36cbeec762da97179312089ccc0f3579a0e1c343798: creating reaper failed: failed to create container
                Test:           TestRegistry

But looks like with t.Setenv("TESTCONTAINERS_RYUK_DISABLED", "true") tests work stable.

DmitriyLewen on Dec 7, 2023

I have had a similar issue, though not exactly the same, figured I could just comment here with my test findings. As far as I can see https://github.com/testcontainers/testcontainers-go/issues/1671 is related to this issue and my issue.

I can open a separate issue if it helps.

When running on Github Actions I some times get this error with version v0.23.0, v0.25.0 and v0.26.0. There are 3 different packages in my repo that use testcontainers-go and it fails after the first package test is done. This issue only occurs when running on Github Actions, never locally on my arm64 Mac where I have been unable to reproduce the issue.

Error response from daemon: No such container: 98712a5e16b34f9494fd6cadfc24de6b5a25032294d690784d86016dd82a3132: creating reaper failed: failed to create container

Fetching the latest version from main with go get -u -t -x github.com/testcontainers/testcontainers-go@main I get the version v0.26.1-0.20231201205740-f36b5a41f090. The latest main version works any observed issues after running it multiple times, seems like https://github.com/testcontainers/testcontainers-go/pull/1904 fixed my issue 👍

mikalsande on Dec 2, 2023

Hello @mdelapenya I’ll try to find time for this next week.

DmitriyLewen on Dec 1, 2023

We just don’t need Ryuk in this scenario.

Got it, thanks! In any case, Ryuk will be there even you don’t want it, because it’s for the entire test session. Just want to match expectations about whether Ryuk is there or not handling the lifecycle of the containers. In this case, it will. Probably it’s not contributing to the root cause.

We do have this open PR about to be merged: https://github.com/testcontainers/testcontainers-go/pull/1904, which could be related if you are running multiple tests in multiple packages in parallel.

mdelapenya on Nov 7, 2023