testcontainers-go: [Bug]: Sporadically fails to create reaper in GitHub Actions

Testcontainers version

v0.27.0

Using the latest Testcontainers version?

Yes

Host OS

Ubuntu 22.04.3 LTS

Host arch

amd64

Go version

1.20

Docker version

GitHub Actions 🤷🏼‍♂️

Docker info

GitHub Actions 🤷🏼‍♂️

What happened?

This does not seem to happen when running locally but occasionally my tests fail in GitHub Actions with the error creating reaper failed.

Relevant log output

2024/01/28 17:56:02 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 24.0.7
  API Version: 1.43
  Operating System: Ubuntu 22.04.3 LTS
  Total Memory: 6921 MB
  Resolved Docker Host: unix:///var/run/docker.sock
  Resolved Docker Socket Path: /var/run/docker.sock
  Test SessionID: 1db7667978fa78cee3873c39e54eaa6e9e4e8bd9b8a5025ceb0b2b7c20578725
  Test ProcessID: c0f2e63c-5ad4-4ae2-a60d-de0250ff089f
2024/01/28 17:56:02 🔥 Reaper obtained from Docker for this test session 24e370b447d41033419fea8aefd579d45338a16882557d81a9ee8e0df38cfaad
2024/01/28 17:56:02 port not found: creating reaper failed: failed to create container

Additional information

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 16
  • Comments: 49 (23 by maintainers)

Commits related to this issue

Most upvoted comments

I’ve seen this quite frequently in Jenkins when running a large number of tests. Disabling the reaper with TESTCONTAINERS_RYUK_DISABLED fixed the issue for me.

If you’re in an ephemeral environment enabling TESTCONTAINERS_RYUK_DISABLED should be a fine workaround without any negative consequences.

Did you manage to solve this issue? I’m facing an issue with reaper failing with the following error… it only happens on GHA:

2024/02/13 14:47:24 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 25.0.3
  API Version: 1.43
  Operating System: Alpine Linux v3.19 (containerized)
  Total Memory: 31291 MB
  Resolved Docker Host: unix:///run/docker.sock
  Resolved Docker Socket Path: /run/docker.sock
  Test SessionID: c00c76c83c7180c78797a94d2146b5216f91200d033f9eb87efd620b53b27115
  Test ProcessID: e60189dd-ddfc-4b4c-98ab-1[70](xxx)99ea7a937
2024/02/13 14:47:24 🐳 Creating container for image testcontainers/ryuk:0.6.0
    internal_handler_test.go:283: failed to start container: look up reaper container returned nil although creation failed due to name conflict: creating reaper failed: failed to create container

not sure why this was closed, still experience it with the latest 0.33 release any way to solve this?

Yep seeing here too @mdelapenya can you reopen this, requires the reaper fixes.

Ok. Will try to make a public example to showcase the bug before reporting an issue. Or at least test with default github runner and not self hosted runner

@mdelapenya I still see this issue on the latest v0.32.0 version.

2024/08/02 07:17:05 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 27.1.1
  API Version: 1.46
  Operating System: Alpine Linux v3.20 (containerized)
  Total Memory: 15633 MB
  Testcontainers for Go Version: v0.32.0
  Resolved Docker Host: unix:///var/run/docker.sock
  Resolved Docker Socket Path: /var/run/docker.sock
  Test SessionID: a5b705ed65f0[28](https://github.com/***/actions/runs/***?pr=13#step:7:29)356e9eea0744e7f92be030ccf0cb0b4a92ed3454a01776d488
  Test ProcessID: fe16248a-b0f0-4768-8477-ee0505fe69d8
{"level":"debug","message":"Failed to get image auth for https://index.docker.io/v1/. Setting empty credentials for the image: testcontainers/ryuk:0.7.0. Error is:open /home/runner/.docker/config.json: no such file or directory"}
{"level":"debug","message":"🐳 Creating container for image testcontainers/ryuk:0.7.0"}
------------------------------
[BeforeSuite] [PANICKED] [5.698 seconds]
[BeforeSuite] 
/home/runner/_work/***/internal/app/app_test.go:56

  [PANICKED] Test Panicked
  In [BeforeSuite] at: /home/runner/_work/***/internal/testcontainers/db.go:44 @ 08/02/24 07:17:11.[37](https://github.com/***/actions/runs/***?pr=13#step:7:38)1

  look up reaper container returned nil although creation failed due to name conflict: creating reaper failed: failed to create container

I have 2 tests running in parallel and one of them failing in 90% of the cases.

Adding a data point here: we never hit this until upgrading testcontainers-go from 0.19 to 0.28.
Soon after, starting hitting this issue fairly regularly (I would estimate about 1 out of 4 jobs fail due to either

look up reaper container returned nil although creation failed due to name conflict

or

connecting to reaper failed: failed to create container

Sounds good to me 😃 In that case, make sure you call testcontainers.TerminateContainer(ctr) for every container you spawn. In the library we ensure that while building it, and we do have two pipelines: with and without ryuk to verify all the tests are cleaning up what it creates.

Is this because the CI is owned by other team?

Yes. We’ll probably ask them to expose a variable so we know we’re in CI, so we can add this, and any future “hack” without pinging them constantly.

everybody will have Ryuk disabled by default

Currently, we prefer manually deleting our local containers than sporadic failures in the pipeline. As our CI/CD creates ephemeral pods for tests, they’re gonna be destroyed either way.

@axilis-marko the env variable must be set at the terminal/shell you use to run the tests, as documented here: https://golang.testcontainers.org/features/garbage_collector/#ryuk

Ah it was autoclosed by merging #2648

You can terminate the containers directly and no longer need the reaper.

@tbrown1979 please remember that the reaper also removes built images, volumes and networks, so if you disable it please carefully remove all those Docker resources.

In any case, we have released a new version of Ryuk. As @stevenh pointed out in https://github.com/testcontainers/testcontainers-go/issues/2172#issuecomment-2226558170, you could give it a try like that.

I added SkipReaper: true so that the ryuk reaper does not start up

For the record, SkipReaper is deprecated in newer versions and will eventually disappear from the Request struct (see v1 branch: https://github.com/testcontainers/testcontainers-go/blob/v1/request.go#L30)

If you want to disable Ryuk, please use the properties file here: https://golang.testcontainers.org/features/configuration/#customizing-ryuk-the-resource-reaper

@mdelapenya Just checked on Windows 11 with WSL2 docker. Test container version: v0.31.0. Had one error: “port not found: creating reaper failed: failed to create container”.

I am experiencing this exact same issue in gitlab runners when running a few different jobs that are all using testcontainers. 4/5 of my jobs always pass and one job (which runs tests in parallel) always fails when the other 4 are running but passes when I retry it (and it is running by itself).