testcontainers-go: [Bug]: Sporadically fails to create reaper in GitHub Actions
Testcontainers version
v0.27.0
Using the latest Testcontainers version?
Yes
Host OS
Ubuntu 22.04.3 LTS
Host arch
amd64
Go version
1.20
Docker version
GitHub Actions 🤷🏼♂️
Docker info
GitHub Actions 🤷🏼♂️
What happened?
This does not seem to happen when running locally but occasionally my tests fail in GitHub Actions with the error creating reaper failed.
Relevant log output
2024/01/28 17:56:02 github.com/testcontainers/testcontainers-go - Connected to docker:
Server Version: 24.0.7
API Version: 1.43
Operating System: Ubuntu 22.04.3 LTS
Total Memory: 6921 MB
Resolved Docker Host: unix:///var/run/docker.sock
Resolved Docker Socket Path: /var/run/docker.sock
Test SessionID: 1db7667978fa78cee3873c39e54eaa6e9e4e8bd9b8a5025ceb0b2b7c20578725
Test ProcessID: c0f2e63c-5ad4-4ae2-a60d-de0250ff089f
2024/01/28 17:56:02 🔥 Reaper obtained from Docker for this test session 24e370b447d41033419fea8aefd579d45338a16882557d81a9ee8e0df38cfaad
2024/01/28 17:56:02 port not found: creating reaper failed: failed to create container
Additional information
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 16
- Comments: 49 (23 by maintainers)
Commits related to this issue
- fix(tests): update port usage for testcontainers Due to the need to update testcontainers lib also we needed to disable testcontainers reaper at Github actions as well as it seems related to https://... — committed to cartesi/rollups-node by fmoura 2 years ago
- fix(tests): disable testcontainers reaper at Github actions This issue seems related to https://github.com/testcontainers/testcontainers-go/issues/2172 — committed to cartesi/rollups-node by fmoura 2 years ago
- ci: disable reaper to avoid failure on testcontainers This issue seems related to https://github.com/testcontainers/testcontainers-go/issues/2172 — committed to cartesi/rollups-node by fmoura 2 years ago
- ci: disable reaper to avoid failure on testcontainers This issue seems related to https://github.com/testcontainers/testcontainers-go/issues/2172 — committed to cartesi/rollups-node by fmoura 2 years ago
I’ve seen this quite frequently in Jenkins when running a large number of tests. Disabling the reaper with
TESTCONTAINERS_RYUK_DISABLEDfixed the issue for me.If you’re in an ephemeral environment enabling
TESTCONTAINERS_RYUK_DISABLEDshould be a fine workaround without any negative consequences.Did you manage to solve this issue? I’m facing an issue with reaper failing with the following error… it only happens on GHA:
not sure why this was closed, still experience it with the latest 0.33 release any way to solve this?
Yep seeing here too @mdelapenya can you reopen this, requires the reaper fixes.
Ok. Will try to make a public example to showcase the bug before reporting an issue. Or at least test with default github runner and not self hosted runner
@mdelapenya I still see this issue on the latest v0.32.0 version.
I have 2 tests running in parallel and one of them failing in 90% of the cases.
Adding a data point here: we never hit this until upgrading testcontainers-go from 0.19 to 0.28.
Soon after, starting hitting this issue fairly regularly (I would estimate about 1 out of 4 jobs fail due to either
or
Sounds good to me 😃 In that case, make sure you call
testcontainers.TerminateContainer(ctr)for every container you spawn. In the library we ensure that while building it, and we do have two pipelines: with and without ryuk to verify all the tests are cleaning up what it creates.Yes. We’ll probably ask them to expose a variable so we know we’re in CI, so we can add this, and any future “hack” without pinging them constantly.
Currently, we prefer manually deleting our local containers than sporadic failures in the pipeline. As our CI/CD creates ephemeral pods for tests, they’re gonna be destroyed either way.
@axilis-marko the env variable must be set at the terminal/shell you use to run the tests, as documented here: https://golang.testcontainers.org/features/garbage_collector/#ryuk
Ah it was autoclosed by merging #2648
@tbrown1979 please remember that the reaper also removes built images, volumes and networks, so if you disable it please carefully remove all those Docker resources.
In any case, we have released a new version of Ryuk. As @stevenh pointed out in https://github.com/testcontainers/testcontainers-go/issues/2172#issuecomment-2226558170, you could give it a try like that.
For the record, SkipReaper is deprecated in newer versions and will eventually disappear from the Request struct (see v1 branch: https://github.com/testcontainers/testcontainers-go/blob/v1/request.go#L30)
If you want to disable Ryuk, please use the properties file here: https://golang.testcontainers.org/features/configuration/#customizing-ryuk-the-resource-reaper
@mdelapenya Just checked on Windows 11 with WSL2 docker. Test container version: v0.31.0. Had one error: “port not found: creating reaper failed: failed to create container”.
I am experiencing this exact same issue in gitlab runners when running a few different jobs that are all using testcontainers. 4/5 of my jobs always pass and one job (which runs tests in parallel) always fails when the other 4 are running but passes when I retry it (and it is running by itself).