testcontainers-dotnet: ResourceReaperException: Initialization has been cancelled

Describe the bug It looks like, that in some rare cases .NET Testcontainers can’t connect to Ryuk. I got a few times the following error:

DotNet.Testcontainers.Containers.ResourceReaperException : Initialization has been cancelled.

According the log messages, .NET Testcontainers tries to connect a couple of times. It looks like Ryuk is not ready to accept a connection. @PSanetra Do you have any idea? May this relate to the recent changes in the image?

To Reproduce -

Expected behavior .NET Testcontainers can connect to Ryuk.

Screenshots -

Desktop (please complete the following information):

  • Docker Desktop 4.7.1 (77678)

Additional context Build 2193. May relate to: #423, #441, #443.

Workaround Disable the ResourceReaper (TestcontainersSettings.ResourceReaperEnabled = false).

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 57 (14 by maintainers)

Commits related to this issue

Most upvoted comments

It would be also interesting if something like this improves it:

diff --git a/src/DotNet.Testcontainers/Containers/ResourceReaper.cs b/src/DotNet.Testcontainers/Containers/ResourceReaper.cs
index a36bf88..31e3181 100644
--- a/src/DotNet.Testcontainers/Containers/ResourceReaper.cs
+++ b/src/DotNet.Testcontainers/Containers/ResourceReaper.cs
@@ -48,6 +48,7 @@ namespace DotNet.Testcontainers.Containers
         .WithExposedPort(RyukPort)
         .WithPortBinding(RyukPort, true)
         .WithBindMount("/var/run/docker.sock", "/var/run/docker.sock", AccessMode.ReadOnly)
+        .WithStartupCallback((resourceReaper, ct) => Task.Delay(TimeSpan.FromSeconds(5), ct))
         .Build();
 
       this.SessionId = sessionId;

Maybe the container field (value) is not up-to-date, and we return a value that’s obsolete in the meantime. You need to remove the timeout in your test then.

Your proposition here is a good idea #443 (reply in thread)

I prefer to find the root cause. The resource reaper must run reliable.

This time the log file looks good. The resource reaper gets the right port (49156). I double-checked your sources (refactor-async-code) and noticed a few flaws. For instance:

var waitOS = EnvironmentTools.IsWindows()
    ? Wait.ForWindowsContainer()
    : Wait.ForUnixContainer();

Doesn’t make sense. Even if you’re running on Windows, it’s still a Linux container. It should be Wait.ForUnixContainer().UntilPortIsAvailable(MongoDbPort) instead. Take a look into the MongoDB example in this repository.

Maybe you can figure out with ResourceReaper.GetAndStartNewAsync() why Testcontainers can’t connect to the resource reaper on your machine.


Configure the logger and set the level to debug. That should help to identify the issue.

It’s included in 2.0.1 🤞.

I wrote a quick “test” to make it easily reproductible

public class ResourceReaperTest
{
    private readonly ITestOutputHelper _testOutputHelper;

    public ResourceReaperTest(ITestOutputHelper testOutputHelper)
    {
        _testOutputHelper = testOutputHelper;
    }

    [Fact]
    public async Task ResourceReaperStressTest()
    {
        var expectedPortRegex = new Regex("\"ExpectedPort\": (\\d+)");
        var hostPortRegex = new Regex("\"HostPort\": \"(\\d+)\"");
        
        var i = 0;
        while (i++ < 100)
        {
            try
            {
                await using var reaper = await ResourceReaper.GetAndStartNewAsync(Guid.NewGuid(), initTimeout: TimeSpan.FromSeconds(2));
                _testOutputHelper.WriteLine($"{i} : success");
            }
            catch (ResourceReaperException ex)
            {
                try
                {
                    var message = ex.Message;
                    var expectedPort = int.Parse(expectedPortRegex.Match(message).Groups[1].Value,
                        CultureInfo.InvariantCulture);
                    var hostPort = int.Parse(hostPortRegex.Matches(message)[1].Groups[1].Value,
                        CultureInfo.InvariantCulture);
                    _testOutputHelper.WriteLine($"{i} : fail - expectedPort: {expectedPort}, hostPort: {hostPort}");
                }
                catch (Exception)
                {
                    //NetworkSettings HostPort is missing usually
                    _testOutputHelper.WriteLine($"{i} : fail");
                }
            }
        }
    }
}

As you can see, it fails a lot!

1 : success 2 : fail - expectedPort: 49527, hostPort: 59414 3 : fail 4 : fail - expectedPort: 49529, hostPort: 59418 5 : fail 6 : success 7 : fail - expectedPort: 49532, hostPort: 59424 8 : fail 9 : fail 10 : fail 11 : fail 12 : fail - expectedPort: 49537, hostPort: 59438 13 : fail 14 : fail - expectedPort: 49539, hostPort: 59446 15 : fail - expectedPort: 49540, hostPort: 59449 16 : fail 17 : fail 18 : success 19 : fail 20 : fail 21 : fail 22 : fail 23 : fail 24 : fail - expectedPort: 49549, hostPort: 59470 25 : fail 26 : success 27 : fail - expectedPort: 49552, hostPort: 59478 28 : fail 29 : fail 30 : fail - expectedPort: 49555, hostPort: 59486 31 : fail

error.txt This is what I was getting.

I can confirm it has not happened again after adding “WithWaitStrategy

No, not lately. Looks stable for my builds, but this is an interesting comment: https://github.com/HofmeisterAn/dotnet-testcontainers/discussions/443#discussioncomment-2809925.