testcontainers-dotnet: [Bug]: Could not connect to a Mongo DB replica set with new Mongo DB driver 3.0.x

Testcontainers version

3.10.0

Using the latest Testcontainers version?

Yes

Host OS

Windows 11

Host arch

x64

.NET version

.NET 8

Docker version

Client: Version: 27.2.0 API version: 1.47 Go version: go1.21.13 Git commit: 3ab4256 Built: Tue Aug 27 14:17:17 2024 OS/Arch: windows/amd64 Context: desktop-linux

Server: Docker Desktop 4.34.2 (167172) Engine: Version: 27.2.0 API version: 1.47 (minimum version 1.24) Go version: go1.21.13 Git commit: 3ab5c7d Built: Tue Aug 27 14:15:15 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.7.20 GitCommit: 8fc6bcff51318944179630522a095cc9dbf9f353 runc: Version: 1.1.13 GitCommit: v1.1.13-0-g58aa920 docker-init: Version: 0.19.0 GitCommit: de40ad0

Docker info

Client: Version: 27.2.0 Context: desktop-linux Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.16.2-desktop.1 Path: C:\Program Files\Docker\cli-plugins\docker-buildx.exe compose: Docker Compose (Docker Inc.) Version: v2.29.2-desktop.2 Path: C:\Program Files\Docker\cli-plugins\docker-compose.exe debug: Get a shell into any image or container (Docker Inc.) Version: 0.0.34 Path: C:\Program Files\Docker\cli-plugins\docker-debug.exe desktop: Docker Desktop commands (Alpha) (Docker Inc.) Version: v0.0.15 Path: C:\Program Files\Docker\cli-plugins\docker-desktop.exe dev: Docker Dev Environments (Docker Inc.) Version: v0.1.2 Path: C:\Program Files\Docker\cli-plugins\docker-dev.exe extension: Manages Docker extensions (Docker Inc.) Version: v0.2.25 Path: C:\Program Files\Docker\cli-plugins\docker-extension.exe feedback: Provide feedback, right in your terminal! (Docker Inc.) Version: v1.0.5 Path: C:\Program Files\Docker\cli-plugins\docker-feedback.exe init: Creates Docker-related starter files for your project (Docker Inc.) Version: v1.3.0 Path: C:\Program Files\Docker\cli-plugins\docker-init.exe sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.) Version: 0.6.0 Path: C:\Program Files\Docker\cli-plugins\docker-sbom.exe scout: Docker Scout (Docker Inc.) Version: v1.13.0 Path: C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server: Containers: 7 Running: 5 Paused: 0 Stopped: 2 Images: 17 Server Version: 27.2.0 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 nvidia runc Default Runtime: runc Init Binary: docker-init containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353 runc version: v1.1.13-0-g58aa920 init version: de40ad0 Security Options: seccomp Profile: unconfined Kernel Version: 5.15.153.1-microsoft-standard-WSL2 Operating System: Docker Desktop OSType: linux Architecture: x86_64 CPUs: 24 Total Memory: 31.22GiB Name: docker-desktop ID: 013b6fcb-be8b-43a8-b578-96ca0a4f9fdf Docker Root Dir: /var/lib/docker Debug Mode: false HTTP Proxy: http.docker.internal:3128 HTTPS Proxy: http.docker.internal:3128 No Proxy: hubproxy.docker.internal Labels: com.docker.desktop.address=npipe://\.\pipe\docker_cli Experimental: false Insecure Registries: hubproxy.docker.internal:5555 127.0.0.0/8 Live Restore Enabled: false

What happened?

Hi, it seems that latest Mongo DB driver update have some specific changes about replica set connections. With MongoDB driver 3.0.0 it is impossible to connect to test-containers mongo using connection string provided by a test-containers builder.

Here is the test-containers and mongo client initialization:

var builder = new MongoDbBuilder().WithUsername("test").WithPassword("123").WithReplicaSet("rs1").Build();
await builder.StartAsync();
var client = new MongoClient(builder.GetConnectionString());
List<BsonDocument> collections = (await client.ListDatabasesAsync()).ToList();

Attemp to connect throws next exception:

A timeout occurred after 30000ms selecting a server using CompositeServerSelector{ Selectors = ReadPreferenceServerSelector{ ReadPreference = { Mode : Primary } }, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 }, OperationsCountServerSelector }. Client view of cluster state is { ClusterId : "1", Type : "ReplicaSet", State : "Disconnected", Servers : [] }.
   at MongoDB.Driver.Core.Clusters.Cluster.ThrowTimeoutException(IServerSelector selector, ClusterDescription description)
   at MongoDB.Driver.Core.Clusters.Cluster.WaitForDescriptionChangedHelper.HandleCompletedTask(Task completedTask)
   at MongoDB.Driver.Core.Clusters.Cluster.WaitForDescriptionChangedAsync(IServerSelector selector, ClusterDescription description, Task descriptionChangedTask, TimeSpan timeout, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Clusters.Cluster.SelectServerAsync(IServerSelector selector, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Clusters.IClusterExtensions.SelectServerAndPinIfNeededAsync(IClusterInternal cluster, ICoreSessionHandle session, IServerSelector selector, IReadOnlyCollection`1 deprioritizedServers, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Bindings.ReadPreferenceBinding.GetReadChannelSourceAsync(IReadOnlyCollection`1 deprioritizedServers, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableReadContext.InitializeAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableReadContext.CreateAsync(IReadBinding binding, Boolean retryRequested, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.ReadCommandOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.ListDatabasesOperation.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoClient.ExecuteReadOperationAsync[TResult](IClientSessionHandle session, IReadOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoClient.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
...

I found it is possible to fix it in two ways:

  1. Rollback to MongoDB driver 2.30.0;
  2. Specify directConnection=true option manually in the connection string provided by test-containers builder. While it works fine on my local setup (Docker Desktop + Windows), I found it doesn’t work on remote Linux server, throwing MongoDB.Driver.MongoNotPrimaryException : Server returned not primary error.

Either I’m doing something wrong, or it is undesired behavior with test-containers+mongodb:3.0.0

Relevant log output

No response

Additional information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16

Most upvoted comments

Good news! I will create a PR.

The last thing I’m not sure about is the internal wait strategy of the MongoDbBuilder - does it apply also before/after my own custom strategy or not. Anyway, I’ve got promising results from the test machine, as with new script it does not fail - ran tests multiple times.

You did everything correct👍. Your custom wait strategy will replace (override) the default wait strategy.

This is at least one of the issues:

var execResult = await container.ExecScriptAsync($"rs.initiate({{'_id':'{configuration.ReplicaSetName}',members:[{{'_id':1,'host':'127.0.0.1:27017'}}]}})");

This returns:

ExitCode = 0
Stderr = ""
Stdout = "true\n"

Which completes the wait strategy. We never actually wait for rs.status().ok to succeed. Good catch 🦅.

Edit: We need to rethrow or use two seperate commands.

Hi there, guys,

Finally, I’ve found some time to continue problem research and investigation. Everything below is about MongoDB driver version 3.2.0 and test-containers version 4.2.0.

I’ve paid some time to play with different driver configurations, and still can surely confirm two issues:

  1. Using default .WithReplicaSet() leads to an A timeout occurred after 30000ms selecting a server using CompositeServerSelector... error on my local Windows machine
  2. Enabling directConnection=true solves the 1st problem, but leads to MongoDB.Driver.MongoNotPrimaryException : Server returned not primary error. on the remote Linux machine

So, in order to beat both this issues I’ve tried some little hacks/configuration changes, etc. Nothing helped. Except… Addition of some scripts that request to MongoDB replica state… Look at this:

var builder = new MongoDbBuilder()
		  .WithUsername(_options.Username ?? string.Empty)
		  .WithPassword(_options.Password ?? string.Empty);

if (_options.EnableReplica)
	builder = builder.WithReplicaSet();

MongoDb = builder.Build();
await MongoDb.StartAsync();

if (_options.EnableReplica)
{
	await MongoDb.ExecScriptAsync("rs.initiate();"); // <-- ALL THESE LINES OF CODE IN THE BRANCH ARE IMPORTANT.
	await MongoDb.ExecScriptAsync("rs.conf()");
	await MongoDb.ExecScriptAsync("rs.status()");
}

MongoUrl = new MongoUrlBuilder(MongoDb.GetConnectionString())
{
	DatabaseName = _options.DefaultDatabaseName,
	AuthenticationSource = _options.AuthSource,
	DirectConnection = true, // <-- THIS IS IMPORTANT TOO.
	ReplicaSetName = "rs0"
}.ToMongoUrl();

RegisterService(services);

How do I get here? Well, I was trying to solve the remote test run problem with directConnection enabled (problem number 2 in the list above). For this purpose, I injected rs.conf() for the first time in order to log replica configuration to the console. When I first ran the code, test passed successfully. I was surprised. I’ve ran same test multiple times and found that it can either fail or succeed at random. I decided to also add rs.status(), and test run became stable in success.

I understand that it seems like a very stupid kind of magic, but that really works for me…

I assume that may be MongoDB changed how the replica set is initialized on startup, and may be, there is some kind of time gap that is needed to be awaited before trying to work with cluster using the mongo driver.

This can explain why time delay on executing rs.conf(); rs.status() is helped. This is also may be confirmed by the fact I have very powerful local machine and niche remote machine, so it is possible the remote machine cannot configure replica set in a given time, and may be some race condition or replica info misconfiguration appears in the driver connection mechanism.

Or may be rs.conf(); rs.status() trigger some work inside the MongoDB server which I’m not familiar with.

Ah, important note: I ensured that remote Linux machine creates an exact single node replica, and it seems that message Server returned not primary error. means that replica set was not properly configured at the moment of the query. I.e. single node replica set had no primary node assigned at the moment of the query, obviously, the single node must be the primary, but it seems that’s not true during the initiation phase.