testcontainers-dotnet: [Bug]: Could not connect to a Mongo DB replica set with new Mongo DB driver 3.0.x
Testcontainers version
3.10.0
Using the latest Testcontainers version?
Yes
Host OS
Windows 11
Host arch
x64
.NET version
.NET 8
Docker version
Client: Version: 27.2.0 API version: 1.47 Go version: go1.21.13 Git commit: 3ab4256 Built: Tue Aug 27 14:17:17 2024 OS/Arch: windows/amd64 Context: desktop-linux
Server: Docker Desktop 4.34.2 (167172) Engine: Version: 27.2.0 API version: 1.47 (minimum version 1.24) Go version: go1.21.13 Git commit: 3ab5c7d Built: Tue Aug 27 14:15:15 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.7.20 GitCommit: 8fc6bcff51318944179630522a095cc9dbf9f353 runc: Version: 1.1.13 GitCommit: v1.1.13-0-g58aa920 docker-init: Version: 0.19.0 GitCommit: de40ad0
Docker info
Client: Version: 27.2.0 Context: desktop-linux Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.16.2-desktop.1 Path: C:\Program Files\Docker\cli-plugins\docker-buildx.exe compose: Docker Compose (Docker Inc.) Version: v2.29.2-desktop.2 Path: C:\Program Files\Docker\cli-plugins\docker-compose.exe debug: Get a shell into any image or container (Docker Inc.) Version: 0.0.34 Path: C:\Program Files\Docker\cli-plugins\docker-debug.exe desktop: Docker Desktop commands (Alpha) (Docker Inc.) Version: v0.0.15 Path: C:\Program Files\Docker\cli-plugins\docker-desktop.exe dev: Docker Dev Environments (Docker Inc.) Version: v0.1.2 Path: C:\Program Files\Docker\cli-plugins\docker-dev.exe extension: Manages Docker extensions (Docker Inc.) Version: v0.2.25 Path: C:\Program Files\Docker\cli-plugins\docker-extension.exe feedback: Provide feedback, right in your terminal! (Docker Inc.) Version: v1.0.5 Path: C:\Program Files\Docker\cli-plugins\docker-feedback.exe init: Creates Docker-related starter files for your project (Docker Inc.) Version: v1.3.0 Path: C:\Program Files\Docker\cli-plugins\docker-init.exe sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.) Version: 0.6.0 Path: C:\Program Files\Docker\cli-plugins\docker-sbom.exe scout: Docker Scout (Docker Inc.) Version: v1.13.0 Path: C:\Program Files\Docker\cli-plugins\docker-scout.exe
Server: Containers: 7 Running: 5 Paused: 0 Stopped: 2 Images: 17 Server Version: 27.2.0 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 nvidia runc Default Runtime: runc Init Binary: docker-init containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353 runc version: v1.1.13-0-g58aa920 init version: de40ad0 Security Options: seccomp Profile: unconfined Kernel Version: 5.15.153.1-microsoft-standard-WSL2 Operating System: Docker Desktop OSType: linux Architecture: x86_64 CPUs: 24 Total Memory: 31.22GiB Name: docker-desktop ID: 013b6fcb-be8b-43a8-b578-96ca0a4f9fdf Docker Root Dir: /var/lib/docker Debug Mode: false HTTP Proxy: http.docker.internal:3128 HTTPS Proxy: http.docker.internal:3128 No Proxy: hubproxy.docker.internal Labels: com.docker.desktop.address=npipe://\.\pipe\docker_cli Experimental: false Insecure Registries: hubproxy.docker.internal:5555 127.0.0.0/8 Live Restore Enabled: false
What happened?
Hi, it seems that latest Mongo DB driver update have some specific changes about replica set connections. With MongoDB driver 3.0.0 it is impossible to connect to test-containers mongo using connection string provided by a test-containers builder.
Here is the test-containers and mongo client initialization:
var builder = new MongoDbBuilder().WithUsername("test").WithPassword("123").WithReplicaSet("rs1").Build();
await builder.StartAsync();
var client = new MongoClient(builder.GetConnectionString());
List<BsonDocument> collections = (await client.ListDatabasesAsync()).ToList();
Attemp to connect throws next exception:
A timeout occurred after 30000ms selecting a server using CompositeServerSelector{ Selectors = ReadPreferenceServerSelector{ ReadPreference = { Mode : Primary } }, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 }, OperationsCountServerSelector }. Client view of cluster state is { ClusterId : "1", Type : "ReplicaSet", State : "Disconnected", Servers : [] }.
at MongoDB.Driver.Core.Clusters.Cluster.ThrowTimeoutException(IServerSelector selector, ClusterDescription description)
at MongoDB.Driver.Core.Clusters.Cluster.WaitForDescriptionChangedHelper.HandleCompletedTask(Task completedTask)
at MongoDB.Driver.Core.Clusters.Cluster.WaitForDescriptionChangedAsync(IServerSelector selector, ClusterDescription description, Task descriptionChangedTask, TimeSpan timeout, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Clusters.Cluster.SelectServerAsync(IServerSelector selector, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Clusters.IClusterExtensions.SelectServerAndPinIfNeededAsync(IClusterInternal cluster, ICoreSessionHandle session, IServerSelector selector, IReadOnlyCollection`1 deprioritizedServers, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Bindings.ReadPreferenceBinding.GetReadChannelSourceAsync(IReadOnlyCollection`1 deprioritizedServers, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Operations.RetryableReadContext.InitializeAsync(CancellationToken cancellationToken)
at MongoDB.Driver.Core.Operations.RetryableReadContext.CreateAsync(IReadBinding binding, Boolean retryRequested, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Operations.ReadCommandOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Operations.ListDatabasesOperation.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)
at MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken)
at MongoDB.Driver.MongoClient.ExecuteReadOperationAsync[TResult](IClientSessionHandle session, IReadOperation`1 operation, CancellationToken cancellationToken)
at MongoDB.Driver.MongoClient.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
...
I found it is possible to fix it in two ways:
- Rollback to MongoDB driver 2.30.0;
- Specify
directConnection=trueoption manually in the connection string provided by test-containers builder. While it works fine on my local setup (Docker Desktop + Windows), I found it doesn’t work on remote Linux server, throwingMongoDB.Driver.MongoNotPrimaryException : Server returned not primary error.
Either I’m doing something wrong, or it is undesired behavior with test-containers+mongodb:3.0.0
Relevant log output
No response
Additional information
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16
Good news! I will create a PR.
You did everything correct👍. Your custom wait strategy will replace (override) the default wait strategy.
This is at least one of the issues:
This returns:
Which completes the wait strategy. We never actually wait for
rs.status().okto succeed. Good catch 🦅.Edit: We need to rethrow or use two seperate commands.
Hi there, guys,
Finally, I’ve found some time to continue problem research and investigation. Everything below is about MongoDB driver version
3.2.0and test-containers version4.2.0.I’ve paid some time to play with different driver configurations, and still can surely confirm two issues:
.WithReplicaSet()leads to anA timeout occurred after 30000ms selecting a server using CompositeServerSelector...error on my localWindowsmachinedirectConnection=truesolves the 1st problem, but leads toMongoDB.Driver.MongoNotPrimaryException : Server returned not primary error.on the remoteLinuxmachineSo, in order to beat both this issues I’ve tried some little hacks/configuration changes, etc. Nothing helped. Except… Addition of some scripts that request to MongoDB replica state… Look at this:
How do I get here? Well, I was trying to solve the remote test run problem with
directConnectionenabled (problem number 2 in the list above). For this purpose, I injectedrs.conf()for the first time in order to log replica configuration to the console. When I first ran the code, test passed successfully. I was surprised. I’ve ran same test multiple times and found that it can either fail or succeed at random. I decided to also addrs.status(), and test run became stable in success.I understand that it seems like a very stupid kind of magic, but that really works for me…
I assume that may be MongoDB changed how the replica set is initialized on startup, and may be, there is some kind of time gap that is needed to be awaited before trying to work with cluster using the mongo driver.
This can explain why time delay on executing
rs.conf(); rs.status()is helped. This is also may be confirmed by the fact I have very powerful local machine and niche remote machine, so it is possible the remote machine cannot configure replica set in a given time, and may be some race condition or replica info misconfiguration appears in the driver connection mechanism.Or may be
rs.conf(); rs.status()trigger some work inside the MongoDB server which I’m not familiar with.Ah, important note: I ensured that remote Linux machine creates an exact single node replica, and it seems that message
Server returned not primary error.means that replica set was not properly configured at the moment of the query. I.e. single node replica set had no primary node assigned at the moment of the query, obviously, the single node must be the primary, but it seems that’s not true during the initiation phase.