testcontainers-python: Bug: concurrency causes Premature destruction of containers and networks (e.g. w/pytest-xdist)

Describe the bug

Testcontainers behaves incredibly flaky when running tests in parallel with pytest-xdist. Specifically containers and networks are destroyed prematurely resulting in network connections being interrupted, ports being reassigned to different containers resulting in calls to the wrong application or instance.

To Reproduce

Note due to the flakiness you might need to tune to number of runs or repeat the test execution to see the errors.

Install the packages used for the tests.

pip install pytest==8.2 pytest-xdist==3.6.1 testcontainers==4.4 requests==2.31

Create a file named example.py

from rand import randrange
from time import sleep

import pytest
import requests
from testcontainers.core.network import Network
from testcontainers.postgres import PostgresContainer
from testcontainers.vault import VaultContainer


@pytest.mark.parametrize("run", range(30))
def test_healthcheck(run: int) -> None:
    with Network() as network:
        with PostgresContainer().with_network(network) as postgres_container:
            with VaultContainer().with_network(network).with_exposed_ports(8200) as vault_container:
                sleep(randrange(1, 3))  # Simulate work and give some time to prematurely stop to showcase buggy behavior
                response = requests.get(f"{vault_container.get_connection_url()}/v1/sys/health")
                assert response.status_code == 200

Run pytest with parallel test execution on a multicore CPU (at least 2 cores).

pytest -n auto example.py

Runtime environment

Linux 6.8.0-76060800daily20240311-generic Python 3.10 3.11 3.12 CPU i7-1165G7 testcontainers 4.4

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 17 (7 by maintainers)

Most upvoted comments

oh i see the issue is with the wait_for_logs function not in the postgres module itself. yeah thats a bit tricky. i suspect that whole API will change with testcontainers v5 to look more like the oop ones (e.g. rust copies the “wait for” api).