testcontainers-python: Bug: Since 4.0.0 `postgres.get_connection_url()` within another container gives broken connection url

Describe the bug

Before 4.0.0, after creating a PostgresContainer within another container, but with the same external daemon, postgres.get_connection_url() would return something like postgresql+psycopg2://test:test@172.17.0.1:32770/test which is the correct connection url.

However, since 4.0.0, the exact same container’s postgres.get_connection_url() returns postgresql+psycopg2://test:test@localhost:5432/test.

So it changed:

  • The exposed port to the internal one
  • The host from the gateway ip to the container host ip

Considering that the newly formatted connection url doesn’t work to create an engine, I wouldn’t expect this behavior.

To Reproduce

# From within a container
from testcontainers.postgres import PostgresContainer
from sqlalchemy.engine import create_engine

with PostgresContainer("postgres:15.2") as postgres:
    with create_engine(postgres.get_connection_url()).connect() as connection:
        ...

Runtime environment

Operating system: Linux d388e0f17614 5.10.179-168.710.amzn2.x86_64 #1 SMP Mon May 22 23:10:22 UTC 2023 x86_64 GNU/Linux Python version: Python 3.11.5

docker info 
Client:
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /root/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.0
    Path:     /root/.docker/cli-plugins/docker-compose

Server:
 Containers: 24
  Running: 5
  Paused: 0
  Stopped: 19
 Images: 97
 Server Version: 20.10.23
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1e1ea6e986c6c86565bc33d52e34b81b3e2bc71f
 runc version: f19387a6bec4944c770f7668ab51c4348d9c2f38
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.179-168.710.amzn2.x86_64
 Operating System: Amazon Linux 2
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 30.99GiB
 Name:
 ID:
 Docker Root Dir: /home/xxxx/data/docker_data
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  xxxxxx
 Live Restore Enabled: false
pip freeze 
aiobotocore==2.12.1
aiohttp==3.9.3
aiohttp-cors==0.7.0
aioitertools==0.11.0
aiosignal==1.3.1
alembic==1.13.1
aniso8601==9.0.1
anyio==4.3.0
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asttokens==2.4.1
async-lru==2.0.4
async-timeout==4.0.3
attrs==23.2.0
autogluon.common==1.0.0
autogluon.core==1.0.0
autogluon.features==1.0.0
autogluon.tabular==1.0.0
Babel==2.14.0
backoff==2.2.1
beautifulsoup4==4.12.3
binaryornot==0.4.4
black==24.2.0
bleach==6.1.0
blessed==1.20.0
blinker==1.7.0
blis==0.7.11
boto3==1.34.51
boto3-stubs==1.34.51
botocore==1.34.51
botocore-stubs==1.34.62
bump2version==1.0.1
cachetools==5.3.3
catalogue==2.0.10
catboost==1.2.3
certifi==2024.2.2
cffi==1.16.0
cfgv==3.4.0
chardet==5.2.0
charset-normalizer==3.3.2
click==8.1.7
cloudpathlib==0.16.0
cloudpickle==2.2.1
cmake==3.28.3
coloredlogs==14.0
colorful==0.5.6
comm==0.2.2
confection==0.1.4
contextlib2==21.6.0
contourpy==1.2.0
cookiecutter==2.6.0
coverage==7.4.3
croniter==2.0.2
cruft==2.15.0
cycler==0.12.1
cymem==2.0.8
dagster==1.6.9
dagster-aws==0.22.9
dagster-graphql==1.6.9
dagster-pipes==1.6.9
dagster-postgres==0.22.9
dagster-webserver==1.6.9
debugpy==1.8.1
decorator==5.1.1
deepdiff==6.7.1
defusedxml==0.7.1
deprecation==2.1.0
dill==0.3.8
distlib==0.3.8
docker==7.0.0
docstring-parser==0.15
entrypoints==0.4
et-xmlfile==1.1.0
execnet==2.0.2
executing==2.0.1
fastai==2.7.14
fastapi==0.110.0
fastcore==1.5.29
fastdownload==0.0.7
fastjsonschema==2.19.1
fastprogress==1.0.3
filelock==3.13.1
Flask==3.0.2
fonttools==4.49.0
fqdn==1.5.1
frozenlist==1.4.1
fsspec==2024.2.0
future==1.0.0
gitdb==4.0.11
GitPython==3.1.42
google-api-core==2.17.1
google-auth==2.28.2
google-pasta==0.2.0
googleapis-common-protos==1.63.0
gpustat==1.1.1
gql==3.5.0
graphene==3.3
graphql-core==3.2.3
graphql-relay==3.2.0
graphviz==0.20.1
greenlet==3.0.3
grpcio==1.62.1
grpcio-health-checking==1.62.1
gunicorn==21.2.0
h11==0.14.0
httpcore==1.0.4
httptools==0.6.1
httpx==0.27.0
humanfriendly==10.0
hyperopt==0.2.7
identify==2.5.35
idna==3.6
importlib-metadata==6.11.0
incremental==22.10.0
iniconfig==2.0.0
ipykernel==6.29.3
ipython==8.22.2
isoduration==20.11.0
itsdangerous==2.1.2
jedi==0.19.1
Jinja2==3.1.3
jmespath==1.0.1
joblib==1.3.2
json5==0.9.22
jsonpointer==2.4
jsonschema==4.17.3
jsonschema-specifications==2023.12.1
jupyter-events==0.6.3
jupyter-lsp==2.2.4
jupyter_client==8.6.1
jupyter_core==5.7.2
jupyter_server==2.10.0
jupyter_server_terminals==0.5.3
jupyterlab==4.1.4
jupyterlab_pygments==0.3.0
jupyterlab_server==2.24.0
kaleido==0.2.1
kiwisolver==1.4.5
langcodes==3.3.0
lightgbm==4.1.0
lit==18.1.1
llvmlite==0.42.0
Mako==1.3.2
Markdown==3.5.2
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib==3.8.3
matplotlib-inline==0.1.6
mdurl==0.1.2
mistune==3.0.2
mlflow==2.11.1
mpmath==1.3.0
msgpack==1.0.8
multidict==6.0.5
multiprocess==0.70.16
murmurhash==1.0.10
mypy==1.8.0
mypy-boto3-s3==1.34.62
mypy-extensions==1.0.0
nbclient==0.10.0
nbconvert==7.16.2
nbformat==5.10.2
nest-asyncio==1.6.0
networkx==3.2.1
nodeenv==1.8.0
notebook==7.1.1
notebook_shim==0.2.4
numba==0.59.0
numpy==1.26.4
nvidia-cublas-cu11==11.10.3.66
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu11==11.7.101
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu11==8.5.0.96
nvidia-cudnn-cu12==8.9.2.26
nvidia-cufft-cu11==10.9.0.58
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu11==10.2.10.91
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu11==11.4.0.1
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu11==11.7.4.91
nvidia-cusparse-cu12==12.1.0.106
nvidia-ml-py==12.535.133
nvidia-nccl-cu11==2.14.3
nvidia-nccl-cu12==2.19.3
nvidia-nvjitlink-cu12==12.4.99
nvidia-nvtx-cu11==11.7.91
nvidia-nvtx-cu12==12.1.105
opencensus==0.11.4
opencensus-context==0.1.3
openpyxl==3.1.2
ordered-set==4.1.0
orjson==3.9.15
overrides==7.7.0
packaging==23.2
pandarallel==1.6.5
pandas==2.1.4
pandocfilters==1.5.1
parse==1.20.1
parse-type==0.6.2
parso==0.8.3
pathos==0.3.2
pathspec==0.12.1
patsy==0.5.6
pendulum==3.0.0
pexpect==4.9.0
pillow==10.2.0
platformdirs==3.11.0
plotly==5.19.0
pluggy==1.4.0
pox==0.3.4
ppft==1.7.6.8
pre-commit==3.6.2
preshed==3.0.9
prometheus_client==0.20.0
prompt-toolkit==3.0.43
protobuf==4.25.3
psutil==5.9.8
psycopg2==2.9.9
psycopg2-binary==2.9.9
ptyprocess==0.7.0
pure-eval==0.2.2
py-cpuinfo==9.0.0
py-spy==0.3.14
py4j==0.10.9.7
pyarrow==15.0.1
pyasn1==0.5.1
pyasn1-modules==0.3.0
pycparser==2.21
pydantic==1.10.14
pygit2==1.14.1
Pygments==2.17.2
pyparsing==3.1.2
pyrsistent==0.20.0
pyspark==3.5.1
pytest==8.0.2
pytest-alembic==0.11.0
pytest-bdd==7.1.1
pytest-benchmark==4.0.0
pytest-cov==4.1.0
pytest-xdist==3.5.0
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
python-json-logger==2.0.7
python-slugify==8.0.4
pytz==2024.1
PyYAML==6.0.1
pyzmq==25.1.2
querystring-parser==1.2.4
ray==2.6.3
referencing==0.33.0
requests==2.31.0
requests-toolbelt==1.0.0
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rich==13.7.1
rpds-py==0.18.0
rsa==4.9
ruff==0.3.1
s3fs==2024.2.0
s3transfer==0.10.0
sagemaker==2.212.0
sagemaker-ssh-helper==2.1.0
schema==0.7.5
scikit-learn==1.4.0
scipy==1.12.0
Send2Trash==1.8.2
shap==0.44.1
six==1.16.0
sklearn-pandas==2.2.0
slack_sdk==3.27.1
slicer==0.0.7
smart-open==6.4.0
smdebug-rulesconfig==1.0.1
smmap==5.0.1
sniffio==1.3.1
soupsieve==2.5
spacy==3.7.4
spacy-legacy==3.0.12
spacy-loggers==1.0.5
SQLAlchemy==2.0.28
sqlparse==0.4.4
srsly==2.4.8
stack-data==0.6.3
starlette==0.36.3
statsmodels==0.14.1
structlog==24.1.0
sympy==1.12
tabulate==0.9.0
tblib==2.0.0
tenacity==8.2.3
tensorboardX==2.6.2.2
terminado==0.18.1
testcontainers==4.0.1
text-unidecode==1.3
thinc==8.2.3
threadpoolctl==3.3.0
time-machine==2.14.0
tinycss2==1.2.1
tokenize-rt==5.2.0
tomli==2.0.1
toposort==1.10
torch==2.0.1
torchvision==0.15.2
tornado==6.4
towncrier==23.11.0
tqdm==4.66.2
traitlets==5.14.2
triton==2.0.0
typer==0.9.0
types-awscrt==0.20.5
types-jsonschema==4.21.0.20240118
types-Markdown==3.5.0.20240129
types-psycopg2==2.9.21.20240218
types-python-dateutil==2.8.19.20240311
types-PyYAML==6.0.12.12
types-requests==2.31.0.20240218
types-s3transfer==0.10.0
typing_extensions==4.10.0
tzdata==2024.1
universal_pathlib==0.2.2
uri-template==1.3.0
urllib3==2.0.7
uvicorn==0.27.1
uvloop==0.19.0
virtualenv==20.21.0
wasabi==1.1.2
watchdog==4.0.0
watchfiles==0.21.0
wcwidth==0.2.13
weasel==0.3.4
webcolors==1.13
webencodings==0.5.1
websocket-client==1.7.0
websockets==12.0
Werkzeug==3.0.1
wrapt==1.16.0
xgboost==2.0.3
yarl==1.9.4
zipp==3.18.0

About this issue

  • Original URL
  • State: open
  • Created 4 months ago
  • Comments: 15 (4 by maintainers)

Most upvoted comments

Hello,

@totallyzen you’re welcome 😃

So I just tested on main and unfortunately:

  1. The new ryuk feature from https://github.com/testcontainers/testcontainers-python/commit/d0198744c3bdc97a7fe41879b54acb2f5ee7becb breaks completely in the setting of using the external daemon as I would expect due to the incorrect host (it breaks here https://github.com/testcontainers/testcontainers-python/blob/main/core/testcontainers/core/container.py#L205)
  2. Without ryuk the problem persists even if the port seems okay now, the host is still localhost (which is not okay for connections as we are within a container and the container we want to reach runs outside, on the host machine)

@bearrito I’d have to test with a dind image, I’m not sure why it wouldn’t work. However I’m sure docker run -v /var/run/docker.sock:/var/run/docker.sock ... works as it is what we are doing right now with Github managed runners.

+2
Tanguy-LeFloch on Mar 20, 2024

I can repro this. But it looks largely intentional based of the PR and some of the related TC projects. A maintainer could comment more.

+1
bearrito on Mar 16, 2024
Read more comments on GitHub
← testcontainers-node: TLS config is not detected resulting in error
testcontainers-python: testcontainers wait indefinitely without effects when running in Gitlab-CI →