colima: Connection refused right after container start
There is a problem with migration from Docker Desktop to Colima docker environment. When the container is just started, it’s mapped port is not available immediately. I’m not quite sure about all what happens underneath (is it an issue of Lima/Colima/jna/docker-java), but let me give my assumption, the demo is provided as well.
We use testcontainers, a popular framework that bootstraps docker containers for test purposes, in our case it’s Postgres. We use image postgres:11.6-alpine.
When the container starts, it allows to retrieve mapped port and right after that the connection is set up.
With Docker desktop it works just fine, but with colima the retry-with-interval workaround is required. Default logic was not expecting this, it was just working fine for years.
My assumption is that the logic that retrieves mapped port is not blocked by startup await, while in Docker desktop there is a strong guarantee about it.
I’ve created a demo project that shows the issue. If you start with Docker Desktop, you will see log:
...testcontainers initialization without issues...
...
jdbc:postgresql://localhost:55001/test?loggerLevel=OFF
>>>> Successfully first time
If you start the same test with Colima, you will see log:
...testcontainers initialization with many retry failures (bootstraps service Ryuk container first)
...
jdbc:postgresql://localhost:49154/test?loggerLevel=OFF
>>>> First exception: org.postgresql.util.PSQLException: Connection to localhost:49154 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
...sleep here...
>>>> Successfully second time
The mapped port is obtained in internal logic in this line:
String jdbcUrl = container.getJdbcUrl();
Worth to mention, that on Apple M1 this failure is more explicit (probably because it’s faster), with Intel I see only testcontainers Ryuk failures (most probably they added this retry policy to cover the described issue), but still there is an obvious problem.
Let me know if I can provide more details or more convenient example (not sure that Java example is good enough for you).
About this issue
- Original URL
- State: open
- Created 5 years ago
- Reactions: 5
- Comments: 19 (9 by maintainers)
Commits related to this issue
- workaround while abiosoft/colima#71 is open — committed to JeanHuguesdeRaigniac/persistent by JeanHuguesdeRaigniac 5 years ago
On my case (MacOS), due to dropping docker desktop I realised that my
/var/run/docker.sockwas still targeting the old file (this would then be making other stuff to fail).Solved it by removing the old symlink and then
sudo ln -s /Users/$USER/.colima/docker.sock /var/run/docker.sock.why is this closed?
There needs to be a proper monitoring of Docker containers and dynamic port forwarding handled by Colima itself. Rather than falling back to Lima’s which can have a delay of up to 3 seconds.
We see this intermittently in about half of DDEV’s automated test runs on Colima. It’s a persistent problem.
This is expected, and due to the way port forwarding in Lima works: the guest agent running inside the VM checks every 5 seconds for open ports. It sends any changes since the previous check to the host agent (running on the host), which will in turn create a tunnel for each new port, and tear down the tunnel for each port that has been closed.
So you should expect to wait on average about 2½s before the port becomes available on the host, and potentially up to 5s (or more if the machine is busy).
To avoid the delay you would need to use an additional vmnet port, via
vde_vmnet. I don’t know if Colima exposes support for this, but Lima can manage the additional daemons automatically. It does requirerootto set up initially.