colima: Slow image pulling when using VZ VM type

Description

Pulling images is very slow when I set the VM type to VZ.

On the docker hello world image, running it from a clean colima instances takes 3 seconds using the default parameters, and 17 seconds when --vm-type=vz is set. It looks like the time is spent initiating the connection because the download speed looks fine. Could be a slow DNS query, a slow TCP handshake, a slow TLS handshake, I don’t know.

Version

Colima Version: 0.5.4 Lima Version: 0.15.0 Qemu Version: 7.2.0

Operating System

macOS M1 >= 13 (Ventura)

Output of colima status

INFO[0000] colima is running using macOS Virtualization.Framework 
INFO[0000] arch: aarch64                                
INFO[0000] runtime: docker                              
INFO[0000] mountType: virtiofs                          
INFO[0000] socket: unix:///Users/fungiboletus/.colima/default/docker.sock 

Reproduction Steps

  1. colima start --cpu 4 --memory 4
  2. time docker run -it --rm hello-world
  3. colima delete
  4. colima start --cpu 4 --memory 4 --vm-type=vz
  5. time docker run -it --rm hello-world
  6. colima delete

Expected behaviour

The duration of pulling and running the hello-world image should be similar.

Additional context

No response

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 6
  • Comments: 21 (4 by maintainers)

Most upvoted comments

Maybe we could have IPv6 disabled by default, as is tradition.

I have done some tests and I think it might be related to DNS and IPv6, with default dns setting of [] in the yaml configuration, which from I gather can be forced on startup using:

colima start --dns "" --disk 200 --cpu 6 --memory 10 --ssh-agent --vm-type vz --mount-type virtiofs

when using ping:

time colima ssh -- ping  www.github.com -c 1

PING www.github.com (...): 56 data bytes
64 bytes from ....: seq=0 ttl=42 time=0.497 ms

--- www.github.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.497/0.497/0.497 ms
colima ssh -- ping www.github.com -c 1  0,12s user 0,09s system 17% cpu 1,203 total

requires over 1 second for just one ping command to finish executing (takes quite long to even start9.

if forcing ping with IPv4:

time colima ssh -- ping -4 www.github.com -c 1

PING www.github.com (....): 56 data bytes
64 bytes from ....: seq=0 ttl=42 time=0.256 ms

--- www.github.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.256/0.256/0.256 ms
colima ssh -- ping -4 www.github.com -c 1  0,11s user 0,08s system 100% cpu 0,193 total

it takes way about 0.2 seconds for everything to finish.

And if I do a docker pull:

docker image rm hello-world:latest

time docker pull hello-world:latest
latest: Pulling from library/hello-world
...: Pull complete
Digest: ....
Status: Downloaded newer image for hello-world:latest
docker.io/library/hello-world:latest
docker pull hello-world:latest  0,04s user 0,02s system 0% cpu 16,414 total

if I override the default DNS with google dns:

colima start --dns 8.8.8.8 --dns 8.8.4.4 --disk 200 --cpu 6 --memory 10 --ssh-agent --vm-type vz --mount-type virtiofs

ping without forcing IP v4:

time colima ssh -- ping www.github.com -c 1

PING www.github.com (...): 56 data bytes
64 bytes from ...: seq=0 ttl=42 time=0.260 ms

--- www.github.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.260/0.260/0.260 ms
colima ssh -- ping www.github.com -c 1  0,12s user 0,09s system 104% cpu 0,199 total

results in basically the same response as before upon forcing IP v4.

This also matches the docker pull results:

docker image rm hello-world:latest

time docker pull hello-world:latest
latest: Pulling from library/hello-world
...: Pull complete
Digest: ...
Status: Downloaded newer image for hello-world:latest
docker.io/library/hello-world:latest
docker pull hello-world:latest  0,04s user 0,02s system 2% cpu 2,318 total

Not sure if the issue is in colima directly or maybe lima? if the second, somebody should create an issue there, if no such issue exists.

However, specifying the correct DNS manually might not be that easy all the time as might change depending on the local/corporate network or VPN. Though I guess for the colima VM itself that might not matter - except if there is a private docker registry that is only reachable using the correct DNS server 😄

@aaronlehmann I would recommend opening a separate issue for that.

Thanks.

OK. I have done some more testing and the best solution seems to supply --dns 192.168.5.3 on startup. This way it sets the correct default DNS and seems to force DNS requests to use IPv4 internally as the issue does not occur.

Also applying dns does not seem to change the file at all after initial instance creation but it still seems to fix the slow dns resolution upon sub-sequent starts.

So e.g.:

colima start --dns 192.168.5.3 --disk 200 --cpu 6 --memory 10 --ssh-agent --vm-type vz --mount-type virtiofs