colima: Docker cannot pull images

Hi,

i just installed colima on a MacBook Pro wit BigSur 11.6.2

colima version 0.3.2
git commit: 272db4732b90390232ed9bdba955877f46a50552

runtime: docker
arch: x86_64
client: v20.10.12
server: v20.10.11

When i want to pull in docker, I get an i/o timeout error. It seems that the colima system doesn’t have internet connection.

docker pull maven Using default tag: latest Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 192.168.5.3:53: read udp 192.168.5.15:56157->192.168.5.3:53: i/o timeout

Are there any post-install steps to get a connection?

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 16
  • Comments: 38 (9 by maintainers)

Commits related to this issue

Most upvoted comments

I also faced the same issue but its resolved by specifying DNS resolver

colima start --dns 1.1.1.1

New colima user here, running into this right off the bat. lima version is 0.8.3, colima 0.3.3. This workaround fixed it for me: https://github.com/abiosoft/colima/issues/140#issuecomment-1028395976

Does anyone have Cisco AnyConnect installed?

I have an intel mac that I just upgraded from Catalina to Monterey. Since the upgrade, I’ve been experiencing various network timeouts, but the dns issues in colima were the most pronounced as they blocked my use of docker pull. Outside of Colima, git was often hanging as well, so I didn’t think it was a uniquely colima issue, so I kept looking after I found this issue.

I have Cisco AnyConnect installed which I occasionally use to connect to a VPN. After the Monterey update, “Cisco AnyConnect Socket Filter” showed up and asked for permission to run a new SystemExtension. I allowed it at that point, but I think that was the culprit behind all my network issues. Here are some other issues people experienced with it: https://apple.stackexchange.com/questions/420773/the-process-com-cisco-anyconnect-macos-acsockext-hogs-mac-cpu-but-cannot-be-kill

This service is suspicious (to me) because its “features” are (based on the docs):

  • DNS proxy (aka: screw up the DNS by doing MITM crap)
  • App/Transparent proxy
  • Content filter

So, I just deleted Cisco AnyConnect Socket Filter (deleted it from the Applications) which removed the SystemExtension. And, I stopped its annoying “notification” service from pestering me about it on reboot.

$ launchctl blame cisco
// this prints a list the services. You want the gui/...cisco.anyconnect.notification... one.
$ launchctl disable gui/<number>/application.com.cisco.anyconnect.notification.<number>.<number>
$ launchctl stop gui/<number>/application.com.cisco.anyconnect.notification.<number>.<number>
$ launchctl kill 9 gui/<number>/application.com.cisco.anyconnect.notification.<number>.<number>

After doing all of that (and another reboot), dns works in colima again!

@abiosoft I’m seeing the same timeout and lookup failure as @navels, only in my case it was triggered by pushing a number of images in quick succession instead of pulling a single large one. I’ve confirmed that docker pull localstack/localstack:latest often fails with endless retry messages for me as well.

% colima version
colima version HEAD-5e2e413
git commit: 5e2e41310e595553dcdc29ba45827d4030af37bb

Other details that might be helpful:

  • Restarting the colima VM with colima stop; colima start resolves the issue temporarily, allowing name lookups to complete again until another large push/pull
  • I noticed this issue against an azure container registry initially, and assumed they were rate limiting. Now that I see it happens with docker.io too, that seems less likely.

Ping output from within the VM used to be very strange with a constantly increasing round trip and DUP packets, but that appears to be fixed in this latest version. 👍

colima stop; colima start works for me after searching a while for the solution. Thank you very much!!!

@abiosoft I’m seeing the same timeout and lookup failure as @navels, only in my case it was triggered by pushing a number of images in quick succession instead of pulling a single large one. I’ve confirmed that docker pull localstack/localstack:latest often fails with endless retry messages for me as well.

% colima version
colima version HEAD-5e2e413
git commit: 5e2e41310e595553dcdc29ba45827d4030af37bb

Other details that might be helpful:

  • Restarting the colima VM with colima stop; colima start resolves the issue temporarily, allowing name lookups to complete again until another large push/pull
  • I noticed this issue against an azure container registry initially, and assumed they were rate limiting. Now that I see it happens with docker.io too, that seems less likely.

Ping output from within the VM used to be very strange with a constantly increasing round trip and DUP packets, but that appears to be fixed in this latest version. 👍

Can anyone try the lastest development version and see if anything changes?

brew install --HEAD colima

@abiosoft Yes that seems to fix things. I ended up using 192.168.5.2, the host, since work runs a dns proxy on my laptop. This way I can resolve private addresses not on the public DNS.

@pedantic79 a lima upgrade should be all that is required.

For troubleshooting purposes, can you kindly try this https://github.com/abiosoft/colima/issues/140#issuecomment-1028395976 and see if the behaviour is different? Note that it requires recreating the VM to see the effect i.e. colima delete (if exits) prior to starting.

This is a good overview of DNS issues in Alpine and might be at the core of some of these DNS issues:

https://support.cloudbees.com/hc/en-us/articles/360040999471-UnknownHostException-caused-by-DNS-Resolution-issue-with-Alpine-Images

Their main fix was to migrate to RedHat’s Universal Base Images (UBI) - https://developers.redhat.com/products/rhel/ubi

There is a workaround as well, that I will try when I have a bit of time to test it.