colima: Colima/Docker cannot download Docker hub base images in a Zscaler environment, while Docker Desktop could

I have: colima version 0.3.4 git commit: 5a4a70481ca8d1e794677f22524e3c1b79a9b4ae

runtime: docker arch: x86_64 client: v20.10.13 server: v20.10.11

I’m working on a MacBook Pro in a Jamf/Zscaler environment. People are able to download images from docker hub in my environment if they use Docker Desktop.

I have placed the root certificates in my ~/.docker/certs.d directory and restarted colima, but I am still getting this error when I try to build a Docker image that has an alpine base image. It fails on the download of the base image:

[+] Building 0.5s (4/4) FINISHED
 => [internal] load build definition from Dockerfile                                                                         0.0s
 => => transferring dockerfile: 37B                                                                                          0.0s
 => [internal] load .dockerignore                                                                                            0.0s
 => => transferring context: 34B                                                                                             0.0s
 => CANCELED [internal] load metadata for docker.io/library/nginx:alpine                                                     0.4s
 => ERROR [internal] load metadata for docker.io/library/node:12.18.1-alpine3.12                                             0.4s
------
 > [internal] load metadata for docker.io/library/node:12.18.1-alpine3.12:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to do request: Head "https://registry-1.docker.io/v2/library/node/manifests/12.18.1-alpine3.12": x509: certificate signed by unknown authority

Is there a particular root CA certificate format I need? Does the file name need to be ca.crt? Does the format need to be anything in particular? I have tried a cert exported from Keychain Access in .cer format and in PEM format.

Perhaps I need to find intermediate certificates? I’m not sure where those would be.

I have tried adding a ~/.docker/daemon.json with an entry like this: "insecure-registries": ["registry-1.docker.io"], but this did not work.

I also triedexporting the root certs in .cer and PEM format and putting them into the ~/.docker/certs.d/ directory and restarting and even deleting colima and starting, but that did not work. Perhaps I am not putting the correct certs in the certs.d folder? Perhaps not in the right format?

Do I need to add something to the override.yaml to make this work? I tried adding this to the override.yaml as seen above:

provision:
  - mode: user
    script: |
    #!/bin/bash
    sudo ln -sf /Users/${USER}/.docker/certs.d /etc/docker/certs.d

But this resulted in a colima stop/start error:

FATA[0000] yaml: line 13: could not find expected ':'
WARN[0000] error detecting Lima config directory: error retrieving lima info: exit status 1
WARN[0000] falling back to default '$HOME/.lima'

_Originally posted by @bmichaud in https://github.com/abiosoft/colima/issues/131#issuecomment-1090920068_

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 2
Comments: 21 (5 by maintainers)

Most upvoted comments

I’m facing the same pb with zscaler. Here a function in my .bashrc.

start_colima() {
  local CERTS="${HOME}/.ca-certificates"
  local URL="registry-1.docker.io:443"
  mkdir -p ${CERTS}
  openssl s_client -showcerts -connect ${URL} </dev/null 2>/dev/null|openssl x509 -outform PEM >${CERTS}/docker-com.pem
  openssl s_client -showcerts -verify 5 -connect ${URL} </dev/null 2>/dev/null | sed -ne '/-BEGIN/,/-END/p' >${CERTS}/docker-com-chain.pem
  colima start
  
  colima ssh -- sudo cp ${CERTS}/* /usr/local/share/ca-certificates/
  colima ssh -- sudo update-ca-certificates
  colima ssh -- sudo service docker restart
}

+32

ghost on Jun 19, 2022

Any update on getting a permanent fix for this instead in colima itself?

royteeuwen on Nov 3, 2022

Lima v0.10.0 now supports specifying additional CA certs

Note that the support for this feature in lima-init for alpine-lima has not been implemented yet, so just moving to lima 0.10 is not enough. I hope to get around to it soon, but not clear yet when I get the time.

jandubois on May 9, 2022

Since Zscaler will intercept (MITM) all your connections, it’s rather tedious to declare certificates for every server individually. The following will solve your issue once and forever. I keep a copy of custom root CAs (e.g. Zscaler) in ${HOME}/.ca-certificates (important: the directory must be inside your home directory, so that it is auto-mounted inside Colima). Then I run:

#!/bin/zsh
CERTS="${HOME}/.ca-certificates"
colima ssh -- sudo cp ${CERTS}/* /usr/local/share/ca-certificates/
colima ssh -- sudo update-ca-certificates

This copies all my custom CA certs inside Colima and registers them as trusted root CAs. Now pulling through Zscaler works from all official registries without further configuration. It also allows me to install additional packages inside the Colima VM using “apk” (which would otherwise also fail because of invalid TLS chain). You may have to restart the docker daemon afterwards:

colima ssh -- sudo service docker restart

(not sure about this, I’m doing it anyways because of other changes I am running during VM setup)

danielbobbert on May 9, 2022

@bmichaud you can use sudo

abiosoft on Apr 12, 2022

Are the certificates in this format https://docs.docker.com/engine/security/certificates/#understand-the-configuration?

$HOME/certs.d/              <-- Certificate directory
└── localhost:5000          <-- Hostname:port
   ├── client.cert          <-- Client certificate
   ├── client.key           <-- Client key
   └── ca.crt               <-- Certificate authority that signed
                                the registry certificate

As a point of clarification, I thought I was supposed to put the certs in ~/.docker/certs.d (see https://github.com/abiosoft/colima/issues/131#issuecomment-1022809521), but your comment above, as a colleague pointed out, is saying I should put them in ~/certs.d instead.

Also, can certs ever be placed directly in the certs.d directory, or must they always be in a subdirectory?

Resolution: I have verified that putting the certs in the ~/.docker/certs.d directory does result in the files getting copied into the colima ssh VM at /etc/docker/certs.d.

bmichaud on Apr 27, 2022