rancher-desktop: Running `nerdctl build` with Rancher Desktop fails to pull base images from custom registry with self-signed cert

I found a workaround, by injecting the corp certs into a secret, and then mounting that into the builder DaemonSet.

$ kubectl create secret generic -n kube-image corp-certs \
                             --from-file=root_ca="/Users/me/certificates/Root-CA.pem" \
                             --from-file=subCA="/Users/me/certificates/subCA.pem"
secret/corp-certs created

Then edit the daemonset: kubectl edit -n kube-image daemonset.apps/builder

Add the secret as a volume:

      volumes:
      ...
      - name: corp-certs
        secret:
          defaultMode: 420
          secretName: corp-certs

Add a volumemount:

        volumeMounts:
        ...
        - mountPath: /usr/local/share/ca-certificates/
          name: corp-certs
          readOnly: true

Add a lifecycle postStart to update the CA certs:

        image: docker.io/moby/buildkit:v0.8.3
        imagePullPolicy: IfNotPresent
        lifecycle:
          postStart:
            exec:
              command:
              - update-ca-certificates

And now, nerdctl/kim can build an image using an image in my private image repo.

You are correct, the workaround only works until Rancher Desktop is restarted.
@jandubois, @dweomer, I hope Rancher has a plan to fix this before January 31, when Docker Desktop can no longer be used. Rancher Desktop is the best replacement I have found, but this issue makes it unusable for developers.

I have not been able to reproduce this issue (with Rancher Desktop 1.0.1).

I’ve setup a private registry with a self-signed cert and basic auth requirements as registry.home. It contains an nginx:test image that I will use as a base image:

C:\Users\Jan\sample>nerdctl build -t testing:latest .
[+] Building 0.2s (3/3) FINISHED
 => [internal] load build definition from Dockerfile                                                               0.1s
 => => transferring dockerfile: 95B                                                                                0.0s
 => [internal] load .dockerignore                                                                                  0.1s
 => => transferring context: 2B                                                                                    0.0s
 => ERROR [internal] load metadata for registry.home/nginx:test                                                    0.1s
------
 > [internal] load metadata for registry.home/nginx:test:
------
Dockerfile:1
--------------------
   1 | >>> FROM registry.home/nginx:test
   2 |     ADD Dockerfile DOCKERFILE
   3 |
--------------------
error: failed to solve: registry.home/nginx:test: failed to do request: Head "https://registry.home/v2/nginx/manifests/test": x509: certificate signed by unknown authority
FATA[0000] unrecognized image format

As expected, this failed because I have not yet installed the CA that I used to sign the certificate.

So I use the “Certificate Import Wizard” (from “Control Panel”) to add my CA to the trust store:

Afterwards I stop Rancher Desktop (via “File | Exit”) and start it again, because certs are only imported into the distro during startup. Then I try again:

C:\Users\Jan\sample>nerdctl build -t testing:latest .
[+] Building 0.3s (3/3) FINISHED
 => [internal] load build definition from Dockerfile                                                               0.1s
 => => transferring dockerfile: 95B                                                                                0.1s
 => [internal] load .dockerignore                                                                                  0.1s
 => => transferring context: 2B                                                                                    0.0s
 => ERROR [internal] load metadata for registry.home/nginx:test                                                    0.1s
------
 > [internal] load metadata for registry.home/nginx:test:
------
Dockerfile:1
--------------------
   1 | >>> FROM registry.home/nginx:test
   2 |     ADD Dockerfile DOCKERFILE
   3 |
--------------------
error: failed to solve: registry.home/nginx:test: pulling from host registry.home failed with status code [manifests test]: 401 Unauthorized
FATA[0001] unrecognized image format

It still fails, because I haven’t authenticated with the registry yet, but the certificate error is gone. So I log in, and try once more:

C:\Users\Jan\sample>nerdctl login registry.home
Enter Username: user
Enter Password:
Login Succeeded

C:\Users\Jan\sample>nerdctl build -t testing:latest .
[+] Building 5.3s (8/8) FINISHED
 => [internal] load build definition from Dockerfile                                                               0.1s
 => => transferring dockerfile: 95B                                                                                0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load metadata for registry.home/nginx:test                                                          0.1s
 => [auth] sharing credentials for registry.home                                                                   0.0s
 => [internal] load build context                                                                                  0.0s
 => => transferring context: 31B                                                                                   0.0s
 => [1/2] FROM registry.home/nginx:test@sha256:ee89b00528ff4f02f2405e4ee221743ebc3f8e8dd0bfd5c4c20a2fa2aaa7ede3    0.0s
 => => resolve registry.home/nginx:test@sha256:ee89b00528ff4f02f2405e4ee221743ebc3f8e8dd0bfd5c4c20a2fa2aaa7ede3    0.0s
 => CACHED [2/2] ADD Dockerfile DOCKERFILE                                                                         0.0s
 => exporting to oci image format                                                                                  5.1s
 => => exporting layers                                                                                            0.0s
 => => exporting manifest sha256:b2cc4ecafca7094099f83a5e08612c7ecc542040cccb993ef01243dd63406d40                  0.0s
 => => exporting config sha256:a231badb0e5071c5ce2bac5220ad538295ca8b4ae40aa983af0b1dc251fbb2a9                    0.0s
 => => sending tarball                                                                                             5.0s
unpacking docker.io/library/testing:latest (sha256:b2cc4ecafca7094099f83a5e08612c7ecc542040cccb993ef01243dd63406d40)...done

C:\Users\Jan\sample>nerdctl images
REPOSITORY    TAG       IMAGE ID        CREATED           PLATFORM       SIZE
testing       latest    b2cc4ecafca7    17 minutes ago    linux/amd64    149.1 MiB

Everything is as I expected.

If you still have problems connecting, can you follow the same steps and report back where you get different results?

Is it feasible for Rancher Desktop or kim to start the builder daemonset by recognizing the root CA certificate on the host machine automatically?

I understand that manual workaround is possible. That said, for any company with sizable number of developers, asking each developer to apply the workaround manually and reinstate the workaround after each restart is not a feasible and scalable solution.

I upgraded my RD to 0.7.1, and saw that the DaemonSet got updated and removed the changes. I had to edit the DaemonSet and add them back, but after doing that it is still working for me.

Are you able to curl to your registry from the builder pod? You’ll need to add it with apk add curl. Mine is able to reach my registry without errors. Make sure you have all of the root and intermediate CAs included in the secret.

If you are going to release note this as a workaround, you’ll need to add that the DaemonSet will need to be updated whenever you upgrade Rancher Desktop. This makes it less user-friendly.