microk8s: upgrade to 1.14 seems to have broken my access to private registries

I got this to work!

I checked cri’s source code and saw this:

for _, e := range c.config.Registry.Mirrors[refspec.Hostname()].Endpoints

If we trace that further, the hostname comes from the ‘pull image request’. I wasn’t entirely sure what that is but I figured it could very well be the hostname of the image repository, so then I tried editing /var/snap/microk8s/current/args/containerd-template.toml with:

[plugins.cri.registry.mirrors."192.168.3.25:5000"]
          endpoint = ["http://192.168.3.25:5000"]

Restarted the service, tried deploying bbox.yaml again and it worked!

Bottom line is it seems we can’t just put any arbitrary text after plugins.cri.registry.mirrors., it has to match the host of the registry.

EDIT: after further investigation, I can confirm that it works like stated above. ParseNormalizedName will convert the image name into a fully qualified one that Docker can use unambiguously. When there’s no host name, docker.io is preppended to the name, hence the default [plugins.cri.registry.mirrors."docker.io"]. See test. This makes me wonder about https://github.com/containerd/cri/blob/master/docs/registry.md, which seems misleading as there’s no description about test.secure-registry.io and test.insecure-registry.io and it leads to think that you can put anything there.

I too can’t access my private registry and getting https://192.168.3.25:5000/v2/my-busybox/manifests/latest: http: server gave HTTP response to HTTPS client.

My setup is:

• Ubuntu server 18.04 (192.168.3.25)
• docker-ce 18.09.4
• microk8s 1.14.0

And I’ve got a private registry at port 5000. Note that this registry is not in Kubernetes.

I’ve edited /etc/docker/daemon.json to include "insecure-registries" : ["192.168.3.25:5000"], so I’m able to:

docker pull busybox
docker tag busybox 192.168.3.25:5000/my-busybox
docker push 192.168.3.25:5000/my-busybox

My /var/snap/microk8s/current/args/containerd-template.toml also includes

[plugins.cri.registry.mirrors."local.insecure-registry.io"]
          endpoint = ["http://192.168.3.25:5000"]

But then when I try to kubectl apply -f bbox.yaml

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: 192.168.3.25:5000/my-busybox
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

I get the dreaded Failed to pull image "192.168.3.25:5000/my-busybox:latest": rpc error: code = Unknown desc = failed to resolve image "192.168.3.25:5000/my-busybox:latest": no available registry endpoint: failed to do request: Head https://192.168.3.25:5000/v2/my-busybox/manifests/latest: http: server gave HTTP response to HTTPS client

After investigating for a while and going through https://github.com/containerd/containerd/issues/2758, I tried doing

$ microk8s.ctr image pull 192.168.3.25:5000/my-busybox:latest
ctr: failed to resolve reference "192.168.3.25:5000/my-busybox:latest": failed to do request: Head https://192.168.3.25:5000/v2/my-busybox/manifests/latest: http: server gave HTTP response to HTTPS client

But then if I try with --plain-http it works!

$ microk8s.ctr image pull --plain-http=true 192.168.3.25:5000/my-busybox:latest
192.168.3.25:5000/my-busybox:latest:                                              resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:f79f7a10302c402c052973e3fa42be0344ae6453245669783a9e16da3d56d5b4: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:fc1a6b909f82ce4b72204198d49de3aaf757b3ab2bb823cb6e47c416b97c5985:    done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:af2f74c517aac1d26793a6ed05ff45b299a037e1a9eefeae5eacda133e70a825:   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.3 s                                                                    total:  738.6  (2.4 MiB/s)
unpacking linux/amd64 sha256:f79f7a10302c402c052973e3fa42be0344ae6453245669783a9e16da3d56d5b4...
done

So it seems containerd within microk8s is not checking the insecure repository mirror entry. Is this a bug or is there anything else I could do?

@maltebaumann, thank you for the detailed description.

There are two things that look strange. First you never tell docker you are talking to an insecure registry yet you can push images there (https://docs.docker.com/registry/insecure/). Second I was able to correctly reference the my-busybox image like this image: localhost:31813/my-busybox while having

[plugins.cri.registry.mirrors."local.insecure-registry.io"]
          endpoint = ["http://127.0.0.1:31813"]

in contanerd configuration.

fwiw, to the folks saying “add the configuration the the containerd config”, that’s not how kubernetes works, i need to be able to set the values using secrets

Thanks to your hint I at least managed to get Polyaxon working again: After inspecting polyaxon/registry.py to find out where the 127.0.0.1 comes from, I adjusted the corresponding values in the global polyaxon-polyaxon-config ConfigMap: POLYAXON_REGISTRY_IN_CLUSTER = “false” # To force use of HOST & PORT POLYAXON_REGISTRY_HOST = “localhost” POLYAXON_REGISTRY_PORT = “31813”

I’m now able to run experiments like I did before, but don’t think this is a ‘clean’ solution.