microk8s: Microk8s install can fail with docker hub rate limiting

When installing Microk8s from a common infrastructure where many users share the same external IP address, it is very easy to trigger the new DockerHub pull rate limit policy. Once that happens, it is no longer possible to install Microk8s.

Example output from microk8s.kubectl describe pod -n kube-system calico-node-mghbl:

  Normal   Scheduled  14m                   default-scheduler  Successfully assigned kube-system/calico-node-mghbl to cicd3
  Normal   Pulling    12m (x4 over 14m)     kubelet            Pulling image "calico/cni:v3.13.2"
  Warning  Failed     12m (x4 over 14m)     kubelet            Error: ErrImagePull
  Warning  Failed     12m (x4 over 14m)     kubelet            Failed to pull image "calico/cni:v3.13.2": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.13.2": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:bbf7e3ac3f80d0a356a6c27b095bd313d1106f8ed84f85850816ed79295843c1: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
  Normal   BackOff    12m (x7 over 14m)     kubelet            Back-off pulling image "calico/cni:v3.13.2"
  Warning  Failed     4m33s (x39 over 14m)  kubelet            Error: ImagePullBackOff

inspection-report-20201104_180022.tar.gz

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 2
Comments: 19 (4 by maintainers)

Most upvoted comments

Another approach to work around the dockerhub late limit is to have containerd login with a dockerhub account. That should involve updating /var/snap/microk8s/current/args/containerd-template.toml and restarting microk8s.stop; microk8s.start. Haven’t tried it though.

Side loading images is something that would be very beneficial in the case of off-line deployments. Although I suspect you will have a private registry in such setups.

On the long run we should address the the dockerhub rate limit by moving some critical/often-used images into the https://rocks.canonical.com image registry.

ktsakalozos on Nov 5, 2020

Hi, I’ve also encountered and struggled a lot on this issue and managed to find a way to add auth for docker’s repositry. As mentioned @ktsakalozos, you can edit your /var/snap/microk8s/current/args/containerd-template.toml to add authentication to your docker repository. The thing is that containerd’s version has changed since microk8s 1.19 (from 1.2.5 to 1.3.7) so its configuration too. You can configure your authentication like so :

...
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
        endpoint = ["https://registry-1.docker.io", ]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:32000"]
        endpoint = ["http://localhost:32000"]
    [plugins."io.containerd.grpc.v1.cri".registry.configs]
      [plugins."io.containerd.grpc.v1.cri".registry.configs."registry-1.docker.io".auth]
        username = "xxxxx"
        password = "xxxxx"
...

The “key” you mention for the “configs” part should match your hostname used in the endpoint array (without “https://”).

Then you should restart your microk8s with microk8s.stop; microk8s.start.

IanniF on Oct 5, 2021

Any updates on this? The docker hub rate limits are really annoying.

aintbrokedontfix on Sep 14, 2021

Once microk8s is installed, this will update it to use a specific registry for all further requests, but it does not solve the initial problem of installation if the rate limit has already been triggered:

sudo snap install microk8s --classic
sudo microk8s.status --wait-ready
sudo sed -i "s|https://registry-1.docker.io|http://YOUR_MIRROR:MIRROR_PORT|" /var/snap/microk8s/current/args/containerd-template.toml
sudo systemctl restart snap.microk8s.daemon-containerd.service

markbeierl on Nov 9, 2020