k3s: k3s server started but pods are not running

Environmental Info: K3s Version:

k3s version v1.22.5+k3s1 (405bf79d) go version go1.16.10

Node(s) CPU architecture, OS, and Version:

Linux rancher 5.4.0-91-generic #102-Ubuntu SMP Fri Nov 5 16:31:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

1-server cluster

Describe the bug:

After k3s server starts successfully, pods in kube-system namespace are stuck at ContainerCreating state.

Steps To Reproduce:

• Installed K3s:
• Followed steps in rancher k3s installation guide: https://rancher.com/docs/rancher/v2.6/en/installation/resources/k8s-tutorials/ha-with-external-db/

Expected behavior:

pods should be running

Actual behavior:

pods are stuck at ContainerCreating state

Additional context / logs:

Everything works okay if I add ‘–docker’ parameter. So I guess this problem is related with containerd. Logs:

1. Logs from stdout E1231 15:31:42.644385 18756 kuberuntime_manager.go:818] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to get sandbox image \"rancher/mirrored-pause:3.1\": failed to pull image \"rancher/mirrored-pause:3.1\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.1\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.1\": failed to do request: Head \"https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.1\": dial tcp: lookup registry-1.docker.io: no such host" pod="kube-system/local-path-provisioner-64ffb68fd-5p9hf"

2. kubectl get pods -n kube-system

coredns-85cb69466-j75wt                  0/1     ContainerCreating   1          19m
traefik-786ff64748-r28g7                 0/1     ContainerCreating   1          17m
svclb-traefik-5mc72                      0/2     ContainerCreating   2          17m
metrics-server-9cf544f65-grqnq           0/1     ContainerCreating   1          19m
local-path-provisioner-64ffb68fd-5p9hf   0/1     ContainerCreating   1          19m

1. kubectl describe pod/coredns-85cb69466-j75wt -n kube-system

Name:                 coredns-85cb69466-j75wt
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 rancher/10.0.2.15
Start Time:           Fri, 31 Dec 2021 15:13:23 +0000
Labels:               k8s-app=kube-dns
                      pod-template-hash=85cb69466
Annotations:          <none>
Status:               Running
IP:                   
IPs:                  <none>
Controlled By:        ReplicaSet/coredns-85cb69466
Containers:
  coredns:
    Container ID:  
    Image:         rancher/mirrored-coredns-coredns:1.8.4
    Image ID:      
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Waiting
      Reason:       ContainerCreating
    Last State:     Terminated
      Reason:       ContainerStatusUnknown
      Message:      The container could not be located when the pod was deleted.  The container used to be Running
      Exit Code:    137
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Mon, 01 Jan 0001 00:00:00 +0000
    Ready:          False
    Restart Count:  1
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=2s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hgpds (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  kube-api-access-hgpds:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              beta.kubernetes.io/os=linux
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                             node-role.kubernetes.io/master:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Normal   Scheduled               20m                  default-scheduler  Successfully assigned kube-system/coredns-85cb69466-j75wt to rancher
  Warning  FailedCreatePodSandBox  20m (x2 over 20m)    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "rancher/mirrored-pause:3.1": failed to pull image "rancher/mirrored-pause:3.1": failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.1": failed to resolve reference "docker.io/rancher/mirrored-pause:3.1": failed to do request: Head "https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.1": dial tcp: lookup registry-1.docker.io: no such host
  Normal   Pulling                 19m                  kubelet            Pulling image "rancher/mirrored-coredns-coredns:1.8.4"
  Normal   Pulled                  18m                  kubelet            Successfully pulled image "rancher/mirrored-coredns-coredns:1.8.4" in 44.483720004s
  Normal   Created                 18m                  kubelet            Created container coredns
  Normal   Started                 18m                  kubelet            Started container coredns
  Warning  FailedCreatePodSandBox  9s (x17 over 3m41s)  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "rancher/mirrored-pause:3.1": failed to pull image "rancher/mirrored-pause:3.1": failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.1": failed to resolve reference "docker.io/rancher/mirrored-pause:3.1": failed to do request: Head "https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.1": dial tcp: lookup registry-1.docker.io: no such host

Backporting

• Needs backporting to older releases