kind: coredns stuck at ContainerCreating

What happened: coredns stuck at ContainerCreating What you expected to happen: coredns Running How to reproduce it (as minimally and precisely as possible): Install and create cluster based on the documents of Kind Check the pod status: kubectl -n kube-system get pod Anything else we need to know?: Finally, my issue fixed by follow this comment: https://github.com/kubernetes/kubernetes/issues/48798#issuecomment-427454190 I wonder it could be resolve in Kine side Environment:

kind version: (use kind version): v0.3.0
Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:23:09Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-17T00:58:35Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

Docker version: (use docker info):

Client:
 Version:           18.09.5
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        e8ff056
 Built:             Thu May  9 23:11:19 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.5
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       e8ff056
  Built:            Thu May  9 22:59:19 2019
  OS/Arch:          linux/amd64
  Experimental:     false

OS (e.g. from /etc/os-release): Ubuntu 18.04.2 LTS

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 2
Comments: 16 (9 by maintainers)

Most upvoted comments

Seems that there is race-condition and sometimes the config template is not processed and is written to the cni configuration folder directly.

🤔 Seems that only (or is more likely to happen) on single nodes clusters.

root@test-control-plane:/# more /etc/cni/net.d/10-kindnet.conflist
{
  "cniVersion": "0.3.1",
  "name": "kindnet",
  "plugins": [
    {
      "type": "bridge",
      "ipMasq": false,
      "isDefaultGateway": true,
      "hairpinMode": true,
      "ipam": {
        "type": "host-local",
        "dataDir": "/run/cni-ipam-state",
        "routes": [
          {"dst": "0.0.0.0/0"},
          {"dst": "::/0"}
        ],
        "ranges": [
          [
            {
              "subnet": "{{ .PodCIDR }}"
            }
          ]
        ]
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}

The symptom is that the coredns pods are stuck

 kubectl get pods --all-namespaces
NAMESPACE     NAME                                         READY   STATUS              RESTARTS   AGE
kube-system   coredns-fb8b8dccf-6bvht                      0/1     ContainerCreating   0          49m
kube-system   coredns-fb8b8dccf-g2sc2                      0/1     ContainerCreating   0          49m
kube-system   etcd-test-control-plane                      1/1     Running             0          48m
kube-system   ip-masq-agent-fclwt                          1/1     Running             0          49m
kube-system   kindnet-whshf                                1/1     Running             1          49m
kube-system   kube-apiserver-test-control-plane            1/1     Running             0          48m
kube-system   kube-controller-manager-test-control-plane   1/1     Running             0          48m
kube-system   kube-proxy-pff9f                             1/1     Running             0          49m
kube-system   kube-scheduler-test-control-plane            1/1     Running             0          48m

The kindnet daemon is working but not updating the cni conf with the PodCIDR

 kubectl -n kube-system logs kindnet-whshf | more
hostIP = 172.17.0.5
podIP = 172.17.0.5
Handling node with IP: 172.17.0.5
handling current node

The node status is ready, however I see this erros in the logs

 = failed to setup network for sandbox "d84a2f8a2834f8a70e718c0f34ae65038c50460ef7f1cbf084def03106c50607": invalid CIDR address: {{ .PodCIDR }}
Jun 04 10:05:37 test-control-plane kubelet[218]: E0604 10:05:37.863761     218 kuberuntime_manager.go:693] createPodSandbox for pod "coredns-fb8b8dccf-g2sc2_kube-system(ade35eeb-86ab-11e9-888c-0242ac110005)" failed: rpc error: code = Unknown des

aojea on Jun 4, 2019

I create another cluster named test in the same host, get the same error:

root@dignify1:~# kubectl -n kube-system get pod
NAME                                         READY   STATUS              RESTARTS   AGE
coredns-fb8b8dccf-g47jc                      0/1     ContainerCreating   0          2m36s
coredns-fb8b8dccf-zbt69                      0/1     ContainerCreating   0          2m36s
etcd-test-control-plane                      1/1     Running             0          106s
ip-masq-agent-zjdd6                          1/1     Running             0          2m36s
kindnet-5npzn                                1/1     Running             1          2m36s
kube-apiserver-test-control-plane            1/1     Running             0          103s
kube-controller-manager-test-control-plane   1/1     Running             0          115s
kube-proxy-q85qd                             1/1     Running             0          2m36s
kube-scheduler-test-control-plane            1/1     Running             0          94s

logs.zip

vincent-pli on Jun 4, 2019