coredns: CoreDns pods are stuck with status ContainerCreating

What happened: NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-6955765f44-54pzh 0/1 ContainerCreating 0 28m kube-system coredns-6955765f44-t9bqw 0/1 ContainerCreating 0 28m

*Pods stuck at ContainerCreating

What you expected to happen: NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-6955765f44-54pzh 1/1 Running 0 28m kube-system coredns-6955765f44-t9bqw 1/1 Running 0 28m

How to reproduce it (as minimally and precisely as possible): I tried to install kubernetes 1.17 with kubeadm and flannel as a CNI resource. And I am not able to bring the coredns pods in running state. ipv6 is enabled, swap memory is off and every other pre check for installing kubernetes is checked with the output as follows :-

NAME STATUS ROLES AGE VERSION indrwfkub.dryicelabs.com Ready master 2d23h v1.17.3

Anything else we need to know?: CNI used Flannel Container Runtime Docker

Environment:

• the version of CoreDNS:

• Corefile: k8s.gcr.io/coredns:1.6.5

• logs, if applicable: ##################################################### Name: coredns-6955765f44-54pzh Namespace: kube-system Priority: 2000000000 Priority Class Name: system-cluster-critical Node: indrwfkub.dryicelabs.com/10.1.50.136 Start Time: Mon, 17 Feb 2020 16:50:13 +0530 Labels: k8s-app=kube-dns pod-template-hash=6955765f44 Annotations: <none> Status: Pending IP: IPs: <none> Controlled By: ReplicaSet/coredns-6955765f44 Containers: coredns: Container ID: Image: k8s.gcr.io/coredns:1.6.5 Image ID: Ports: 53/UDP, 53/TCP, 9153/TCP Host Ports: 0/UDP, 0/TCP, 0/TCP Args: -conf /etc/coredns/Corefile State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Limits: memory: 170Mi Requests: cpu: 100m memory: 70Mi Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5 Readiness: http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3 Environment: <none> Mounts: /etc/coredns from config-volume (ro) /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-2zfcw (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: config-volume: Type: ConfigMap (a volume populated by a ConfigMap) Name: coredns Optional: false coredns-token-2zfcw: Type: Secret (a volume populated by a Secret) SecretName: coredns-token-2zfcw Optional: false QoS Class: Burstable Node-Selectors: beta.kubernetes.io/os=linux Tolerations: CriticalAddonsOnly node-role.kubernetes.io/master:NoSchedule node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message

  ---

  Normal Scheduled 40m default-scheduler Successfully assigned kube-system/coredns-6955765f44-54pzh to indrwfkub.dryicelabs.com Warning FailedCreatePodSandBox 40m kubelet, indrwfkub.dryicelabs.com Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “dddd0d7b48b13b7be556727ab6779448003e05467387097ec4a85e7035f41844” network for pod “coredns-6955765f44-54pzh”: networkPlugin cni failed to set up pod “coredns-6955765f44-54pzh_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/ Warning FailedCreatePodSandBox 40m kubelet, indrwfkub.dryicelabs.com Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “b2c5b47f5723db3eff548f1dcf61744af3b0abbba587fa014a6b3abdd9ad0746” network for pod “coredns-6955765f44-54pzh”: networkPlugin cni failed to set up pod “coredns-6955765f44-54pzh_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/ Warning FailedCreatePodSandBox 40m kubelet, indrwfkub.dryicelabs.com Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “e6ed824635d8bb4dcdddbf89cdb3b72ae1e4cf558b8a265dfd31d13346aa48e2” network for pod “coredns-6955765f44-54pzh”: networkPlugin cni failed to set up pod “coredns-6955765f44-54pzh_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/ Warning FailedCreatePodSandBox 40m kubelet, indrwfkub.dryicelabs.com Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “0cc3db192cdf414380a4a0f80089703c896bfb78282457676f795f3fd7d75f2c” network for pod “coredns-6955765f44-54pzh”: networkPlugin cni failed to set up pod “coredns-6955765f44-54pzh_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/ Warning FailedCreatePodSandBox 40m kubelet, indrwfkub.dryicelabs.com Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “d8850d246b450960cf237f48a7fe876bb2c1d3db3139e36d2550c92edae15d1d” network for pod “coredns-6955765f44-54pzh”: networkPlugin cni failed to set up pod “coredns-6955765f44-54pzh_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/ Warning FailedCreatePodSandBox 40m kubelet, indrwfkub.dryicelabs.com Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “9cabce9aeb9bec55794d4089a65be75172e803d39751575683d32a54bfad2b3c” network for pod “coredns-6955765f44-54pzh”: networkPlugin cni failed to set up pod “coredns-6955765f44-54pzh_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/ Warning FailedCreatePodSandBox 40m kubelet, indrwfkub.dryicelabs.com Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “9efe64985e3ac3499217bca3b7a6f869fb08c6b58bb24147db7ecfc20cf45e3c” network for pod “coredns-6955765f44-54pzh”: networkPlugin cni failed to set up pod “coredns-6955765f44-54pzh_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/ Warning FailedCreatePodSandBox 40m kubelet, indrwfkub.dryicelabs.com Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “fd956061710a06249ece03ec8343cc50b88f69533c004460f85be152bd34ba67” network for pod “coredns-6955765f44-54pzh”: networkPlugin cni failed to set up pod “coredns-6955765f44-54pzh_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/ Warning FailedCreatePodSandBox 40m kubelet, indrwfkub.dryicelabs.com Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “806734d38d1dc01a77f06f84eb5c8f5d1a7854f6b0172435d47a27c4e5628b27” network for pod “coredns-6955765f44-54pzh”: networkPlugin cni failed to set up pod “coredns-6955765f44-54pzh_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/ Warning FailedCreatePodSandBox 25m (x818 over 40m) kubelet, indrwfkub.dryicelabs.com (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “9f805bdb23481a37cc0f86227458e9718f6b24e44dfe6023cb71fdc47e5b0a0f” network for pod “coredns-6955765f44-54pzh”: networkPlugin cni failed to set up pod “coredns-6955765f44-54pzh_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/ Normal SandboxChanged 39s (x2205 over 40m) kubelet, indrwfkub.dryicelabs.com Pod sandbox changed, it will be killed and re-created. #################################################################

• OS (e.g: cat /etc/os-release): NAME=“CentOS Linux” VERSION=“7 (Core)” ID=“centos” ID_LIKE=“rhel fedora” VERSION_ID=“7” PRETTY_NAME=“CentOS Linux 7 (Core)” ANSI_COLOR=“0;31” CPE_NAME=“cpe:/o:centos:centos:7” HOME_URL=“https://www.centos.org/” BUG_REPORT_URL=“https://bugs.centos.org/”

CENTOS_MANTISBT_PROJECT=“CentOS-7” CENTOS_MANTISBT_PROJECT_VERSION=“7” REDHAT_SUPPORT_PRODUCT=“centos” REDHAT_SUPPORT_PRODUCT_VERSION=“7”

• Others: I checked the following issues but didn’t get any help from these :-

https://github.com/coredns/deployment/issues/87 https://github.com/kubernetes-sigs/kind/issues/586 https://github.com/kubernetes/kubernetes/issues/80771