kubeadm: Coredns pods on master are in CrashLoopBackOff after setting up pod networking

Thanks @anushakamath97 I guess you’ve encountered a bug with CoreDNS on ubuntu.

$ kubectl logs -f coredns-fb8b8dccf-jgsqf -n kube-system
...
2019-07-26T05:12:51.129Z [FATAL] plugin/loop: Loop (127.0.0.1:37926 -> :53) detected for zone ".", see https://coredns.io/plugins/loop#troubleshooting. Query: "HINFO 6687652606367193539.3502358703036269394."

You need to manually edit CoreDNS ConfigMap:

$ kubectl edit cm coredns -n kube-system

Then CoreDNS Pod should can work when it restarts.

try a different CNI plugin instead of flannel. i recommend Calico or WeaveNet.

Thanks @SataQiu. The coredns pods show running but there is Readiness probe failure error in the pod. I am running a predator application as well.

kubectl get pods --all-namespaces
NAMESPACE     NAME                                                              READY   STATUS      RESTARTS   AGE
default       predator.6ac899c2-8548-4e17-b174-b9fed22d0a4a-156427239352h6c9c   0/1     Completed   0          36m
default       predator.de2af166-7c80-4dc4-b6e3-5cd4b023ae93-156427264155dwc5l   0/1     Completed   0          32m
default       wintering-gorilla-predator-cf6c86458-6blqg                        1/1     Running     0          68m
kube-system   coredns-5c98db65d4-2z4nb                                          1/1     Running     6          4h36m
kube-system   coredns-5c98db65d4-zsb8v                                          1/1     Running     7          4h36m
kube-system   etcd-kmaster                                                      1/1     Running     2          4h43m
kube-system   kube-apiserver-kmaster                                            1/1     Running     2          4h43m
kube-system   kube-controller-manager-kmaster                                   1/1     Running     2          4h43m
kube-system   kube-proxy-89v7n                                                  1/1     Running     0          4h31m
kube-system   kube-proxy-rgt45                                                  1/1     Running     2          4h44m
kube-system   kube-scheduler-kmaster                                            1/1     Running     2          4h43m
kube-system   tiller-deploy-5f669f7664-hk79h                                    1/1     Running     0          80m
kube-system   weave-net-rsq9l                                                   2/2     Running     6          4h41m
kube-system   weave-net-v45qc                                                   2/2     Running     0          69m

Coredns pod describe

root@kmaster:~# kubectl describe pod coredns-5c98db65d4-2z4nb -n kube-system
Name:                 coredns-5c98db65d4-2z4nb
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 kmaster/10.112.187.83
Start Time:           Sun, 28 Jul 2019 01:36:23 +0530
Labels:               k8s-app=kube-dns
                      pod-template-hash=5c98db65d4
Annotations:          <none>
Status:               Running
IP:                   10.32.0.4
Controlled By:        ReplicaSet/coredns-5c98db65d4
Containers:
  coredns:
    Container ID:  docker://a13f51e2fc0e9f2ef707affd7221e2404139c5794badda6ae3846800ba54b40a
    Image:         k8s.gcr.io/coredns:1.3.1
    Image ID:      docker-pullable://k8s.gcr.io/coredns@sha256:02382353821b12c21b062c59184e227e001079bb13ebd01f9d3270ba0fcbf1e4
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Running
      Started:      Sun, 28 Jul 2019 05:41:52 +0530
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Sun, 28 Jul 2019 05:40:52 +0530
      Finished:     Sun, 28 Jul 2019 05:41:29 +0530
    Ready:          True
    Restart Count:  6
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8080/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-92254 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-92254:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-92254
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                 From              Message
  ----     ------     ----                ----              -------
  Warning  Unhealthy  28m                 kubelet, kmaster  Readiness probe failed: HTTP probe failed with statuscode: 503
  Warning  Unhealthy  27m (x2 over 27m)   kubelet, kmaster  Readiness probe failed: Get http://10.32.0.4:8080/health: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  Warning  BackOff    27m (x4 over 129m)  kubelet, kmaster  Back-off restarting failed container
  Normal   Created    27m (x6 over 151m)  kubelet, kmaster  Created container coredns
  Normal   Pulled     27m (x6 over 151m)  kubelet, kmaster  Container image "k8s.gcr.io/coredns:1.3.1" already present on machine
  Normal   Started    27m (x6 over 151m)  kubelet, kmaster  Started container coredns
  Warning  Unhealthy  27m (x3 over 129m)  kubelet, kmaster  Readiness probe failed: Get http://10.32.0.4:8080/health: dial tcp 10.32.0.4:8080: connect: connection refused

Thank you @SataQiu I have tried this and removed loop from the configmap. The problem still persists.

Here is the configmap of coredns

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           upstream
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        #loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2019-07-26T01:14:28Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "52884"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: ea4b5256-d6bd-4968-b922-07722d6eacb1