kubernetes: kube-dns - Failed create pod sandbox

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened:

cluster initialized with kubeadm, kube-dns pods hang in ContainerCreating status. With failing pod sandbox creation.

What you expected to happen:

kube-dns pods running

How to reproduce it (as minimally and precisely as possible): created cluster via kubeadm:

sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --token-ttl=0 --token=bd11ac.54147b1b3fd9620d --apiserver-cert-extra-sans=kube,kube.internal

kubectl taint nodes kube node-role.kubernetes.io/master-

with flannel network plugin:

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

checked also with kube-router with same problem:

kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml

Anything else we need to know?:

NAMESPACE     NAME                           READY     STATUS              RESTARTS   AGE       IP            NODE
kube-system   etcd-kube                      1/1       Running             2          48m       10.10.10.12   kube
kube-system   kube-apiserver-kube            1/1       Running             2          48m       10.10.10.12   kube
kube-system   kube-controller-manager-kube   1/1       Running             2          49m       10.10.10.12   kube
kube-system   kube-dns-545bc4bfd4-bskll      0/3       ContainerCreating   0          49m       <none>        kube
kube-system   kube-flannel-ds-h9mcw          1/1       Running             2          43m       10.10.10.12   kube
kube-system   kube-proxy-f75q9               1/1       Running             2          49m       10.10.10.12   kube
kube-system   kube-scheduler-kube            1/1       Running             2          49m       10.10.10.12   kube

Name:           kube-dns-545bc4bfd4-bskll
Namespace:      kube-system
Node:           kube/10.10.10.12
Start Time:     Wed, 06 Dec 2017 23:58:55 +0300
Labels:         k8s-app=kube-dns
                pod-template-hash=1016706980
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"kube-dns-545bc4bfd4","uid":"69d564df-dac7-11e7-9ac1-00155d02...
Status:         Pending
IP:
Created By:     ReplicaSet/kube-dns-545bc4bfd4
Controlled By:  ReplicaSet/kube-dns-545bc4bfd4
Containers:
  kubedns:
    Container ID:
    Image:         gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5
    Image ID:
    Ports:         10053/UDP, 10053/TCP, 10055/TCP
    Args:
      --domain=cluster.local.
      --dns-port=10053
      --config-dir=/kube-dns-config
      --v=2
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:      100m
      memory:   70Mi
    Liveness:   http-get http://:10054/healthcheck/kubedns delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:  http-get http://:8081/readiness delay=3s timeout=5s period=10s #success=1 #failure=3
    Environment:
      PROMETHEUS_PORT:  10055
    Mounts:
      /kube-dns-config from kube-dns-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-jsxv8 (ro)
  dnsmasq:
    Container ID:
    Image:         gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5
    Image ID:
    Ports:         53/UDP, 53/TCP
    Args:
      -v=2
      -logtostderr
      -configDir=/etc/k8s/dns/dnsmasq-nanny
      -restartDnsmasq=true
      --
      -k
      --cache-size=1000
      --log-facility=-
      --server=/cluster.local/127.0.0.1#10053
      --server=/in-addr.arpa/127.0.0.1#10053
      --server=/ip6.arpa/127.0.0.1#10053
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        150m
      memory:     20Mi
    Liveness:     http-get http://:10054/healthcheck/dnsmasq delay=60s timeout=5s period=10s #success=1 #failure=5
    Environment:  <none>
    Mounts:
      /etc/k8s/dns/dnsmasq-nanny from kube-dns-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-jsxv8 (ro)
  sidecar:
    Container ID:
    Image:         gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5
    Image ID:
    Port:          10054/TCP
    Args:
      --v=2
      --logtostderr
      --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A
      --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        10m
      memory:     20Mi
    Liveness:     http-get http://:10054/metrics delay=60s timeout=5s period=10s #success=1 #failure=5
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-jsxv8 (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
Volumes:
  kube-dns-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kube-dns
    Optional:  true
  kube-dns-token-jsxv8:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kube-dns-token-jsxv8
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.alpha.kubernetes.io/notReady:NoExecute for 300s
                 node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age                 From               Message
  ----     ------                  ----                ----               -------
  Warning  FailedScheduling        42m (x22 over 47m)  default-scheduler  No nodes are available that match all of the predicates: NodeNotReady (1).
  Normal   Scheduled               41m                 default-scheduler  Successfully assigned kube-dns-545bc4bfd4-bskll to kube
  Normal   SuccessfulMountVolume   41m                 kubelet, kube      MountVolume.SetUp succeeded for volume "kube-dns-config"
  Normal   SuccessfulMountVolume   41m                 kubelet, kube      MountVolume.SetUp succeeded for volume "kube-dns-token-jsxv8"
  Warning  FailedCreatePodSandBox  41m                 kubelet, kube      Failed create pod sandbox.
  Warning  FailedSync              39m (x11 over 41m)  kubelet, kube      Error syncing pod
  Normal   SandboxChanged          36m (x25 over 41m)  kubelet, kube      Pod sandbox changed, it will be killed and re-created.
  Normal   SuccessfulMountVolume   30m                 kubelet, kube      MountVolume.SetUp succeeded for volume "kube-dns-config"
  Normal   SuccessfulMountVolume   30m                 kubelet, kube      MountVolume.SetUp succeeded for volume "kube-dns-token-jsxv8"
  Warning  FailedSync              28m (x11 over 30m)  kubelet, kube      Error syncing pod
  Normal   SandboxChanged          25m (x25 over 30m)  kubelet, kube      Pod sandbox changed, it will be killed and re-created.
  Normal   SuccessfulMountVolume   24m                 kubelet, kube      MountVolume.SetUp succeeded for volume "kube-dns-config"
  Normal   SuccessfulMountVolume   24m                 kubelet, kube      MountVolume.SetUp succeeded for volume "kube-dns-token-jsxv8"
  Warning  FailedSync              22m (x11 over 24m)  kubelet, kube      Error syncing pod
  Normal   SandboxChanged          4m (x94 over 24m)   kubelet, kube      Pod sandbox changed, it will be killed and re-created.
  Normal   SuccessfulMountVolume   3m                  kubelet, kube      MountVolume.SetUp succeeded for volume "kube-dns-config"
  Normal   SuccessfulMountVolume   3m                  kubelet, kube      MountVolume.SetUp succeeded for volume "kube-dns-token-jsxv8"
  Warning  FailedSync              1m (x11 over 3m)    kubelet, kube      Error syncing pod
  Normal   SandboxChanged          48s (x12 over 3m)   kubelet, kube      Pod sandbox changed, it will be killed and re-created.

Dec 07 10:30:44 kube kubelet[1209]: W1207 10:30:44.077574    1209 docker_sandbox.go:343] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "kube-dns-545bc4bfd4-bskll_kube-system": CNI failed to retrieve network namespace path: Cannot find network namespace for the terminated container "5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6"
Dec 07 10:30:44 kube kubelet[1209]: W1207 10:30:44.078511    1209 cni.go:265] CNI failed to retrieve network namespace path: Cannot find network namespace for the terminated container "5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6"
Dec 07 10:30:44 kube kubelet[1209]: E1207 10:30:44.078791    1209 cni.go:319] Error deleting network: failed to find plugin "portmap" in path [/opt/flannel/bin /opt/cni/bin]
Dec 07 10:30:44 kube kubelet[1209]: E1207 10:30:44.079367    1209 remote_runtime.go:115] StopPodSandbox "5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "kube-dns-545bc4bfd4-bskll_kube-system" network: failed to find plugin "portmap" in path [/opt/flannel/bin /opt/cni/bin]
Dec 07 10:30:44 kube kubelet[1209]: E1207 10:30:44.079564    1209 kuberuntime_manager.go:781] Failed to stop sandbox {"docker" "5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6"}
Dec 07 10:30:44 kube kubelet[1209]: E1207 10:30:44.079741    1209 kuberuntime_manager.go:581] killPodWithSyncResult failed: failed to "KillPodSandbox" for "69d619f2-dac7-11e7-9ac1-00155d02520b" with KillPodSandboxError: "rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod \"kube-dns-545bc4bfd4-bskll_kube-system\" network: failed to find plugin \"portmap\" in path [/opt/flannel/bin /opt/cni/bin]"
Dec 07 10:30:44 kube kubelet[1209]: E1207 10:30:44.079912    1209 pod_workers.go:182] Error syncing pod 69d619f2-dac7-11e7-9ac1-00155d02520b ("kube-dns-545bc4bfd4-bskll_kube-system(69d619f2-dac7-11e7-9ac1-00155d02520b)"), skipping: failed to "KillPodSandbox" for "69d619f2-dac7-11e7-9ac1-00155d02520b" with KillPodSandboxError: "rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod \"kube-dns-545bc4bfd4-bskll_kube-system\" network: failed to find plugin \"portmap\" in path [/opt/flannel/bin /opt/cni/bin]"
Dec 07 10:30:51 kube kubelet[1209]: W1207 10:30:51.086671    1209 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Dec 07 10:30:56 kube kubelet[1209]: W1207 10:30:56.077148    1209 docker_sandbox.go:343] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "kube-dns-545bc4bfd4-bskll_kube-system": CNI failed to retrieve network namespace path: Cannot find network namespace for the terminated container "5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6"
Dec 07 10:30:56 kube kubelet[1209]: W1207 10:30:56.077596    1209 cni.go:265] CNI failed to retrieve network namespace path: Cannot find network namespace for the terminated container "5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6"
Dec 07 10:30:56 kube kubelet[1209]: E1207 10:30:56.077911    1209 cni.go:319] Error deleting network: failed to find plugin "portmap" in path [/opt/flannel/bin /opt/cni/bin]
Dec 07 10:30:56 kube kubelet[1209]: E1207 10:30:56.078520    1209 remote_runtime.go:115] StopPodSandbox "5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "kube-dns-545bc4bfd4-bskll_kube-system" network: failed to find plugin "portmap" in path [/opt/flannel/bin /opt/cni/bin]
Dec 07 10:30:56 kube kubelet[1209]: E1207 10:30:56.078545    1209 kuberuntime_manager.go:781] Failed to stop sandbox {"docker" "5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6"}
Dec 07 10:30:56 kube kubelet[1209]: E1207 10:30:56.078573    1209 kuberuntime_manager.go:581] killPodWithSyncResult failed: failed to "KillPodSandbox" for "69d619f2-dac7-11e7-9ac1-00155d02520b" with KillPodSandboxError: "rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod \"kube-dns-545bc4bfd4-bskll_kube-system\" network: failed to find plugin \"portmap\" in path [/opt/flannel/bin /opt/cni/bin]"
Dec 07 10:30:56 kube kubelet[1209]: E1207 10:30:56.078590    1209 pod_workers.go:182] Error syncing pod 69d619f2-dac7-11e7-9ac1-00155d02520b ("kube-dns-545bc4bfd4-bskll_kube-system(69d619f2-dac7-11e7-9ac1-00155d02520b)"), skipping: failed to "KillPodSandbox" for "69d619f2-dac7-11e7-9ac1-00155d02520b" with KillPodSandboxError: "rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod \"kube-dns-545bc4bfd4-bskll_kube-system\" network: failed to find plugin \"portmap\" in path [/opt/flannel/bin /opt/cni/bin]"
Dec 07 10:31:01 kube kubelet[1209]: W1207 10:31:01.096087    1209 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Dec 07 10:31:09 kube kubelet[1209]: W1207 10:31:09.077450    1209 docker_sandbox.go:343] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "kube-dns-545bc4bfd4-bskll_kube-system": CNI failed to retrieve network namespace path: Cannot find network namespace for the terminated container "5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6"
Dec 07 10:31:09 kube kubelet[1209]: W1207 10:31:09.078299    1209 cni.go:265] CNI failed to retrieve network namespace path: Cannot find network namespace for the terminated container "5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6"
Dec 07 10:31:09 kube kubelet[1209]: E1207 10:31:09.078626    1209 cni.go:319] Error deleting network: failed to find plugin "portmap" in path [/opt/flannel/bin /opt/cni/bin]

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:28:34Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:17:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:

vm hosted on hyper-v with 4 cores, 10240MB ram configuration

• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="17.04 (Zesty Zapus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 17.04"
VERSION_ID="17.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=zesty
UBUNTU_CODENAME=zesty

• Kernel (e.g. uname -a):

Linux kube 4.10.0-40-generic #44-Ubuntu SMP Thu Nov 9 14:49:09 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:

kubeadm 1.8.4

• Others:

docker version

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:45 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:41:24 2017
 OS/Arch:      linux/amd64
 Experimental: false