kubernetes: Getting "runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"

What happened: Used kubeadm init --pod-network-cidr=10.244.0.0/16. And then executed suggested commands. Selected flannel pod-network: sysctl net.bridge.bridge-nf-call-iptables=1. wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml. kubectl apply -f kube-flannel.yml. kubectl get pods --all-namespaces: When I run kubectl get nodes I see the state of Master node as NotReady. Running kubectl describe nodes shows one error message “runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized” dockuser@Test-LPAR-Ubuntu:/var/log/containers$ kubectl describe nodes Name: test-lpar-ubuntu Roles: master Labels: beta.kubernetes.io/arch=ppc64le beta.kubernetes.io/os=linux kubernetes.io/hostname=test-lpar-ubuntu node-role.kubernetes.io/master= Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Tue, 27 Nov 2018 16:39:37 +0530 Taints: node-role.kubernetes.io/master:NoSchedule Unschedulable: false Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message

OutOfDisk False Tue, 04 Dec 2018 11:56:34 +0530 Tue, 27 Nov 2018 16:39:34 +0530 KubeletHasSufficientDisk kubelet has sufficient disk space available MemoryPressure False Tue, 04 Dec 2018 11:56:34 +0530 Tue, 27 Nov 2018 16:39:34 +0530 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Tue, 04 Dec 2018 11:56:34 +0530 Tue, 27 Nov 2018 16:39:34 +0530 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Tue, 04 Dec 2018 11:56:34 +0530 Tue, 27 Nov 2018 16:39:34 +0530 KubeletHasSufficientPID kubelet has sufficient PID available Ready False Tue, 04 Dec 2018 11:56:34 +0530 Tue, 27 Nov 2018 16:39:34 +0530 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Addresses: InternalIP: 10.23.172.121 Hostname: test-lpar-ubuntu Capacity: cpu: 16 ephemeral-storage: 94803804Ki hugepages-16Gi: 0 hugepages-16Mi: 0 memory: 4129728Ki pods: 110 Allocatable: cpu: 16 ephemeral-storage: 87371185622 hugepages-16Gi: 0 hugepages-16Mi: 0 memory: 4027328Ki pods: 110 System Info: Machine ID: 71c8632936a76f8c0b2353b95bfd159b System UUID: IBM,02785626X Boot ID: 44384248-711a-43d1-bf9b-c124a25269ed Kernel Version: 4.4.0-62-generic OS Image: Ubuntu 16.04.5 LTS Operating System: linux Architecture: ppc64le Container Runtime Version: docker://17.3.2 Kubelet Version: v1.12.3 Kube-Proxy Version: v1.12.3 Non-terminated Pods: (4 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits

kube-system etcd-test-lpar-ubuntu 0 (0%) 0 (0%) 0 (0%) 0 (0%) kube-system kube-apiserver-test-lpar-ubuntu 250m (1%) 0 (0%) 0 (0%) 0 (0%) kube-system kube-controller-manager-test-lpar-ubuntu 200m (1%) 0 (0%) 0 (0%) 0 (0%) kube-system kube-scheduler-test-lpar-ubuntu 100m (0%) 0 (0%) 0 (0%) 0 (0%) Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits

cpu 550m (3%) 0 (0%) memory 0 (0%) 0 (0%) Events: <none> dockuser@Test-LPAR-Ubuntu:

What you expected to happen: The state of master should be Ready

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?: I tried looking for CNI files in /etc/cni/net.d but there is nothing, this directory structure does not exists. journalctl -r -u kubelet throws this error: dockuser@Test-LPAR-Ubuntu:/etc/systemd/system/kubelet.service.d$ journalctl -r -u kubelet – Logs begin at Mon 2018-12-03 16:13:22 IST, end at Tue 2018-12-04 13:12:57 IST. – Dec 04 13:12:57 Test-LPAR-Ubuntu kubelet[27165]: E1204 13:12:57.753600 27165 kubelet.go:2167] Container runtime network not ready: NetworkReady=false reason:NetworkPl Dec 04 13:12:57 Test-LPAR-Ubuntu kubelet[27165]: W1204 13:12:57.753306 27165 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d Dec 04 13:12:52 Test-LPAR-Ubuntu kubelet[27165]: E1204 13:12:52.751990 27165 kubelet.go:2167] Container runtime network not ready: NetworkReady=false reason:NetworkPl Dec 04 13:12:52 Test-LPAR-Ubuntu kubelet[27165]: W1204 13:12:52.751707 27165 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d Dec 04 13:12:49 Test-LPAR-Ubuntu kubelet[27165]: E1204 13:12:49.135996 27165 pod_workers.go:186] Error syncing pod 9913e1c265cb1c42c725dd08db6af81d ("kube-controller- Dec 04 13:12:47 Test-LPAR-Ubuntu kubelet[27165]: E1204 13:12:47.750447 27165 kubelet.go:2167] Container runtime network not ready: NetworkReady=false reason:NetworkPl Dec 04 13:12:47 Test-LPAR-Ubuntu kubelet[27165]: W1204 13:12:47.750180 27165 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d Dec 04 13:12:42 Test-LPAR-Ubuntu kubelet[27165]: E1204 13:12:42.748933 27165 kubelet.go:2167] Container runtime network not ready: NetworkReady=false reason:NetworkPl Dec 04 13:12:42 Test-LPAR-Ubuntu kubelet[27165]: W1204 13:12:42.748618 27165 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d Dec 04 13:12:37 Test-LPAR-Ubuntu kubelet[27165]: E1204 13:12:37.747341 27165 kubelet.go:2167] Container runtime network not ready: NetworkReady=false reason:NetworkPl Dec 04 13:12:37 Test-LPAR-Ubuntu kubelet[27165]: W1204 13:12:37.747037 27165 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d Dec 04 13:12:35 Test-LPAR-Ubuntu kubelet[27165]: E1204 13:12:35.136057 27165 pod_workers.go:186] Error syncing pod 9913e1c265cb1c42c725dd08db6af81d ("kube-controller- Dec 04 13:12:32 Test-LPAR-Ubuntu kubelet[27165]: E1204 13:12:32.745632 27165 kubelet.go:2167] Container runtime network not ready: NetworkReady=false reason:NetworkPl Dec 04 13:12:32 Test-LPAR-Ubuntu kubelet[27165]: W1204 13:12:32.745344 27165 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d Dec 04 13:12:27 Test-LPAR-Ubuntu kubelet[27165]: E1204 13:12:27.744124 27165 kubelet.go:2167] Container runtime network not ready: NetworkReady=false reason:NetworkPl Dec 04 13:12:27 Test-LPAR-Ubuntu kubelet[27165]: W1204 13:12:27.743828 27165 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d

In /etc/systemd/system/kubelet.service.d/10-kubeadm.conf there is no environment variable present for NETWORK Environment:

Kubernetes version (use kubectl version): kubectl version Client Version: version.Info{Major:“1”, Minor:“12”, GitVersion:“v1.12.3”, GitCommit:“435f92c719f279a3a67808c80521ea17d5715c66”, GitTreeState:“clean”, BuildDate:“2018-11-26T12:57:14Z”, GoVersion:“go1.10.4”, Compiler:“gc”, Platform:“linux/ppc64le”} Server Version: version.Info{Major:“1”, Minor:“12”, GitVersion:“v1.12.3”, GitCommit:“435f92c719f279a3a67808c80521ea17d5715c66”, GitTreeState:“clean”, BuildDate:“2018-11-26T12:46:57Z”, GoVersion:“go1.10.4”, Compiler:“gc”, Platform:“linux/ppc64le”} dockuser@Test-LPAR-Ubuntu:/etc/selinux$ and kubeadm version is : kubeadm version kubeadm version: &version.Info{Major:“1”, Minor:“12”, GitVersion:“v1.12.3”, GitCommit:“435f92c719f279a3a67808c80521ea17d5715c66”, GitTreeState:“clean”, BuildDate:“2018-11-26T12:54:02Z”, GoVersion:“go1.10.4”, Compiler:“gc”, Platform:“linux/ppc64le”} dockuser@Test-LPAR-Ubuntu:/etc/selinux$
Cloud provider or hardware configuration: IBM POWER8 Server
OS (e.g. from /etc/os-release): Ubuntu 16.04.5 LTS (xenial) NAME=“Ubuntu” VERSION=“16.04.5 LTS (Xenial Xerus)” ID=ubuntu ID_LIKE=debian PRETTY_NAME=“Ubuntu 16.04.5 LTS” VERSION_ID=“16.04” HOME_URL=“http://www.ubuntu.com/” SUPPORT_URL=“http://help.ubuntu.com/” BUG_REPORT_URL=“http://bugs.launchpad.net/ubuntu/” VERSION_CODENAME=xenial UBUNTU_CODENAME=xenial
Kernel (e.g. uname -a): Linux Test-LPAR-Ubuntu 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:09:19 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux
Install tools:
Others: Docker version dockuser@Test-LPAR-Ubuntu:~$ sudo docker version Client: Version: 17.03.2-ce API version: 1.27 Go version: go1.6.2 Git commit: f5ec1e2 Built: Thu Jul 5 23:06:06 2018 OS/Arch: linux/ppc64le

Server: Version: 17.03.2-ce API version: 1.27 (minimum version 1.12) Go version: go1.6.2 Git commit: f5ec1e2 Built: Thu Jul 5 23:06:06 2018 OS/Arch: linux/ppc64le Experimental: false dockuser@Test-LPAR-Ubuntu:~$

/kind bug

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 18 (7 by maintainers)

Most upvoted comments

Just in case someone comes across this who is using a g4dn family instance on AWS. I was stuck on this for a while because the version of the CNI plugin I was using didn’t support that family. After upgrading the CNI plugin it worked. https://docs.aws.amazon.com/eks/latest/userguide/cni-upgrades.html

eightlimbed on Dec 5, 2019

@langerkunal This is probably due to the fact that the dockershim socket is not available for the kubelet to use when you do kubeadm init or kubeadm join.

You can verify that with:

# ss -pl -x -a | grep -i dockershim

Which should output something like

u_str  LISTEN     0      128    /var/run/dockershim.sock 19862                 * 0                     users:(("kubelet",pid=876,fd=7))
u_str  ESTAB      0      0      /var/run/dockershim.sock 18903                 * 17946                 users:(("kubelet",pid=876,fd=13))
u_str  ESTAB      0      0      /var/run/dockershim.sock 18904                 * 25471                 users:(("kubelet",pid=876,fd=14))

When that socket is not available kubeadm can’t contact the kubelet to understand what is the config it should download from the apiserver to configure CNI, and then it fails with that error message.

fntlnz on Dec 27, 2018