amazon-vpc-cni-k8s: network is not ready: [runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady...

I fixed this issue by upgrading Kubernetes components. I had the same problem in my AWS EKS cluster. So ran below commands using eksctl CLI tool.

eksctl utils update-kube-proxy --name Your_Cluster_Name --approve eksctl utils update-aws-node --name Your_Cluster_Name --approve eksctl utils update-coredns --name Your_Cluster_Name --approve

Leaving this here as this issue was the first result on Google.

The problem for me was that my kube-proxy daemonset was using the --resource-container flag, which was removed on Kubernetes 1.16, resulting in this “cni config uninitialized” error and nodes getting stuck in the NotReady state.

I had to manually edit this daemonset and remove the flag ($ kubectl edit ds kube-proxy -n kube-system).

For reference, this is the daemonset command I’m using now, with kube-proxy 1.16.8:

      - command:
        - /bin/sh
        - -c
        - kube-proxy --oom-score-adj=-998 --master=https://MYCLUSTER.eks.amazonaws.com
          --kubeconfig=/var/lib/kube-proxy/kubeconfig --proxy-mode=iptables --v=2
          1>>/var/log/kube-proxy.log 2>&1

Just in case someone comes across this who is using a g4dn family instance on AWS. I was stuck on this for a while because the version of the CNI plugin I was using didn’t support that family. After upgrading the CNI plugin it worked. https://docs.aws.amazon.com/eks/latest/userguide/cni-upgrades.html

Recording this here in case it helps others.

I had a MutatingWebhookConfiguration hanging around that was no longer relevant and there were no pods available to service it. This was stopping nodes from becoming Ready. The kubelet logs and describe node messages had the exact same error as recorded here.

In my case, running kubectl delete MutatingWebhookConfiguration <name> and then restarting one of the kubelets caused all nodes to become healthy/ready.

Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

These logs still occurs on some occasions.

Coming from AWS support: It is possible that issue is caused by changes to security group requirements for worker nodes [1] introduced in EKS platform v3 [2]

Another possible cause is my old AWS provider. I use 1.60.0.

Hope this helps

[1] https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html [2] https://docs.aws.amazon.com/eks/latest/userguide/platform-versions.html

Okay for anyone that might still face this issue. In my case, we had a cluster that was initially set up a while ago at version 1.14 and was later upgraded to 1.16. I was able to deploy x86 nodes without any issue. However when I tried to deploy arm64 node groups they would fail with the error described in this issue

After running the steps that @SarasaGunawardhana mentioned

I fixed this issue by upgrading Kubernetes components. I had the same problem in my AWS EKS cluster. So ran below commands using eksctl CLI tool.

eksctl utils update-kube-proxy --name Your_Cluster_Name --approve eksctl utils update-aws-node --name Your_Cluster_Name --approve eksctl utils update-coredns --name Your_Cluster_Name --approve

I noticed that the aws-node pod was still using the 1.6.xx CNI version, when another cluster I had that was also running kubernetes v.1.16 was running CNI v1.7.xx

So after further troubleshooting I had to manually upgrade the CNI addon on my cluster. I also had to modify the daemon set configuration for kube-proxy as instructed here

Leaving this here as this issue was the first result on Google.

The problem for me was that my kube-proxy daemonset was using the --resource-container flag, which was removed on Kubernetes 1.16, resulting in this “cni config uninitialized” error and nodes getting stuck in the NotReady state.

I had to manually edit this daemonset and remove the flag ($ kubectl edit ds kube-proxy -n kube-system).

For reference, this is the daemonset command I’m using now, with kube-proxy 1.16.8:

      - command:
        - /bin/sh
        - -c
        - kube-proxy --oom-score-adj=-998 --master=https://MYCLUSTER.eks.amazonaws.com
          --kubeconfig=/var/lib/kube-proxy/kubeconfig --proxy-mode=iptables --v=2
          1>>/var/log/kube-proxy.log 2>&1

I tried to use this method, kube-proxy still cannot be started properly, then I refer to this tutorial https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html

The pod security policy admission controller is enabled on Amazon EKS clusters running Kubernetes version 1.13 or later. If you’re upgrading your cluster to Kubernetes version 1.13 or later, ensure that the proper pod security policies are in place before you update to avoid any issues. You can check for the default policy with the following command:

then I install default pod security policy install psp and followed What you need to do before upgrading to 1.16 in tutorial updated my kube-proxy and everything is ok!

We are trying to create a new cluster using eksctl and face the same error . The cluster is created successfully but the nodes are not become ready . Detail is below.

• EKS: version 1.14

• CNI: amazon-k8s-cni:v1.5.3

• AMI: amazon-eks-node-1.14-v20190927 (ami-02e124a380df41614)

• create cluster in the existing vpc and subnets which have sufficient ips.

• Kubelet error on the nodes ↓:

W1101 03:31:48.212631    3705 cni.go:213] Unable to update cni config: No networks found in /etc/cni/net.d
E1101 03:31:48.430668    3705 kubelet.go:2172] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

• ↓ exec sudo bash eks-log-collector.sh on the node. but all log files in the ipamd directory are empty.

I experienced this issue after updating EKS to version 1.16 and @SarasaGunawardhana commands did the trick for me.

This issue contains a mix of CNI versions and EKS cluster versions. I think @ppaepam and @SarasaGunawardhana are both right, and if anyone has similar issues please open a new issue to track that specific case.

I tried to run /opt/cni/bin/aws-cni-support.sh on the node with pod aws-node-hhtrt but I get this error:

The second one is same with https://github.com/aws/amazon-vpc-cni-k8s/issues/285

The line in the script should be updated as command -v kubectl > /dev/null && kubectl get --kubeconfig=/var/lib/kubelet/kubeconfig --raw=/api/v1/pods or something.