amazon-vpc-cni-k8s: AWK EKS network: add cmd: failed to assign an IP address to container

i have issue when redeploying pods

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e9f85302d108de589c61061d5aee17b38c1ec1c44031a5c44d61fe9e4208bad2" network for pod "templater-api-7589dd5d85-xddrd": networkPlugin cni failed to set up pod "templater-api-7589dd5d85-xddrd_a-plus" network: add cmd: failed to assign an IP address to container

kubectl describe daemonset aws-node -n kube-system | grep Image | cut -d "/" -f 2
amazon-k8s-cni-init:v1.7.10
amazon-k8s-cni:v1.7.5-eksbuild.1

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

Container runtime version
docker://19.3.13

kubelet version
v1.19.6-eks-49a6c0

Kernel version: Linux ip-192-168-46-20.us-west-2.compute.internal 5.4.110-54.189.amzn2.x86_64 #1 SMP Mon Apr 26 21:25:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

gathered info here eks.tar.gz

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 3
Comments: 17 (9 by maintainers)

Commits related to this issue

Increasing instance type - ENI Refer: https://github.com/aws/amazon-vpc-cni-k8s/issues/1480 — committed to vigneshragupathy/terraform-eks-prometheus-thanos by vigneshragupathy 2 years ago

Most upvoted comments

@dgoscn happy to help.

Since the instances are failing to join we would need to check the instance logs. Since previously you mentioned it has sensitive information. So is it possible to open a support ticket? we can help check the internal error.

jayanthvn on Jun 29, 2021

@dgoscn, Thanks for confirming. The IP scale you are getting per node is expected. Since with custom networking we won’t be allocating secondary IPs for primary ENI so the scale reduces by 1 ENI. So with t3.medium you will have just 2 ENIs and 10 IPs. Have you also updated the max pods?

New maxPods with custom networking enabled -

maxPods = (number of interfaces - 1) * (max IPv4 addresses per interface - 1) + 2

T3.medium -

maxPods = (3 - 1) * ( 6 - 1 ) + 2 = 12

Thanks!

jayanthvn on Jun 29, 2021

@dgoscn, t3 medium supports 3 ENIs with 5 IP each. So are you using custom networking or have configured MAX_ENI?

jayanthvn on Jun 28, 2021

Hi @dgoscn

Will you be able to email (varavaj@amazon.com) me logs from sudo bash /opt/cni/bin/aws-cni-support.sh?

jayanthvn on Jun 24, 2021

Hi guys,

I’m having the same issue:

[Normal][default/jenkins-backend-agent-91xqs-xlff7][SandboxChanged] Pod sandbox changed, it will be killed and re-created.
[Warning][default/jenkins-backend-agent-91xqs-xlff7][FailedCreatePodSandBox] Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ffb6870508775581644da56ec0462a4959bcf055e4548196d8ace8147aab0859" network for pod "jenkins-backend-agent-91xqs-xlff7": networkPlugin cni failed to set up pod "jenkins-backend-agent-91xqs-xlff7_default" network: add cmd: failed to assign an IP address to container

Anyone knows how to solve it?

Thanks!

RogerRey14 on Jun 14, 2021