amazon-vpc-cni-k8s: AWS EKS - add cmd: failed to assign an IP address to container

What happened:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “efafab567a68ffb237e67050c4d70d2d1084bf1aca3631b74e5a0802146d150a” network for pod “xxx-9d8b7c98d-j8ldn”: networkPlugin cni failed to set up pod “xxx-9d8b7c98d-j8ldn_default” network: add cmd: failed to assign an IP address to container

eks_i-089d9482970086cc5_2021-12-11_1011-UTC_0.6.2.tar.gz

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:38:26Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.2-eks-06eac09", GitCommit:"5f6d83fe4cb7febb5f4f4e39b3b2b64ebbbe3e97", GitTreeState:"clean", BuildDate:"2021-09-13T14:20:15Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

CNI Version

amazon-k8s-cni-init:v1.10.1-eksbuild.1
amazon-k8s-cni:v1.10.1-eksbuild.1

OS (e.g: cat /etc/os-release):

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

Kernel (e.g. uname -a):

Linux ip-192-168-75-48.ap-northeast-2.compute.internal 5.4.156-83.273.amzn2.x86_64 #1 SMP Sat Oct 30 12:59:07 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 6
Comments: 24 (11 by maintainers)

Most upvoted comments

Hi @GoGoPenguin , I see the pod is assigned successfully later, after the failed to assign IP address events:

After the IP address assignment fails for container 56a700c514d4b769fa64f7af3daaxxxxx for pod cityfarm-development-7997fb8976-gtgxp , the same pod gets IP successfully with container ID 79d32e3aae71f0xxxxx as seen below:

{"level":"info","ts":"2021-12-13T02:48:48.916Z","caller":"routed-eni-cni-plugin/cni.go:111","msg":"Received CNI add request: ContainerID(79d32e3aae71f0xxxxx) Netns(/proc/6756/ns/net) IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=cityfarm-development-7997fb8976-gtgxp;K8S_POD_INFRA_CONTAINER_ID=79d32e3aae71f0xxxxx) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"debug","ts":"2021-12-13T02:48:48.916Z","caller":"routed-eni-cni-plugin/cni.go:111","msg":"MTU value set is 9001:"}
{"level":"info","ts":"2021-12-13T02:48:48.925Z","caller":"routed-eni-cni-plugin/cni.go:111","msg":"Received add network response for container 79d32e3aae71f0xxxxx interface eth0: Success:true IPv4Addr:\"192.168.74.87\" DeviceNumber:1 VPCv4CIDRs:\"192.168.0.0/16\""}

Were you able to see the pod deployment running successfully later?

If you were wondering why the initial errors were seen before the successful deployment, I have stated the reason below:

The previous failures were because of 13 of 15 total IPs being used and 2 IPs being present in cool down throughout the failed to assign IP address period. When one of the IP from cool down is return to warm pool, it gets assigned to the pod mentioned above as seen below in ipamd.json logs. As instance type is t3.medium as @jayanthvn mentioned, it supports 3 ENIs and 5 secondary IPs so only 15 IPs will be available at any time.

{"level":"debug","ts":"2021-12-13T02:48:48.920Z","caller":"datastore/data_store.go:757","msg":"Returning Free IP 192.168.74.87"}
{"level":"debug","ts":"2021-12-13T02:48:48.920Z","caller":"datastore/data_store.go:680","msg":"New IP from CIDR pool- 192.168.74.87"}
{"level":"info","ts":"2021-12-13T02:48:48.920Z","caller":"datastore/data_store.go:784","msg":"AssignPodIPv4Address: Assign IP 192.168.74.87 to sandbox aws-cni/79d32e3aae71f0xxxxx"}

Note: I have replaced contained IDs with xxxxx at end. You could use the timestamps to find mappings in your log.

Shreya027 on Dec 17, 2021

Thanks, Looking into it, will get back soon.

Shreya027 on Dec 16, 2021