amazon-vpc-cni-k8s: Coredns stuck on ContainerCreating with `FailedCreatePodSandBox` warning for CNI versions 1.7.6 with Cilium 1.9.1

What happened: New cluster with nodes restarted. coredns stuck on ContainerCreating when using CNI v1.7.6 and Cilium 1.9.1. Other pods are also experiencing the same behavior ( ContainerCreating )

coredns:v1.6.6-eksbuild.1

Attach logs

Warning  FailedCreatePodSandBox  29s  kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "112861d5995ca8f44c1dc17f00c947d72a44cf69c9deda34fbaf56b204742874" network for pod "coredns-6d857998c6-gxsd7": networkPlugin cni failed to set up pod "coredns-6d857998c6-gxsd7_kube-system" network: invalid character '{' after top-level value

What you expected to happen: I expected coredns and other pods to be in running state

How to reproduce it (as minimally and precisely as possible): Deploy cni version 1.7.6 and cilium 1.9.1 on EKS 1.17

Anything else we need to know?: We have Cilium running in chaining mode (v1.9.1) [(https://docs.cilium.io/en/v1.9/gettingstarted/cni-chaining-aws-cni/)]

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.12-eks-7684af", GitCommit:"7684af4ac41370dd109ac13817023cb8063e3d45", GitTreeState:"clean", BuildDate:"2020-10-20T22:57:40Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
  • CNI Version
amazon-k8s-cni-init:v1.7.6
amazon-k8s-cni:v1.7.6
  • OS (e.g: cat /etc/os-release):
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
  • Kernel (e.g. uname -a): Linux REDACTED.compute.internal 4.14.203-156.332.amzn2.x86_64 #1 SMP Fri Oct 30 19:19:33 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 1
  • Comments: 28 (12 by maintainers)

Most upvoted comments

Hi @shaikatz

Sorry for the delay. Will take this up in rel 1.7.9 planned for January.

@jayanthvn what is the ETA to release that fix that doesn’t require any manual modification of the nodes? 2 versions already released after this PR was merged, but this fix was ignored in both of them.

Thanks again @jayanthvn