amazon-eks-ami: Kernel 5.1+ in EKS Amazon Linux AMI to resolve conntrack race conditions

What is the state of this issue? Is there an ETA for the fix?

We are working with the Amazon Linux kernel team to backport the fixes into 4.14 kernel. I will keep this thread updated with the progress. Appreciate the patience!

The worker AMI release has the conntrack race condition fixes backported into 4.14 kernel. More specifically the following patches

• http://patchwork.ozlabs.org/patch/937963/
• http://patchwork.ozlabs.org/patch/1032812/

Please give it a try!

For customers looking to use Linux kernel 5.4 as a solution to this, there are now two options.

1 Upgrade the kernel use amazon-linux-extras

sudo amazon-linux-extras install kernel-5.4
sudo reboot

2 Upgrade to Kubernetes version 1.19

EKS optimized AMIs 1.19+ will now come with the 5.4 kernel by default, so you can upgrade your clusters to 1.19 to use the latest AMI with 5.4.

This is badly needed. Any ETA for the patch to be GA?

Thanks @isaacegglestone! Any ETA for upgrading the AMI to use the new kernel? We need other features that would be enabled by kernel 4.19+ as well.

@rchernobelskiy @owlwalks @daniel-ciaglia You probably know this already but for others who wander here wondering how to resolve this still. I just downloaded the latest amazon linux source code ( linux-4.19.84) onto a clean instance after updating the kernel on it. I do see the two patches are there in the code now.

Method used to check: sudo amazon-linux-extras install kernel-ng && sudo reboot After reboot* yumdownloader --source kernel mkdir tmp-kernel-dir cd tmp-kernel-dir rpm2cpio …kernel-4.19.84-33.70.amzn2.src.rpm | cpio -idmv

cd tmp-kernel-dir/linux-4.19.84/net/netfilter

less the nf_conntrack_core.c

Validate against above two patches.

We are going to rollout new amis with this patch and I’ll update here if we see the problem disappear or whatever our result was.

I’ve managed to fix my DNS related problem. Looks like it was not Conntrack issue related. I am not sure exactly what the problem itself was, but it was definitely related to the fact I have created my EKS cluster in an existing VPC. That VPC already contained a Kops K8s cluster plus some other workloads/instances, outside the cluster. Apparently they shouldn’t cause a problem each other. Once I’ve created my EKS cluster in a dedicated VPC it was all good, using managed node groups and the same AMI (1.15.10-20200228) which initially failed for me.

To optimize your cluster you may also consider using FQDN addresses, at least the internal ones. It also helps setting ndots DNS param to 1. These two optimizations alone helped me a lot in the old cluster which had DNS problem - they managed to lower the fail rate considerably.

Hopefully these tips will help somebody out there.

We’ve had a report from a customer that they are still seeing issues with conntrack race conditions on the 5.4 kernel. It may be related to customers using the CNI feature per pod security group. We’ve been working with the AmazonLinux team over the last couple of days and have identified patches that seem to fix the issue. Work is being done to merge these patches and get them out in the next AL2 release. Once a new kernel is released, customers can run yum update kernel and reboot their nodes to get the latest patch. Alternatively, EKS will release a new EKS optimized AMI once the patch is available, and you can replace your nodes with the latest AMIs.

We’ll update here when the patch and the new AMIs are available.

It looks like you can get the 4.19 kernel on the EKS AMI via: sudo amazon-linux-extras install kernel-ng && sudo reboot More info: https://aws.amazon.com/about-aws/whats-new/2019/07/amazon-linux-2-extras-provides-aws-optimized-versions-of-new-linux-kernels/

We were planning to release 5.4.x kernel with EKS 1.19 release followed by updating AMIs of all supported Kubernetes versions. However, an Amazon engineer found a kernel bug in versions 4.19.86 and higher. We have submitted a patch upstream and are waiting for that to be merged.

We are sorry you continue to face this issue. We have been working on a reproduction and have been unsuccessful so far in the latest EKS worker AMIs.

@palmbardier @betinro : Can you provide more detailed reproduction steps on your environment where this is happening?

Today I built a custom Debian based image which is supposed to solve this issue. I got inspired from this post and I have managed to build a Debian based amazon-eks-node-1.15-v20200331 AMI with Packer. This is what kubectl get nodes reports: Debian GNU/Linux 9 (stretch) / 4.19.0-0.bpo.6-amd64 / docker://18.6.3

Unfortunately I see no improvement for DNS queries. They still fail at about the same rate. What it’s strange though is that it behaves quite different for different pods. All nslookups are fast and no timeout from some pods, while from some other pods I get more than 50% failures. And there is no considerable difference between these pods, just different micro-services but all based on the same Alpine image. I started to suppose this is not actually a conntrack race issue, or at least not only this one. And what’s more strange is that on the kops cluster there is no single issue and the nodes run Ubuntu 18.04.3 with kernel 4.15.0-1054-aws which is actually older than the Debian one. Both EKS and kops clusters run the very same YAMLs. I’ve already spent three full days on this issue and found no solution for EKS to properly work.