kops: Kernel throttling bug patch in kops node images

What?

Linux Kernel has a bug in which un-necessary CPU throttling happens due to the bug in Kernel. KernelBug: https://bugzilla.kernel.org/show_bug.cgi?id=198197 KubernetesIssue: https://github.com/kubernetes/kubernetes/issues/67577

Many of us use kops default node images and have been experiencing problems due to un-necessary CPU throttling in our pods.

Please suggest if the latest kops node images has the kernel patch which has the fix for CPUThrottling. If not then can we bring this patch to these node images please.

Node Images

kope.io/k8s-1.17-debian-stretch-amd64-hvm-ebs-2020-01-17
kope.io/k8s-1.16-debian-stretch-amd64-hvm-ebs-2020-01-17
kope.io/k8s-1.15-debian-stretch-amd64-hvm-ebs-2020-01-17

cc @nuru @justinsb

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 39 (32 by maintainers)

Most upvoted comments

if you are fine with some minimal nftable mixed with legacy iptables and all your pods supports nftables, Buster will work. Kops may look into defaulting to Ubuntu 20.04 over Buster because of this though.

olemarkus on Apr 30, 2020

Ubuntu 20.04 should have these all fixed. I am not sure how many made it to Debian Buster. Personally I use Ubuntu and Debian official images. Kops one are just Debian Stretch images with some packages pre-installed to have faster startup.

You can get an idea about how the default images are created and what they contain here: https://github.com/kubernetes-sigs/image-builder/blob/master/images/kube-deploy/imagebuilder/templates/1.15-stretch.yml

hakman on Apr 22, 2020

I have been trying to track the distribution of this patch as best I can, but have not been completely successful. So far, my best information is that

I have reproduced the issue in the current kops stable 1.15 AMI: kope.io/k8s-1.15-debian-stretch-amd64-hvm-ebs-2020-01-17
The patch probably made into Debian buster, since that is based on Linux kernel 4.19 and the fix landed in 4.19.84
The patch never made it into Debian stretch, but probably has been backported to stretch-backports since it appears now to be based on 4.19.98.

Since buster is the current stable release, I would recommend updating the AMI to the current Debian buster, which is 10.3.

Nuru on Apr 22, 2020

@prashantkalkar you can use Ubuntu 20.04 by manually setting the image for each instance group: https://github.com/kubernetes/kops/blob/e323f188b3e7e2407184228dbcc6757b9cc19cac/channels/stable#L47

hakman on Jul 14, 2020

@alok87 there is no check for minimum k8s version for Ubuntu 20.04.

hakman on May 1, 2020

One good news is that Ubuntu 20.04 support will be available in Kops 1.16.2: https://github.com/kubernetes/kops/commit/19386520fcec0f6ea775d7570c47062b6823cb0a.

hakman on Apr 30, 2020

@alok87 I believe there are blockers for using Debian Buster with k8s < 1.17, see https://github.com/kubernetes/kops/issues/8224

kzap on Apr 30, 2020

no, but should be very similar to the stretch one or buster one from here https://github.com/kubernetes-sigs/image-builder/pull/205.

hakman on Apr 22, 2020

Does the Debian buster or Ubuntu 20.04 has the bug fixed? Should we move to these images? Does the node images needs to be custom build for Kubernetes?

alok87 on Apr 22, 2020