kubernetes: GracefulNodeShutdown not work
What happened?
For 1.21, by checking https://kubernetes.io/blog/2021/04/21/graceful-node-shutdown-beta/,
I added below args to kubelet-config.json:
"shutdownGracePeriod": "1m30s",
"shutdownGracePeriodCriticalPods": "30s"
I can see 99-kubelet.conf:
ubuntu@ip-10-120-80-6:~$ uname -a
Linux ip-10-120-80-6 5.11.0-1022-aws #23~20.04.1-Ubuntu SMP Mon Nov 15 14:03:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@ip-10-120-80-6:~$ cat /etc/systemd/logind.conf.d/99-kubelet.conf
# Kubelet logind override
[Login]
InhibitDelayMaxSec=90
But I can’t see inhibitor from kubelet:
ubuntu@ip-10-120-80-6:~$ sudo systemd-inhibit --list
WHO UID USER PID COMM WHAT WHY MODE
Unattended Upgrades Shutdown 0 root 588 unattended-upgr shutdown Stop ongoing upgrades or perform upgrades before shutdown delay
1 inhibitors listed.
What did you expect to happen?
I should be able to see inhibitors from kubelet like this?
kubelet-node ~ # systemd-inhibit --list
Who: kubelet (UID 0/root, PID 1515/kubelet)
What: shutdown
Why: Kubelet needs time to handle node shutdown
Mode: delay
1 inhibitors listed.
How can we reproduce it (as minimally and precisely as possible)?
start a new k8s node running 1.21 and run sudo systemd-inhibit --list
Anything else we need to know?
No response
Kubernetes version
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.5-eks-bc4871b", GitCommit:"5236faf39f1b7a7dabea8df12726f25608131aa9", GitTreeState:"clean", BuildDate:"2021-10-29T23:35:14Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.2-eks-06eac09", GitCommit:"5f6d83fe4cb7febb5f4f4e39b3b2b64ebbbe3e97", GitTreeState:"clean", BuildDate:"2021-09-13T14:20:15Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
Cloud provider
AWS EKS with custom AMI
base os: Ubuntu 20.04
OS version
# On Linux:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
$ uname -a
Linux ip-10-120-52-174 5.11.0-1022-aws #23~20.04.1-Ubuntu SMP Mon Nov 15 14:03:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
Install tools
Container runtime (CRI) and and version (if applicable)
Related plugins (CNI, CSI, …) and versions (if applicable)
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 28 (13 by maintainers)
Commits related to this issue
- Configure Graceful Node Shutdown and lengthen max inhibitor delay * Configure Kubelet Graceful Node Shutdown to detect system shutdown events and stop running containers gracefully when possible * Al... — committed to poseidon/typhoon by dghubble 2 years ago
- Configure Graceful Node Shutdown and lengthen max inhibitor delay * Configure Kubelet Graceful Node Shutdown to detect system shutdown events and stop running containers gracefully when possible * Al... — committed to poseidon/typhoon by dghubble 2 years ago
- Configure Graceful Node Shutdown and lengthen max inhibitor delay * Configure Kubelet Graceful Node Shutdown to detect system shutdown events and stop running containers gracefully when possible * Al... — committed to poseidon/terraform-onprem-kubernetes by dghubble 2 years ago
- Configure Graceful Node Shutdown and lengthen max inhibitor delay * Configure Kubelet Graceful Node Shutdown to detect system shutdown events and stop running containers gracefully when possible * Al... — committed to poseidon/terraform-google-kubernetes by dghubble 2 years ago
- Configure Graceful Node Shutdown and lengthen max inhibitor delay * Configure Kubelet Graceful Node Shutdown to detect system shutdown events and stop running containers gracefully when possible * Al... — committed to poseidon/terraform-azure-kubernetes by dghubble 2 years ago
- Configure Graceful Node Shutdown and lengthen max inhibitor delay * Configure Kubelet Graceful Node Shutdown to detect system shutdown events and stop running containers gracefully when possible * Al... — committed to poseidon/terraform-aws-kubernetes by dghubble 2 years ago
- Configure Graceful Node Shutdown and lengthen max inhibitor delay * Configure Kubelet Graceful Node Shutdown to detect system shutdown events and stop running containers gracefully when possible * Al... — committed to poseidon/terraform-digitalocean-kubernetes by dghubble 2 years ago
- Configure Graceful Node Shutdown and lengthen max inhibitor delay * Configure Kubelet Graceful Node Shutdown to detect system shutdown events and stop running containers gracefully when possible * Al... — committed to aristanetworks/monsoon by dghubble 2 years ago
It’s set by
unattended-upgrades, let me remove this and try.I got around this by naming my logind conf file
/etc/systemd/logind.conf.d/zz-max-delay.conf, which gives it higher precedence than the config file for unattended-upgrades, which is namedunattended-upgrades-logind-maxdelay.conf. It’s a little unfortunate they didn’t follow the numbering pattern here, but thezz-prefix is a decent workaround for getting the Kube config to load lexicographically after the unattended-upgrades config for this value.