kubernetes: GracefulNodeShutdown not work

What happened?

For 1.21, by checking https://kubernetes.io/blog/2021/04/21/graceful-node-shutdown-beta/, I added below args to kubelet-config.json:

  "shutdownGracePeriod": "1m30s",
  "shutdownGracePeriodCriticalPods": "30s"

I can see 99-kubelet.conf:

ubuntu@ip-10-120-80-6:~$ uname -a
Linux ip-10-120-80-6 5.11.0-1022-aws #23~20.04.1-Ubuntu SMP Mon Nov 15 14:03:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

ubuntu@ip-10-120-80-6:~$ cat /etc/systemd/logind.conf.d/99-kubelet.conf
# Kubelet logind override
[Login]
InhibitDelayMaxSec=90

But I can’t see inhibitor from kubelet:

ubuntu@ip-10-120-80-6:~$ sudo systemd-inhibit --list
WHO                          UID USER PID COMM            WHAT     WHY                                                       MODE
Unattended Upgrades Shutdown 0   root 588 unattended-upgr shutdown Stop ongoing upgrades or perform upgrades before shutdown delay

1 inhibitors listed.

What did you expect to happen?

I should be able to see inhibitors from kubelet like this?

kubelet-node ~ # systemd-inhibit --list
    Who: kubelet (UID 0/root, PID 1515/kubelet)
    What: shutdown
    Why: Kubelet needs time to handle node shutdown
    Mode: delay

1 inhibitors listed.

How can we reproduce it (as minimally and precisely as possible)?

start a new k8s node running 1.21 and run sudo systemd-inhibit --list

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.5-eks-bc4871b", GitCommit:"5236faf39f1b7a7dabea8df12726f25608131aa9", GitTreeState:"clean", BuildDate:"2021-10-29T23:35:14Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.2-eks-06eac09", GitCommit:"5f6d83fe4cb7febb5f4f4e39b3b2b64ebbbe3e97", GitTreeState:"clean", BuildDate:"2021-09-13T14:20:15Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

AWS EKS with custom AMI base os: Ubuntu 20.04

OS version

# On Linux:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
$ uname -a
Linux ip-10-120-52-174 5.11.0-1022-aws #23~20.04.1-Ubuntu SMP Mon Nov 15 14:03:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and and version (if applicable)

Related plugins (CNI, CSI, …) and versions (if applicable)

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 28 (13 by maintainers)

Commits related to this issue

Most upvoted comments

current value of InhibitDelayMaxSec (30s) is less than requested ShutdownGracePeriod

It’s set by unattended-upgrades, let me remove this and try.

ubuntu@ip-10-120-29-222:/usr/lib/systemd/logind.conf.d$ cat unattended-upgrades-logind-maxdelay.conf
[Login]
# delay
InhibitDelayMaxSec=30

@wzshiming since unattended-upgrades is a default enabled service in ubuntu, will this be a common issue in ubuntu box?

Yes, but if the total time does not exceed 30s, this configuration conflict should be avoided. There is currently no better way to avoid this

I got around this by naming my logind conf file /etc/systemd/logind.conf.d/zz-max-delay.conf, which gives it higher precedence than the config file for unattended-upgrades, which is named unattended-upgrades-logind-maxdelay.conf. It’s a little unfortunate they didn’t follow the numbering pattern here, but the zz- prefix is a decent workaround for getting the Kube config to load lexicographically after the unattended-upgrades config for this value.