kubernetes: Missing taint: node.kubernetes.io/unreachable:NoExecute when nodes enter NotReady state on new installs of 1.19+

What happened: I have been testing some ansible roles to deploy Kubernetes using kubeadm on Ubuntu virtual machines, when testing I noticed that pods were not being terminated + re-created on additional nodes when the existing node that the pod lived on went into a NotReady state. The node that goes into a NotReady state does not have the node.kubernetes.io/unreachable:NoExecute taint applied. Thus preventing the pods to be evicted(terminated) and re-created. (I confirmed this when installing manually(non ansible) too, also I used dockershim/containerd as the cri, tested using flannel/calico as the cni)

This occurs in new installations of kubernetes when using any version greater than 1.18.12 i.e. 1.19.x. This has been tested & confirmed with kubeadm running on Ubuntu and kind running on Mac

Strangely I can confirm though, that if kubeadm/kubelet=1.8.12 is installed first, then upgraded to 1.19.x on Ubuntu using APT that the taint is added to the NotReady node and the pods are evicted(terminated) and re-created on additional nodes. (I haven’t confirmed this using kind)

What you expected to happen: I expected that any pods running on a node which has entered the NotReady state to be terminated and re-created on any other available node because the node.kubernetes.io/unreachable:NoExecute taint is applied.

How to reproduce it (as minimally and precisely as possible):

Using kind

kind create cluster --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
EOF

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
spec:
  replicas: 1
  selector:
    matchLabels:
      srv: test
  template:
    metadata:
      labels:
        srv: test
    spec:
      containers:
      - image: nginx
        name: test
        ports:
        - containerPort: 80
        resources: {}
      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 10
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 10
EOF

Working/Tested on k8s 1.18.2 (kind v0.8.1):

kubectl get nodes
NAME                 STATUS   ROLES    AGE   VERSION
kind-control-plane   Ready    master   13m   v1.18.2
kind-worker          Ready    <none>   13m   v1.18.2
kind-worker2         Ready    <none>   13m   v1.18.2

kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP           NODE          NOMINATED NODE   READINESS GATES
test-67bff98d4b-ch2ng   1/1     Running   0          25s   10.244.2.3   kind-worker   <none>           <none>

docker stop kind-worker
kind-worker

kubectl get nodes
NAME                 STATUS     ROLES    AGE   VERSION
kind-control-plane   Ready      master   23m   v1.18.2
kind-worker          NotReady   <none>   22m   v1.18.2
kind-worker2         Ready      <none>   23m   v1.18.2

kubectl get pods -o wide
NAME                    READY   STATUS        RESTARTS   AGE     IP           NODE           NOMINATED NODE   READINESS GATES
test-67bff98d4b-ch2ng   1/1     Terminating   0          2m38s   10.244.2.3   kind-worker    <none>           <none>
test-67bff98d4b-kqkcv   1/1     Running       0          37s     10.244.3.3   kind-worker2   <none>           <none>

kubectl describe node kind-worker | grep node.kubernetes.io
Taints:             node.kubernetes.io/unreachable:NoExecute
                    node.kubernetes.io/unreachable:NoSchedule

Not Working/Tested on k8s 1.19.1 (kind v0.9.0):

kubectl get nodes
NAME                 STATUS   ROLES    AGE     VERSION
kind-control-plane   Ready    master   4m37s   v1.19.1
kind-worker          Ready    <none>   4m1s    v1.19.1
kind-worker2         Ready    <none>   4m1s    v1.19.1

kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
test-5f8dd86d7c-hrlwg   1/1     Running   0          48s   10.244.1.2   kind-worker2   <none>           <none>

docker stop kind-worker2
kind-worker2

kubectl get nodes
NAME                 STATUS     ROLES    AGE     VERSION
kind-control-plane   Ready      master   5m46s   v1.19.1
kind-worker          Ready      <none>   5m10s   v1.19.1
kind-worker2         NotReady   <none>   5m10s   v1.19.1

kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE     IP           NODE           NOMINATED NODE   READINESS GATES
test-5f8dd86d7c-hrlwg   1/1     Running   0          3m36s   10.244.1.2   kind-worker2   <none>           <none>

kubectl describe node kind-worker2 | grep node.kubernetes.io
Taints:             node.kubernetes.io/unreachable:NoSchedule

Using kubeadm

Please see my comment in another issue

Anything else we need to know?:

Please see the comment from @adampl confirming the bug here

Environment:

• Kubernetes version (use kubectl version): 1.19.x
• Cloud provider or hardware configuration: Bare-metal/Mac
• OS (e.g: cat /etc/os-release): Ubuntu 20.04/MacOS Catalina
• Kernel (e.g. uname -a): Linux captain1.k8s.someadmin.com 5.4.0-56-generic #62-Ubuntu SMP Mon Nov 23 19:20:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: kubeadm/kind
• Network plugin and version (if this is a network-related bug): calico
• Others: