kubernetes: Some static pods fail to start on K8S 1.22 and 1.23

What happened?

In Kubernetes 1.23 we are seeing some static pods fail to start. These static pods failures are on quick remove and adds leaving the static pod not started.

We need to revert kubernetes/kubernetes#104743.

cc @gjkim42

What did you expect to happen?

All static pods should restart.

How can we reproduce it (as minimally and precisely as possible)?

diff --git a/pkg/kubelet/pod_workers_test.go b/pkg/kubelet/pod_workers_test.go
index 4028c06c292..0e852594a40 100644
--- a/pkg/kubelet/pod_workers_test.go
+++ b/pkg/kubelet/pod_workers_test.go
@@ -880,3 +880,34 @@ func Test_allowPodStart(t *testing.T) {
 		})
 	}
 }
+
+func TestUpdatePodWithQuickAddRemoveStaticPod(t *testing.T) {
+	podWorkers, _ := createPodWorkers()
+	staticPodA := newStaticPod("0000-0000-0000", "running-static-pod")
+	staticPodB := newStaticPod("0000-0000-0000", "running-static-pod")
+
+	podWorkers.UpdatePod(UpdatePodOptions{
+		Pod:        staticPodA,
+		UpdateType: kubetypes.SyncPodCreate,
+	})
+
+	podWorkers.UpdatePod(UpdatePodOptions{
+		Pod:        staticPodA,
+		UpdateType: kubetypes.SyncPodKill,
+	})
+
+	drainAllWorkers(podWorkers)
+
+	podWorkers.UpdatePod(UpdatePodOptions{
+		Pod:        staticPodB,
+		UpdateType: kubetypes.SyncPodCreate,
+	})
+
+	drainAllWorkers(podWorkers)
+
+	t.Logf("rphillips waitingToStartStaticPodsByFullName=%v", podWorkers.waitingToStartStaticPodsByFullname)
+	if status, ok := podWorkers.podSyncStatuses["0000-0000-0000"]; ok {
+		t.Logf("rphillips podSyncStatuses=%+v", status)
+	}
+	t.Logf("rphillips startedStaticPodsByFullname=%v", podWorkers.startedStaticPodsByFullname)
+}

Anything else we need to know?

No response

Kubernetes version

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and and version (if applicable)

Related plugins (CNI, CSI, …) and versions (if applicable)