kubernetes: 1:20: Pods stuck as failed with "Pod Predicate NodeAffinity failed"

What happened?

pod status is NodeAffinity,

ceph-rgw-548fdf5fc5-2whgg          1/1     Running        0          4h25m   10.232.1.28   node-5   <none>           <none>
ceph-rgw-548fdf5fc5-86gkn          1/1     Running        0          4h25m   10.232.2.26   node-6   <none>           <none>
ceph-rgw-548fdf5fc5-rc55v          1/1     Running        1          4h25m   10.232.0.29   node-4   <none>           <none>
ceph-rgw-548fdf5fc5-t5k8d          0/1     NodeAffinity   0          4h25m   <none>        node-5   <none>           <none>

pod replicas=3 but i cant make it happend again

What did you expect to happen?

pod which has NodeAffinity status shouldn’t have occured

How can we reproduce it (as minimally and precisely as possible)?

can’t

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.14-es", GitCommit:"f8d6e5ec17034b4c5acacf72de8653b15920bea5", GitTreeState:"clean", BuildDate:"2022-04-22T06:19:09Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/arm64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.14-es", GitCommit:"f8d6e5ec17034b4c5acacf72de8653b15920bea5", GitTreeState:"clean", BuildDate:"2022-04-22T06:18:52Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/arm64"}

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, …) and versions (if applicable)