autoscaler: predicate checking error: node(s) didn't match Pod's node affinity; predicateName=NodeAffinity; reasons: node(s) didn't match Pod's node affinity; debugInfo=

Which component are you using?:

cluster-autoscaler

What version of the component are you using?:

Component version: 9.9.2 (via Helm chart)

What k8s version are you using (kubectl version)?:

kubectl version

Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.6-eks-49a6c0", GitCommit:"49a6c0bf091506e7bafcdb1b142351b69363355a", GitTreeState:"clean", BuildDate:"2020-12-23T22:10:21Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"

What environment is this in?:

Cluster: EKS (k8s 1.19) original ASG’s node type: r5.2xlarge Other ASGs’ node types: m5.2xlarge and m5.xlarge

What did you expect to happen?:

After a node from an ASG was drained and terminated, the evicted pod indefinitely went in the Pending state with the following error:

predicate checking error: node(s) didn't match Pod's node affinity; predicateName=NodeAffinity; reasons: node(s) didn't match Pod's node affinity; debugInfo=

Note: The ASG group, which had that pod hosted in one of its nodes, was at max, and cannot spin new nodes. While the other ASGs could spin new nodes. Also, I tried to increase the number of nodes in the other ASGs, but the pod still could not be assign to those nodes with the same error. The CPU/RAM request and limit of the pod is less than of those of all ASGs.

What happened instead?:

The evicted pod should allocated to other nodes, or a new node from other ASGs should be added so that pod can be allocated.

How to reproduce it (as minimally and precisely as possible):

Possible scenario to reproduce: Have all the nodes of an ASG cordoned. Then, drain one of them, and then terminate it while reducing the max of that ASG by one at the same time.

Anything else we need to know?:

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 9
Comments: 17 (1 by maintainers)

Most upvoted comments

I saw this same error with one of our pods. The reason seemed to be because the pod had a nodeSelector that I didn’t have a tag hint for in my ASG. I never had issues with pod scheduling with this pod if a node without taints was available, only on scale from zero.

sidewinder12s on Jun 9, 2022

I’m running into this same issue. Is there a way for the cluster-autoscaler to say exhaustively which things did not match? It’s very hard to determine why a match isn’t being made from this error message.

malcolmgreaves on Dec 22, 2022

This error could occur when the node has reached the maximum pods it can schedule due to the maximum pods available for the node instance type and you try to deploy a Daemonset that is supposed to schedule a pod on that node.

For example t3.medium can schedule a max 17 pods.

From the command below to check pods current nodes can run:

% kubectl get nodes -o=custom-columns=NODE:.metadata.name,MAX_PODS:.status.allocatable.pods,CAPACITY_PODS:.status.capacity.pods,INSTANCE_TYPE:.metadata.labels."node\.kubernetes\.io/instance-type"

NODE                                            MAX_PODS   CAPACITY_PODS   INSTANCE_TYPE
ip-192-168-115-52.eu-west-1.compute.internal    17         17              t3.medium
ip-192-168-143-137.eu-west-1.compute.internal   17         17              t3.medium
ip-192-168-168-152.eu-west-1.compute.internal   17         17              t3.medium

Check pods running on these nodes:

% for node in $(k get nodes | awk '{if (NR!=1) {print $1}}'); do echo""; echo "Checking ${node}..."; k describe node ${node} | grep "Non-terminated" ; done


Checking ip-192-168-115-52.eu-west-1.compute.internal...
Non-terminated Pods:          (11 in total)

Checking ip-192-168-143-137.eu-west-1.compute.internal...
Non-terminated Pods:          (17 in total)

Checking ip-192-168-168-152.eu-west-1.compute.internal...
Non-terminated Pods:          (15 in total)

The daemonset below couldn’t schedule pod on the node: ip-192-168-143-137.eu-west-1.compute.internal

% kubectl --namespace=kube-system get pods -l "app=csi-secrets-store-provider-aws" -o wide                                                                 
NAME                                   READY   STATUS    RESTARTS   AGE   IP                NODE                                            NOMINATED NODE   READINESS GATES
csi-secrets-store-provider-aws-7rmxz   1/1     Running   0          27m   192.168.115.52    ip-192-168-115-52.eu-west-1.compute.internal    <none>           <none>
csi-secrets-store-provider-aws-gstxp   0/1     Pending   0          28m   <none>            <none>                                          <none>           <none>
csi-secrets-store-provider-aws-sthzx   1/1     Running   0          28m   192.168.168.152   ip-192-168-168-152.eu-west-1.compute.internal   <none>           <none>

Since each pod is assigned its own IP address, the number of IP addresses supported by an instance type is a factor in determining the number of pods that can run on the instance. Checkout Amazon EKS recommended maximum pods for each Amazon EC2 instance type

A way around this: Change the instance type to one you can run more pods.

berry2012 on Jul 22, 2022

I got the same error here, I have TAG the node group by k8s.io/cluster-autoscaler/node-template/label: something But using the node selector for the deployment it kept saying node(s) didn't match Pod's node affinity/selector; predicateName=NodeAffinity; reasons: node(s) didn't match Pod's node affinity/selector; I’m not sure why, I’m using EKS node group, and scale from 0

thi-peony on Apr 3, 2023