karpenter: Unable to schedule daemonset pods due to resource constraints on provisioned nodes

Description

Observed Behavior: When karpenter provisioned a node for the specific workloads to run, the instance size it selects from the provisioner config is not big enough to host all the workloads and deaomsets, which leaving us with a scenario where a daemonset pod is stuck in pending state with this error:

0/26 nodes are available: 1 Insufficient cpu, 22 node(s) didn't match Pod's node affinity/selector, 3 node(s) had taint {CriticalAddonsOnly: }, that the pod didn't tolerate.

it supposed to get scheduled on to the node which doesn’t have CPUs left because it has other pods running on it before daemonset could schedule itself on to it

Expected Behavior:

Daemonset pod should be in running state.

when a new node is provisioned it should consider all the available daemonset and its resource requirements then add any other required resources on top of it, then decide which would be the right instance size for the workload to run.

Reproduction Steps (Please include YAML):

Below is the provisioner yaml which karpenter used to schedule a node and as we can see that sizes as large, xlarge, 2xlarge , the node which it provisioned is of type instance-type=t3.large , if it had provisioned xlarge instead we wouldn’t be in the current situation and have a daemonset in pending state.

Provisioner yaml:

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: frontend-app-nodes
spec:
  labels:
    role: frontend-app-nodes
  kubeletConfiguration:
    maxPods: 110
  consolidation:
    enabled: true
  limits:
    resources:
      cpu: 1000
      memory: 1000Gi
  requirements:
    - key: karpenter.k8s.aws/instance-category
      operator: In
      values: ["t", "m"]
    - key: karpenter.k8s.aws/instance-generation
      operator: Gt
      values: ["2"]
    - key: karpenter.k8s.aws/instance-size
      operator: In
      values: ["large", "xlarge", "2xlarge"]
    - key: kubernetes.io/arch
      operator: In
      values: ["amd64"]
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["spot"]
    - key: karpenter.k8s.aws/instance-hypervisor
      operator: In
      values: ["nitro"]
  providerRef:
    name: karpenter-node-template

Versions:

• Chart Version: v0.25.0
• Kubernetes Version (kubectl version): v1.22.17-eks-0a21954

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave “+1” or “me too” comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment