karpenter-provider-aws: Karpenter is not respecting per-node Daemonsets

Version

Karpenter: v0.7.2

Kubernetes: v1.21.5

Context

We run several different daemonsets on a per-nodes basis: metrics, logging, EBS CSI, secrets-store CSI. These need to be present on every node as they provide their functionality to every pod on a node.

(This could be a configuration / unset flag issue, looking for more information)

Expected Behavior

When choosing an instance type to provision for pending pods, Karpenter should take into account any Daemonsets that will be running on the node, not just the pending service pods that it will schedule there.

Actual Behavior

This is most noticeable in a brand new cluster, but has also been seen with mature clusters: When Karpenter brings up a node, it will correctly calculate the resources required to support the new service pod / replica. The aws-node and kube-proxy pods will be started and then the service pod.

When using a larger metrics / logging / CSI pod with requests of e.g 1Gb RAM / 0.5-1 CPU each, these pods will be perpetually stuck in a pending state and will never start, as there isn’t enough room on the node for them.

This was most noticeable when creating a new cluster when the aws-load-balancer-controller was deployed, which only requires 0.05 CPU. Therefore even with 3 replicas, Karpenter spun up a t3a.small instance to support these. Even when adding more replicas (tested with 25 replicas), it continued to spin up t3a.small instances, presumably because they were the cheapest option, but leaving all the daemonset pods in a pending state, apart from one node where there was only one aws-load-balancer-controller pod - in this case one of the daemonset pods started, the rest were stuck in pending.

I believe this is due to how Karpenter is scheduling the pods on the node (something about node-binding in the docs?):

• As aws-node and kube-proxy are in the system-node-critical priority_class, they are always scheduled first
• Potentially Karpenter is then scheduling the service pod next
• The other daemonsets, some with a much higher priority_class, are not scheduled until after the service pod and therefore get stuck in a pending state if there is not enough room for them

Steps to Reproduce the Problem

• Create a fresh cluster with Karpenter deployed and a default provisioner
• Create n daemonsets with a highish resource consumption that will run on every node
• Create a service deployment for a service with very low resource consumption, using the node selector for a karpenter provisioner
• Karpenter should select an instance-type suitable for the service pods, but not able to support the daemonset(s)

Resource Specs and Logs

### Default Provisioner
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  annotations:
    meta.helm.sh/release-name: karpenter-default-provisioner-chart
    meta.helm.sh/release-namespace: default
  labels:
    app.kubernetes.io/managed-by: Helm
  name: karpenter-default
spec:
  labels:
    env: karpenter-default
  provider:
    apiVersion: extensions.karpenter.sh/v1alpha1
    kind: AWS
    launchTemplate: <launch_template>
    subnetSelector:
      Service: Private
      kubernetes.io/cluster/<cluster_name>: '*'
  requirements:
  - key: karpenter.sh/capacity-type
    operator: In
    values:
    - on-demand
    - spot
  - key: kubernetes.io/arch
    operator: In
    values:
    - amd64
  ttlSecondsAfterEmpty: 30

Logs

Do not have access to these logs at this time - but it was correctly trying to schedule the pending pods, and calculating the instance size based on the service pod requests