karpenter-provider-aws: Karpenter 0.31.0 can't create new node.

Description

Observed Behavior: getting this error on v0.31.0

karpenter-5dd479847d-kk6v6 controller 2023-10-02T06:57:48.315Z	ERROR	controller.provisioner	Could not schedule pod, incompatible with provisioner "monitoring", daemonset overhead={"cpu":"655m","memory":"628000Ki","pods":"7"}, no instance type satisfied resources {"cpu":"755m","memory":"1037600Ki","pods":"8"} and requirements karpenter.k8s.aws/instance-category In [c m r], karpenter.k8s.aws/instance-encryption-in-transit-supported In [true], karpenter.k8s.aws/instance-generation Exists >2, karpenter.sh/capacity-type In [on-demand spot], karpenter.sh/provisioner-name In [monitoring], kubernetes.io/arch In [amd64 arm64], kubernetes.io/os In [linux], nodetype In [monitoring], provisioner In [monitoring], topology.kubernetes.io/zone In [us-east-1d], type In [karpenter] (no instance type met the scheduling requirements or had a required offering); incompatible with provisioner "system", daemonset overhead={"cpu":"655m","memory":"628000Ki","pods":"7"}, did not tolerate CriticalAddonsOnly=true:NoSchedule; incompatible with provisioner "bitbucket", daemonset overhead={"cpu":"655m","memory":"628000Ki","pods":"7"}, did not tolerate bitbucket=true:NoSchedule; incompatible with provisioner "default", daemonset overhead={"cpu":"655m","memory":"628000Ki","pods":"7"}, did not tolerate default=true:NoSchedule	{"commit": "322822a", "pod": "monitoring/alertmanager-0"}

CRDs are installed from version 0.31.0. For some reason it can’t find any instances.

Expected Behavior: v0.30.0

karpenter-849c9cff87-tchft controller 2023-10-02T07:05:08.818Z	INFO	controller.provisioner	created machine	{"commit": "637a642", "provisioner": "monitoring", "machine": "monitoring-9zpps", "requests": {"cpu":"3955m","memory":"23108896Ki","pods":"10"}, "instance-types": "c5a.4xlarge, c5a.8xlarge, c5ad.4xlarge, c5ad.8xlarge, c5n.4xlarge and 95 other(s)"}

Reproduction Steps (Please include YAML): Install v0.31.0. Here is my provisioner and node configs:

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: bottlerocket
spec:
  blockDeviceMappings:
    - deviceName: /dev/xvdb
      ebs:
        volumeType: gp3
        volumeSize: 40Gi
        deleteOnTermination: true
  amiFamily: Bottlerocket
  instanceProfile: ${iam-instance-profile-id}
  subnetSelector:
    aws-ids: "${aws-ids}"
  securityGroupSelector:
    karpenter.sh/discovery/${eks-cluster-id}: "${eks-cluster-id}"
  metadataOptions: # require IMDSv2
    httpEndpoint: enabled
    httpProtocolIPv6: disabled
    httpPutResponseHopLimit: 1
    httpTokens: required

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: monitoring
spec:
  requirements:
    - key: "topology.kubernetes.io/zone"
      operator: In
      values: [${azs}]
    - key: "karpenter.sh/capacity-type"
      operator: In
      values: ["spot", "on-demand"]
    - key: "kubernetes.io/arch"
      operator: In
      values: ["arm64", "amd64"]
    - key: "karpenter.k8s.aws/instance-encryption-in-transit-supported"
      operator: In
      values: ["true"]
  kubeletConfiguration:
    containerRuntime: containerd
    systemReserved:
      cpu: 100m
      memory: 100Mi
      ephemeral-storage: 1Gi
    kubeReserved:
      cpu: 200m
      memory: 100Mi
      ephemeral-storage: 3Gi
  limits:
    resources:
      cpu: 100
      memory: 400Gi
  providerRef:
    name: bottlerocket
  labels:
    type: karpenter
    nodetype: monitoring
    provisioner: monitoring
  taints:
    - key: monitoring
      value: "true"
      effect: NoSchedule
  ttlSecondsUntilExpired: 604800
  consolidation:
    enabled: true

Versions:

• Chart Version: 0.31.0
• Kubernetes Version (kubectl version): Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.1-eks-43840fb

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave “+1” or “me too” comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment