karpenter-provider-aws: Karpenter 0.31.0 can't create new node.

Description

Observed Behavior: getting this error on v0.31.0

karpenter-5dd479847d-kk6v6 controller 2023-10-02T06:57:48.315Z	ERROR	controller.provisioner	Could not schedule pod, incompatible with provisioner "monitoring", daemonset overhead={"cpu":"655m","memory":"628000Ki","pods":"7"}, no instance type satisfied resources {"cpu":"755m","memory":"1037600Ki","pods":"8"} and requirements karpenter.k8s.aws/instance-category In [c m r], karpenter.k8s.aws/instance-encryption-in-transit-supported In [true], karpenter.k8s.aws/instance-generation Exists >2, karpenter.sh/capacity-type In [on-demand spot], karpenter.sh/provisioner-name In [monitoring], kubernetes.io/arch In [amd64 arm64], kubernetes.io/os In [linux], nodetype In [monitoring], provisioner In [monitoring], topology.kubernetes.io/zone In [us-east-1d], type In [karpenter] (no instance type met the scheduling requirements or had a required offering); incompatible with provisioner "system", daemonset overhead={"cpu":"655m","memory":"628000Ki","pods":"7"}, did not tolerate CriticalAddonsOnly=true:NoSchedule; incompatible with provisioner "bitbucket", daemonset overhead={"cpu":"655m","memory":"628000Ki","pods":"7"}, did not tolerate bitbucket=true:NoSchedule; incompatible with provisioner "default", daemonset overhead={"cpu":"655m","memory":"628000Ki","pods":"7"}, did not tolerate default=true:NoSchedule	{"commit": "322822a", "pod": "monitoring/alertmanager-0"}

CRDs are installed from version 0.31.0. For some reason it can’t find any instances.

Expected Behavior: v0.30.0

karpenter-849c9cff87-tchft controller 2023-10-02T07:05:08.818Z	INFO	controller.provisioner	created machine	{"commit": "637a642", "provisioner": "monitoring", "machine": "monitoring-9zpps", "requests": {"cpu":"3955m","memory":"23108896Ki","pods":"10"}, "instance-types": "c5a.4xlarge, c5a.8xlarge, c5ad.4xlarge, c5ad.8xlarge, c5n.4xlarge and 95 other(s)"}

Reproduction Steps (Please include YAML): Install v0.31.0. Here is my provisioner and node configs:

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: bottlerocket
spec:
  blockDeviceMappings:
    - deviceName: /dev/xvdb
      ebs:
        volumeType: gp3
        volumeSize: 40Gi
        deleteOnTermination: true
  amiFamily: Bottlerocket
  instanceProfile: ${iam-instance-profile-id}
  subnetSelector:
    aws-ids: "${aws-ids}"
  securityGroupSelector:
    karpenter.sh/discovery/${eks-cluster-id}: "${eks-cluster-id}"
  metadataOptions: # require IMDSv2
    httpEndpoint: enabled
    httpProtocolIPv6: disabled
    httpPutResponseHopLimit: 1
    httpTokens: required
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: monitoring
spec:
  requirements:
    - key: "topology.kubernetes.io/zone"
      operator: In
      values: [${azs}]
    - key: "karpenter.sh/capacity-type"
      operator: In
      values: ["spot", "on-demand"]
    - key: "kubernetes.io/arch"
      operator: In
      values: ["arm64", "amd64"]
    - key: "karpenter.k8s.aws/instance-encryption-in-transit-supported"
      operator: In
      values: ["true"]
  kubeletConfiguration:
    containerRuntime: containerd
    systemReserved:
      cpu: 100m
      memory: 100Mi
      ephemeral-storage: 1Gi
    kubeReserved:
      cpu: 200m
      memory: 100Mi
      ephemeral-storage: 3Gi
  limits:
    resources:
      cpu: 100
      memory: 400Gi
  providerRef:
    name: bottlerocket
  labels:
    type: karpenter
    nodetype: monitoring
    provisioner: monitoring
  taints:
    - key: monitoring
      value: "true"
      effect: NoSchedule
  ttlSecondsUntilExpired: 604800
  consolidation:
    enabled: true

Versions:

  • Chart Version: 0.31.0
  • Kubernetes Version (kubectl version): Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.1-eks-43840fb
  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave “+1” or “me too” comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

About this issue

  • Original URL
  • State: closed
  • Created 9 months ago
  • Reactions: 2
  • Comments: 28 (11 by maintainers)

Most upvoted comments

Assigning @engedaam as the POC for taking up the fix. We should add this to the v0.32.2 patch version.

but with v0.31 and higher it must be like this

Ah, yep. That definitely looks like a bug. We should still support the comma-delimited list with spaces in it. We’ll get a PR out to fix this.