karpenter: karpenter does not add `karpenter.sh/capacity-type` label to the nodes

Description

Observed Behavior: For the nodes which is karpenter spinning up, karpenter is not adding karpenter.sh/capacity-type as both machine and provisioner has this label. Expected Behavior: karpenter should add the expected label for consolidation to work. Reproduction Steps (Please include YAML): provisionerspec

Name:         core-karpenter-ethos-core-karpenter-worker2
Namespace:    
Labels:       adobe.com/appname=karpenter-ethos101-dev-va6
              app.kubernetes.io/instance=core-karpenter
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ethos-core-karpenter
              app.kubernetes.io/version=v0.31.0
              ethos.adobe.net/managed-by=k8s-infrastructure.helm
              helm.sh/chart=ethos-core-karpenter-0.0.5
Annotations:  karpenter.sh/provisioner-hash: 1417663675986826836
API Version:  karpenter.sh/v1alpha5
Kind:         Provisioner
Metadata:
  Creation Timestamp:  2023-10-28T00:44:25Z
  Generation:          2
  Resource Version:    2306126405
  UID:                 6a302235-6c20-4a36-aa03-1c31caa7c7f3
Spec:
  Consolidation:
    Enabled:  true
  Labels:
    ethos.adobe.net/node-templateVersion:     a8a12c5ee31a4668f93b01ccf92ebf7c54679f693bd3a14b4bd86086432388
    kubernetes.io/os:                         linux
    node.kubernetes.io/ethos-workload.amd64:  true
    node.kubernetes.io/node-group:            worker2-group
    node.kubernetes.io/role:                  worker
  Limits:
    Resources:
      Cpu:     1k
      Memory:  1000Gi
  Provider Ref:
    Name:  core-karpenter-ethos-core-karpenter-worker2
  Requirements:
    Key:       karpenter.k8s.aws/instance-family
    Operator:  In
    Values:
      c5
      c6g
      m5
      m6g
      r5
      r6g
    Key:       karpenter.k8s.aws/instance-generation
    Operator:  Gt
    Values:
      4
    Key:       kubernetes.io/arch
    Operator:  In
    Values:
      amd64
    Key:       karpenter.sh/capacity-type
    Operator:  In
    Values:
      on-demand
    Key:       topology.kubernetes.io/zone
    Operator:  In
    Values:
      us-east-1a
      us-east-1b
      us-east-1c
    Key:       kubernetes.io/os
    Operator:  In
    Values:
      linux
Status:
  Resources:
    Attachable - Volumes - Aws - Ebs:  25
    Cpu:                               16
    Ephemeral - Storage:               194314260Ki
    Memory:                            32054796Ki
    Pods:                              112
Events:                                <none>

machinespec

Name:         core-karpenter-ethos-core-karpenter-worker2-zjxnb
Namespace:    
Labels:       ethos.adobe.net/node-templateVersion=a8a12c5ee31a4668f93b01ccf92ebf7c54679f693bd3a14b4bd86086432388
              karpenter.k8s.aws/instance-category=c
              karpenter.k8s.aws/instance-cpu=72
              karpenter.k8s.aws/instance-encryption-in-transit-supported=false
              karpenter.k8s.aws/instance-family=c5
              karpenter.k8s.aws/instance-generation=5
              karpenter.k8s.aws/instance-hypervisor=nitro
              karpenter.k8s.aws/instance-memory=147456
              karpenter.k8s.aws/instance-network-bandwidth=25000
              karpenter.k8s.aws/instance-pods=737
              karpenter.k8s.aws/instance-size=18xlarge
              karpenter.sh/capacity-type=on-demand
              karpenter.sh/provisioner-name=core-karpenter-ethos-core-karpenter-worker2
              kubernetes.io/arch=amd64
              kubernetes.io/os=linux
              node.kubernetes.io/ethos-workload.amd64=true
              node.kubernetes.io/instance-type=c5.18xlarge
              node.kubernetes.io/node-group=worker2-group
              node.kubernetes.io/role=worker
              topology.kubernetes.io/region=us-east-1
              topology.kubernetes.io/zone=us-east-1a
Annotations:  karpenter.k8s.aws/nodetemplate-hash: 7594172995892393789
              karpenter.sh/managed-by: ethos101-dev-va6-k8s-eks-master
              karpenter.sh/provisioner-hash: 1417663675986826836
API Version:  karpenter.sh/v1alpha5
Kind:         Machine
Metadata:
  Creation Timestamp:  2023-10-30T19:02:52Z
  Finalizers:
    karpenter.sh/termination
  Generate Name:  core-karpenter-ethos-core-karpenter-worker2-
  Generation:     1
  Owner References:
    API Version:           karpenter.sh/v1alpha5
    Block Owner Deletion:  true
    Kind:                  Provisioner
    Name:                  core-karpenter-ethos-core-karpenter-worker2
    UID:                   6a302235-6c20-4a36-aa03-1c31caa7c7f3
  Resource Version:        2306109951
  UID:                     6b20ee72-270f-428e-98e4-dad254147d58
Spec:
  Machine Template Ref:
    Name:  core-karpenter-ethos-core-karpenter-worker2
  Requirements:
    Key:       karpenter.sh/provisioner-name
    Operator:  In
    Values:
      core-karpenter-ethos-core-karpenter-worker2
    Key:       ethos.adobe.net/node-templateVersion
    Operator:  In
    Values:
      a8a12c5ee31a4668f93b01ccf92ebf7c54679f693bd3a14b4bd86086432388
    Key:       karpenter.sh/capacity-type
    Operator:  In
    Values:
      on-demand
    Key:       topology.kubernetes.io/zone
    Operator:  In
    Values:
      us-east-1a
      us-east-1b
    Key:       node.kubernetes.io/ethos-workload.amd64
    Operator:  In
    Values:
      true
    Key:       node.kubernetes.io/role
    Operator:  In
    Values:
      worker
    Key:       karpenter.k8s.aws/instance-generation
    Operator:  Gt
    Values:
      4
    Key:       kubernetes.io/arch
    Operator:  In
    Values:
      amd64
    Key:       kubernetes.io/os
    Operator:  In
    Values:
      linux
    Key:       node.kubernetes.io/instance-type
    Operator:  In
    Values:
      c5.12xlarge
      c5.18xlarge
      c5.24xlarge
      c5.4xlarge
      c5.9xlarge
      c5.metal
      m5.12xlarge
      m5.16xlarge
      m5.24xlarge
      m5.4xlarge
      m5.8xlarge
      m5.metal
      r5.12xlarge
      r5.16xlarge
      r5.24xlarge
      r5.4xlarge
      r5.8xlarge
      r5.metal
    Key:       node.kubernetes.io/node-group
    Operator:  In
    Values:
      worker2-group
    Key:       karpenter.k8s.aws/instance-family
    Operator:  In
    Values:
      c5
      c6g
      m5
      m6g
      r5
      r6g
  Resources:
    Requests:
      Cpu:     10237m
      Memory:  23845317289
      Pods:    19
Status:
  Allocatable:
    Cpu:                  71750m
    Ephemeral - Storage:  179Gi
    Memory:               127934Mi
    Pods:                 737
  Capacity:
    Cpu:                  72
    Ephemeral - Storage:  200Gi
    Memory:               136396Mi
    Pods:                 737
  Conditions:
    Last Transition Time:  2023-10-30T19:04:45Z
    Status:                True
    Type:                  MachineInitialized
    Last Transition Time:  2023-10-30T19:02:55Z
    Status:                True
    Type:                  MachineLaunched
    Last Transition Time:  2023-10-30T19:04:45Z
    Status:                True
    Type:                  Ready
    Last Transition Time:  2023-10-30T19:04:13Z
    Status:                True
    Type:                  MachineRegistered
  Node Name:               ip-10-95-64-98.ec2.internal
  Provider ID:             aws:///us-east-1a/i-0fc6c888b14b25ac9
Events:
  Type    Reason                 Age                  From       Message
  ----    ------                 ----                 ----       -------
  Normal  DeprovisioningBlocked  30s (x73 over 152m)  karpenter  Cannot deprovision Machine: Required label "karpenter.sh/capacity-type" doesn't exist

error on the node same as of machine Versions:

• Chart Version: 0.31.0
• Kubernetes Version (kubectl version): v1.26.9

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave “+1” or “me too” comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment