karpenter-provider-aws: Karpenter nodes stuck in "NotReady" status and fail to join EKS cluster

We are also running cilium and eks. We didn’t had any issues when running the amazon-ec2 images for our nodes also ssm is working.

I tried to reproduce your problem and its repeatable for me.

SSM works with Bottlerocket and Cilium

It seems there is is a configuration issue in your setup. 😃

What we do for our nodes is the following in terraform:

Setting up the eks-irsa-role like this:

resource "aws_iam_instance_profile" "karpenter" {
  name = "KarpenterNodeInstanceProfile-${local.eks_cluster_name}"
  role = module.eks-ng.eks_managed_node_groups["worker"].iam_role_name
}

module "karpenter_irsa" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "5.3.0"

  role_name                          = "karpenter-controller-${local.eks_cluster_name}"
  attach_karpenter_controller_policy = true

  karpenter_controller_cluster_id = module.eks-ng.cluster_id
  karpenter_controller_node_iam_role_arns = [
    module.eks-ng.eks_managed_node_groups["worker"].iam_role_arn
  ]

  oidc_providers = {
    ex = {
      provider_arn               = module.eks-ng.oidc_provider_arn
      namespace_service_accounts = ["kube-system:karpenter"]
    }
  }

Important is here the karpenter_controller_node_iam_role_arns there we are referencing the default node role from the eks managed node groups.

In the default managed node group we are passing the required ssm policy to ensure that ssm is working.

  eks_managed_node_group_defaults = {
    ami_type = "AL2_x86_64"
    instance_types = [
      "m5a.large"
    ]

    taints = [
      {
        key    = "node.cilium.io/agent-not-ready"
        value  = "true"
        effect = "NO_EXECUTE"
      }
    ]


    bootstrap_extra_args = "--kubelet-extra-args '--max-pods=100'"
    iam_role_additional_policies = [
      "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
    ]

    block_device_mappings = {
      xvda = {
        device_name = "/dev/xvda"
        ebs = {
          volume_size           = 50
          volume_type           = "gp3"
          iops                  = 1000
          throughput            = 250
          encrypted             = true
          delete_on_termination = true
        }
      }
    }

    post_bootstrap_user_data = <<-EOT
      cd /tmp
      sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
      sudo systemctl enable amazon-ssm-agent
      sudo systemctl start amazon-ssm-agent
    EOT
  }

Check that your nodes have the policy for AmazonSSMManagedInstanceCore than it should work out of the box. Since already mentioned ssm is preinstalled.

Cilium and Bottlerocket Troubleshoot

First of all I can see the same behaviour

ip-10-20-35-219.eu-central-1.compute.internal   NotReady   <none>   22m     v1.23.7-eks-7709a84

I could connect to the instance with ssm without any problems.

aws ssm start-session --target i-03fede25ff7580879

/bin/bash
          Welcome to Bottlerocket's control container!
[ssm-user@control]$ /bin/bash
[ssm-user@control]$ enter-admin-container
Confirming admin container is enabled...
Waiting for admin container to start......
Entering admin container
[root@admin]# sudo sheltie
bash-5.1# journalctl -u kubelet

Interessting Logs from the Kubelet

Sep 11 06:58:55 10.20.35.219 kubelet[1583]: E0911 06:58:55.039163    1583 remote_runtime.go:704] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = failed to exec in container: failed to create exec \"76a6dcaed6a7de6294437a02e20d6b740d46eadc92d12ce2275a833753051717\": cannot exec in a stopped state: unknown" containerID="beaa01acf2c9874853e99ff42c7608d0098ef9f8b8daf74f2a42c3777f05858e" cmd=[nsenter --target=1 --mount -- /bin/sh -c #!/bin/bash
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: set -o errexit
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: set -o pipefail
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: set -o nounset
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # When running in AWS ENI mode, it's likely that 'aws-node' has
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # had a chance to install SNAT iptables rules. These can result
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # in dropped traffic, so we should attempt to remove them.
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # We do it using a 'postStart' hook since this may need to run
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # for nodes which might have already been init'ed but may still
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # have dangling rules. This is safe because there are no
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # dependencies on anything that is part of the startup script
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # itself, and can be safely run multiple times per node (e.g. in
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # case of a restart).
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: if [[ "$(iptables-save | grep -c AWS-SNAT-CHAIN)" != "0" ]];
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: then
Sep 11 06:58:55 10.20.35.219 kubelet[1583]:     echo 'Deleting iptables rules created by the AWS CNI VPC plugin'
Sep 11 06:58:55 10.20.35.219 kubelet[1583]:     iptables-save | grep -v AWS-SNAT-CHAIN | iptables-restore
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: fi
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: echo 'Done!'
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: ]
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: E0911 06:58:55.039253    1583 kuberuntime_container.go:284] "Failed to execute PostStartHook" err="rpc error: code = Unknown desc = failed to exec in container: failed to create exec \"76a6dcaed6a7de6294437a02e20d6b740d46eadc92d12ce2275a833753051717\": cannot exec in a stopped state: unknown" pod="kube-system/cilium-node-init-9767n" podUID=cd02bb16-c6ea-4978-bd3a-ffafb9566c97 containerName="node-init" containerID="containerd://beaa01acf2c9874853e99ff42c7608d0098ef9f8b8daf74f2a42c3777f05858e"
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: E0911 06:58:55.117006    1583 kuberuntime_manager.go:919] container &Container{Name:node-init,Image:quay.io/cilium/startup-script:d69851597ea019af980891a4628fb36b7880ec26,Command:[],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:STARTUP_SCRIPT,Value:#!/bin/bash
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: set -o errexit
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: set -o pipefail
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: set -o nounset
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: echo "Link information:"
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: ip link
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: echo "Routing table:"
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: ip route
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: echo "Addressing:"
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: ip -4 a
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: ip -6 a
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: mkdir -p "/tmp/cilium-bootstrap.d"
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: date > "/tmp/cilium-bootstrap.d/cilium-bootstrap-time"
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: echo "Node initialization complete"
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: ,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{100 -3} {<nil>} 100m DecimalSI},memory: {{104857600 0} {<nil>} 100Mi BinarySI},},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-f89lv,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:&Lifecycle{PostStart:&LifecycleHandler{Exec:&ExecAction{Command:[nsenter --target=1 --mount -- /bin/sh -c #!/bin/bash
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: set -o errexit
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: set -o pipefail
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: set -o nounset
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # When running in AWS ENI mode, it's likely that 'aws-node' has
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # had a chance to install SNAT iptables rules. These can result
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # in dropped traffic, so we should attempt to remove them.
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # We do it using a 'postStart' hook since this may need to run
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # for nodes which might have already been init'ed but may still
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # have dangling rules. This is safe because there are no
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # dependencies on anything that is part of the startup script
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # itself, and can be safely run multiple times per node (e.g. in
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # case of a restart).
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: if [[ "$(iptables-save | grep -c AWS-SNAT-CHAIN)" != "0" ]];
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: then
Sep 11 06:58:55 10.20.35.219 kubelet[1583]:     echo 'Deleting iptables rules created by the AWS CNI VPC plugin'
Sep 11 06:58:55 10.20.35.219 kubelet[1583]:     iptables-save | grep -v AWS-SNAT-CHAIN | iptables-restore
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: fi
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: echo 'Done!'
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: ],},HTTPGet:nil,TCPSocket:nil,},PreStop:nil,},TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[SYS_MODULE NET_ADMIN SYS_ADMIN SYS_CHROOT SYS_PTRACE],Drop:[],},Privileged:*false,SELinuxOptions:&SELinuxOptions{User:,Role:,Type:spc_t,Level:s0,},RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod cilium-node-init-9767n_kube-system(cd02bb16-c6ea-4978-bd3a-ffafb9566c97): PostStartHookError: Exec lifecycle hook ([nsenter --target=1 --mount -- /bin/sh -c #!/bin/bash
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: set -o errexit
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: set -o pipefail
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: set -o nounset
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # When running in AWS ENI mode, it's likely that 'aws-node' has
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # had a chance to install SNAT iptables rules. These can result
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # in dropped traffic, so we should attempt to remove them.
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # We do it using a 'postStart' hook since this may need to run
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # for nodes which might have already been init'ed but may still
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # have dangling rules. This is safe because there are no
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # dependencies on anything that is part of the startup script
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # itself, and can be safely run multiple times per node (e.g. in
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: # case of a restart).
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: if [[ "$(iptables-save | grep -c AWS-SNAT-CHAIN)" != "0" ]];
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: then
Sep 11 06:58:55 10.20.35.219 kubelet[1583]:     echo 'Deleting iptables rules created by the AWS CNI VPC plugin'
Sep 11 06:58:55 10.20.35.219 kubelet[1583]:     iptables-save | grep -v AWS-SNAT-CHAIN | iptables-restore
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: fi
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: echo 'Done!'
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: ]) for Container "node-init" in Pod "cilium-node-init-9767n_kube-system(cd02bb16-c6ea-4978-bd3a-ffafb9566c97)" failed - error: rpc error: code = Unknown desc = failed to exec in container: failed to create exec "76a6dcaed6a7de6294437a02e20d6b740d46eadc92d12ce2275a833753051717": cannot exec in a stopped state: unknown, message: ""
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: E0911 06:58:55.117106    1583 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"node-init\" with PostStartHookError: \"Exec lifecycle hook ([nsenter --target=1 --mount -- /bin/sh -c #!/bin/bash\\n\\nset -o errexit\\nset -o pipefail\\nset -o nounset\\n\\n# When running in AWS ENI mode, it's likely that 'aws-node' has\\n# had a chance to install SNAT iptables rules. These can result\\n# in dropped traffic, so we should attempt to remove them.\\n# We do it using a 'postStart' hook since this may need to run\\n# for nodes which might have already been init'ed but may still\\n# have dangling rules. This is safe because there are no\\n# dependencies on anything that is part of the startup script\\n# itself, and can be safely run multiple times per node (e.g. in\\n# case of a restart).\\nif [[ \\\"$(iptables-save | grep -c AWS-SNAT-CHAIN)\\\" != \\\"0\\\" ]];\\nthen\\n    echo 'Deleting iptables rules created by the AWS CNI VPC plugin'\\n    iptables-save | grep -v AWS-SNAT-CHAIN | iptables-restore\\nfi\\necho 'Done!'\\n]) for Container \\\"node-init\\\" in Pod \\\"cilium-node-init-9767n_kube-system(cd02bb16-c6ea-4978-bd3a-ffafb9566c97)\\\" failed - error: rpc error: code = Unknown desc = failed to exec in container: failed to create exec \\\"76a6dcaed6a7de6294437a02e20d6b740d46eadc92d12ce2275a833753051717\\\": cannot exec in a stopped state: unknown, message: \\\"\\\"\"" pod="kube-system/cilium-node-init-9767n" podUID=cd02bb16-c6ea-4978-bd3a-ffafb9566c97
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: I0911 06:58:55.293941    1583 scope.go:110] "RemoveContainer" containerID="d68c37d3e92f32cbe10dc5c45c442475e3c5204f1d61c0a29c1de0f2c775e806"
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: I0911 06:58:55.294399    1583 scope.go:110] "RemoveContainer" containerID="beaa01acf2c9874853e99ff42c7608d0098ef9f8b8daf74f2a42c3777f05858e"
Sep 11 06:58:55 10.20.35.219 kubelet[1583]: E0911 06:58:55.294914    1583 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"node-init\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=node-init pod=cilium-node-init-9767n_kube-system(cd02bb16-c6ea-4978-bd3a-ffafb9566c97)\"" pod="kube-system/cilium-node-init-9767n" podUID=cd02bb16-c6ea-4978-bd3a-ffafb9566c97

The main problem is that the node-init Pod from Cilium can’t start since it doesn’t have a normal bash on Bottlerocket.

After some googling I found a possible solution for it.

You need only to disable the node-init pod with nodeinit.enabled=false on bottlerocket and everything should work as excepted.

helm install cilium cilium/cilium  \
--version 1.9.5 \
--namespace kube-system \
--set eni=true \
--set ipam.mode=eni \
--set egressMasqueradeInterfaces=eth0 \
--set tunnel=disabled \
—set nodeinit.enabled=false

https://github.com/bottlerocket-os/bottlerocket/issues/1405#issuecomment-804196007

I don’t have time to test it fully in cluster with the networking change.

@triceras Would be nice to hear your feedback if it works for you.

I hope this helps some people. :neckbeard:

Bottlerocket and amazon Linux AMIs come with SSM preinstalled.