karpenter-provider-aws: Karpenter nodes stuck in "NotReady" status and fail to join EKS cluster
Version
Karpenter: v0.16.1
Kubernetes: v1.22.9
Expected Behavior
The pods should be schedule on the nodes spun up by Karpenter
Actual Behavior
No pods are being scheduled onto the Karpenter nodes. Those nodes are stuck in a NotReady
status and show the same error related to a failed lease: Lease: Failed to get lease: leases.coordination.k8s.io "ip-10-50-34-118.ec2.internal" not found
This is a continuation of the issue #2286 which was closed prematurely and the Karpenter nodes still present the same issue reported in there.
The EKS cluster nodes are using bottlerocket OS and cilium is the CNI. We are using terraform for provisioning the EKS cluster.
Cilium version: 1.12.1 Bottlerocket AMI ID: ami-0ab0c02538ad82487
Steps to Reproduce the Problem
The Provisioner and AWSNodeTemplates are below
# kubectl describe provisioner default
Name: default
Namespace:
Labels: cartax.io/argo-instance=eks-rabdalla-dev-karpenter
Annotations: <none>
API Version: karpenter.sh/v1alpha5
Kind: Provisioner
Metadata:
Creation Timestamp: 2022-08-30T22:15:14Z
Generation: 5
Managed Fields:
API Version: karpenter.sh/v1alpha5
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:labels:
.:
f:cartax.io/argo-instance:
f:spec:
.:
f:limits:
.:
f:resources:
.:
f:cpu:
f:providerRef:
.:
f:name:
f:requirements:
f:startupTaints:
f:ttlSecondsAfterEmpty:
f:ttlSecondsUntilExpired:
Manager: argocd-application-controller
Operation: Update
Time: 2022-08-31T18:35:37Z
Resource Version: 941471
UID: 49f8c266-4502-4a7c-8346-070e7b268cd2
Spec:
Limits:
Resources:
Cpu: 1k
Provider Ref:
Name: default
Requirements:
Key: kubernetes.io/arch
Operator: In
Values:
amd64
Key: topology.kubernetes.io/zone
Operator: In
Values:
us-east-1a
us-east-1b
us-east-1c
Key: karpenter.sh/capacity-type
Operator: In
Values:
on-demand
Key: node.kubernetes.io/instance-type
Operator: In
Values:
t3.xlarge
Startup Taints:
Effect: NoExecute
Key: node.cilium.io/agent-not-ready
Value: true
Ttl Seconds After Empty: 60
Ttl Seconds Until Expired: 2592000
Status:
Events: <none>
---
# kubectl describe awsnodetemplate default
Name: default
Namespace:
Labels: cartax.io/argo-instance=eks-rabdalla-dev-karpenter
Annotations: <none>
API Version: karpenter.k8s.aws/v1alpha1
Kind: AWSNodeTemplate
Metadata:
Creation Timestamp: 2022-08-30T17:52:51Z
Generation: 7
Managed Fields:
API Version: karpenter.k8s.aws/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:labels:
.:
f:cartax.io/argo-instance:
f:spec:
.:
f:amiFamily:
f:amiSelector:
.:
f:aws-ids:
f:blockDeviceMappings:
f:instanceProfile:
f:securityGroupSelector:
.:
f:karpenter.sh/discovery:
f:subnetSelector:
.:
f:karpenter.sh/discovery:
f:userData:
Manager: argocd-application-controller
Operation: Update
Time: 2022-08-30T19:21:15Z
Resource Version: 886319
UID: 2f1734ae-f2c7-4403-ad18-bfd0c726b736
Spec:
Ami Family: Bottlerocket
Ami Selector:
Aws - Ids: ami-0ab0c02538ad82487
Block Device Mappings:
Device Name: /dev/xvda
Ebs:
Delete On Termination: true
Volume Size: 60Gi
Volume Type: gp2
Instance Profile: KarpenterNodeInstanceProfile-eks-rabdalla-dev
Security Group Selector:
karpenter.sh/discovery: eks-rabdalla-dev
Subnet Selector:
karpenter.sh/discovery: eks-rabdalla-dev
User Data: [settings.kernel]
"lockdown" = "integrity"
[settings.kernel.sysctl]
"net.ipv4.conf.default.rp_filter" = 0
"net.ipv4.conf.all.rp_filter" = 0
"fs.inotify.max_user_watches" = 1048576
Events: <none>
The IRSA Kubernetes module used is the following:
module "karpenter_irsa" {
source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
version = "5.3.1"
role_name = "karpenter-controller-${local.name}"
attach_karpenter_controller_policy = true
karpenter_tag_key = "karpenter.sh/discovery"
karpenter_controller_cluster_id = var.cluster_id
karpenter_controller_node_iam_role_arns = [
module.eks_managed_node_group.iam_role_arn
]
oidc_providers = {
main = {
provider_arn = var.oidc_provider_arn
namespace_service_accounts = ["karpenter:karpenter"]
}
}
tags = local.tags
}
The final state of the nodes is as follows:
k get nodes
NAME STATUS ROLES AGE VERSION
ip-10-50-110-116.ec2.internal NotReady <none> 67s
ip-10-50-111-149.ec2.internal Ready <none> 26h v1.22.9-eks-0857b39
ip-10-50-119-64.ec2.internal NotReady <none> 67s
ip-10-50-131-21.ec2.internal Ready <none> 26h v1.22.9-eks-0857b39
ip-10-50-158-184.ec2.internal Ready <none> 26h v1.22.9-eks-0857b39
ip-10-50-185-72.ec2.internal Ready <none> 26h v1.22.9-eks-0857b39
ip-10-50-5-178.ec2.internal Ready <none> 26h v1.22.9-eks-0857b39
ip-10-50-52-0.ec2.internal Ready <none> 26h v1.22.9-eks-0857b39
ip-10-50-59-107.ec2.internal Ready <none> 26h v1.22.9-eks-0857b39
ip-10-50-65-77.ec2.internal Ready <none> 26h v1.22.9-eks-0857b39
ip-10-50-72-199.ec2.internal NotReady <none> 67s
ip-10-50-83-35.ec2.internal Ready <none> 26h v1.22.9-eks-0857b39
The state of one of the karpenter managed nodes:
k describe node ip-10-50-119-64.ec2.internal
Name: ip-10-50-119-64.ec2.internal
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
k8s.io/cloud-provider-aws=6f56019d471bf630c9432bfca43855fa
karpenter.k8s.aws/instance-category=t
karpenter.k8s.aws/instance-cpu=4
karpenter.k8s.aws/instance-family=t3
karpenter.k8s.aws/instance-generation=3
karpenter.k8s.aws/instance-hypervisor=nitro
karpenter.k8s.aws/instance-memory=16384
karpenter.k8s.aws/instance-pods=58
karpenter.k8s.aws/instance-size=xlarge
karpenter.sh/capacity-type=on-demand
karpenter.sh/provisioner-name=default
kubernetes.io/arch=amd64
kubernetes.io/os=linux
node.kubernetes.io/instance-type=t3.xlarge
topology.kubernetes.io/region=us-east-1
topology.kubernetes.io/zone=us-east-1b
Annotations: node.alpha.kubernetes.io/ttl: 0
CreationTimestamp: Wed, 31 Aug 2022 11:43:57 -0700
Taints: node.cilium.io/agent-not-ready=true:NoExecute
node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unreachable:NoSchedule
Unschedulable: false
Lease: Failed to get lease: leases.coordination.k8s.io "ip-10-50-119-64.ec2.internal" not found
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
Ready Unknown Wed, 31 Aug 2022 11:43:57 -0700 Wed, 31 Aug 2022 11:45:01 -0700 NodeStatusNeverUpdated Kubelet never posted node status.
MemoryPressure Unknown Wed, 31 Aug 2022 11:43:57 -0700 Wed, 31 Aug 2022 11:45:01 -0700 NodeStatusNeverUpdated Kubelet never posted node status.
DiskPressure Unknown Wed, 31 Aug 2022 11:43:57 -0700 Wed, 31 Aug 2022 11:45:01 -0700 NodeStatusNeverUpdated Kubelet never posted node status.
PIDPressure Unknown Wed, 31 Aug 2022 11:43:57 -0700 Wed, 31 Aug 2022 11:45:01 -0700 NodeStatusNeverUpdated Kubelet never posted node status.
Addresses:
System Info:
Machine ID:
System UUID:
Boot ID:
Kernel Version:
OS Image:
Operating System:
Architecture:
Container Runtime Version:
Kubelet Version:
Kube-Proxy Version:
ProviderID: aws:///us-east-1b/i-0dc3c403d33754dd6
Non-terminated Pods: (0 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 0 (0%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal RegisteredNode 3m19s node-controller Node ip-10-50-119-64.ec2.internal event: Registered Node ip-10-50-119-64.ec2.internal in Controller
Resource Specs and Logs
2022-08-31T18:43:54.826Z INFO controller.provisioning Computed 3 new node(s) will fit 7 pod(s) {"commit": "b157d45"}
2022-08-31T18:43:54.826Z INFO controller.provisioning Launching node with 3 pods requesting {"cpu":"3285m","memory":"532Mi","pods":"8"} from types t3.xlarge {"commit": "b157d45", "provisioner": "default"}
2022-08-31T18:43:54.827Z INFO controller.provisioning Launching node with 3 pods requesting {"cpu":"3285m","memory":"532Mi","pods":"8"} from types t3.xlarge {"commit": "b157d45", "provisioner": "default"}
2022-08-31T18:43:54.827Z INFO controller.provisioning Launching node with 1 pods requesting {"cpu":"1285m","memory":"532Mi","pods":"6"} from types t3.xlarge {"commit": "b157d45", "provisioner": "default"}
2022-08-31T18:43:55.417Z DEBUG controller.provisioning.cloudprovider Created launch template, Karpenter-eks-rabdalla-dev-10366144926913742821 {"commit": "b157d45", "provisioner": "default"}
2022-08-31T18:43:57.231Z INFO controller.provisioning.cloudprovider Launched instance: i-086ba86d09b25e186, hostname: ip-10-50-110-116.ec2.internal, type: t3.xlarge, zone: us-east-1b, capacityType: on-demand {"commit": "b157d45", "provisioner": "default"}
2022-08-31T18:43:57.260Z INFO controller.provisioning.cloudprovider Launched instance: i-0dc3c403d33754dd6, hostname: ip-10-50-119-64.ec2.internal, type: t3.xlarge, zone: us-east-1b, capacityType: on-demand {"commit": "b157d45", "provisioner": "default"}
2022-08-31T18:43:57.282Z INFO controller.provisioning.cloudprovider Launched instance: i-066cf340fe575b474, hostname: ip-10-50-72-199.ec2.internal, type: t3.xlarge, zone: us-east-1b, capacityType: on-demand {"commit": "b157d45", "provisioner": "default"}
2022-08-31T18:45:56.988Zernal DEBUG controller.aws.launchtemplate Deleted launch template Karpenter-eks-rabdalla-dev-10366144926913742821 (lt-05ced4c5f64da911a) {"commit": "b157d45"}
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 4
- Comments: 44 (20 by maintainers)
We are also running cilium and eks. We didn’t had any issues when running the amazon-ec2 images for our nodes also ssm is working.
I tried to reproduce your problem and its repeatable for me.
SSM works with Bottlerocket and Cilium
It seems there is is a configuration issue in your setup. 😃
What we do for our nodes is the following in terraform:
Setting up the eks-irsa-role like this:
Important is here the
karpenter_controller_node_iam_role_arns
there we are referencing the default node role from the eks managed node groups.In the default managed node group we are passing the required ssm policy to ensure that ssm is working.
Check that your nodes have the policy for
AmazonSSMManagedInstanceCore
than it should work out of the box. Since already mentioned ssm is preinstalled.Cilium and Bottlerocket Troubleshoot
First of all I can see the same behaviour
I could connect to the instance with ssm without any problems.
Interessting Logs from the Kubelet
The main problem is that the
node-init
Pod from Cilium can’t start since it doesn’t have a normal bash on Bottlerocket.After some googling I found a possible solution for it.
You need only to disable the node-init pod with
nodeinit.enabled=false
on bottlerocket and everything should work as excepted.https://github.com/bottlerocket-os/bottlerocket/issues/1405#issuecomment-804196007
I don’t have time to test it fully in cluster with the networking change.
@triceras Would be nice to hear your feedback if it works for you.
I hope this helps some people. :neckbeard:
Bottlerocket and amazon Linux AMIs come with SSM preinstalled.