aws-ebs-csi-driver: csi-provisioner is unable to create volume via ebs-plugin with no helpful error

/kind bug

What happened?

I installed a fresh Kubernetes 1.13.3 cluster using kubeadm on AWS. I created the CSINodeInfo CRD via https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/storage-crds/csinodeinfo.yaml

And then applied controller.yaml + node.yaml.

After adjusting the volumes for the controller (see: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/pull/212) things started up nicely and I was able to query the CSINodeInfo objects from my nodes just fine.

I created the most basic storage class I could think of:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: ebs
provisioner: ebs.csi.aws.com
parameters:
  type: gp2
reclaimPolicy: Retain
volumeBindingMode: Immediate

and then applied a corresponding PVC:

apiVersion: v1                                                                                                                                                                                                                                                                                                                 
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs 
  resources:
    requests:
      storage: 10Gi

The ebs-csi-controller, or rather the csi-provisioner picked up the request to CreateVolume, but then things fail miserably and without (at least to me) a helpful error message - even when running ebs-plugin with v=9:

+ ebs-csi-controller-0 › cluster-driver-registrar
+ ebs-csi-controller-0 › csi-attacher
+ ebs-csi-controller-0 › csi-provisioner
+ ebs-csi-controller-0 › ebs-plugin
ebs-csi-controller-0 csi-provisioner I0214 17:43:39.263549       1 csi-provisioner.go:82] Version: v1.0.1-0-g9a258a0a
ebs-csi-controller-0 csi-provisioner I0214 17:43:39.263600       1 csi-provisioner.go:96] Building kube configs for running in cluster...
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:37.821382       1 main.go:77] Version: v1.0.1-0-g0660c6c6
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:37.821512       1 main.go:80] Attempting to open a gRPC connection with: "/csi/csi.sock"
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:37.821535       1 connection.go:69] Connecting to /csi/csi.sock
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:37.821813       1 connection.go:96] Still trying, connection is CONNECTING
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:37.822145       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:38.822216       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
ebs-csi-controller-0 csi-attacher I0214 17:43:40.721128       1 main.go:76] Version: v1.0.1-0-gb7dadac
ebs-csi-controller-0 csi-attacher I0214 17:43:40.722553       1 connection.go:89] Connecting to /var/lib/csi/sockets/pluginproxy/csi.sock
ebs-csi-controller-0 ebs-csi-controller-0 csi-attacherebs-csi-controller-0 cluster-driver-registrar I0214 17:43:39.864132       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0214 17:43:40.722806       1 connection.go:116] Still trying, connection is CONNECTING
 ebs-pluginebs-csi-controller-0 csi-attacher  I0214 17:43:36.330221       1 driver.go:49] Driver: ebs.csi.aws.com Version: 0.3.0-alpha
I0214 17:43:40.722991       1 connection.go:116] Still trying, connection is TRANSIENT_FAILURE
ebs-csi-controller-0ebs-csi-controller-0  csi-attacher cluster-driver-registrarI0214 17:43:41.723107       1 connection.go:116] Still trying, connection is CONNECTING
 I0214 17:43:41.040443       1 connection.go:96] Still trying, connection is CONNECTING
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:41.040493       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:42.106398       1 connection.go:96] Still trying, connection is CONNECTING
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:42.106437       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
ebs-csi-controller-0 ebs-plugin I0214 17:43:42.429605       1 mount_linux.go:174] Cannot run systemd-run, assuming non-systemd OS
ebs-csi-controller-0 ebs-csi-controller-0ebs-plugin I0214 17:43:42.429619       1 mount_linux.go:175] systemd-run failed with: exit status 1
 ebs-csi-controller-0 ebs-plugin I0214 17:43:42.429628       1 mount_linux.go:176] systemd-run output: Failed to create bus connection: No such file or directory
csi-provisionerebs-csi-controller-0  ebs-plugin I0214 17:43:42.375961       1 controller.go:197] Connecting to /var/lib/csi/sockets/pluginproxy/csi.sock
I0214 17:43:42.429859       1 driver.go:93] Listening for connections on address: &net.UnixAddr{Name:"/var/lib/csi/sockets/pluginproxy/csi.sock", Net:"unix"}
ebs-csi-controller-0 csi-provisioner I0214 17:43:42.376205       1 controller.go:224] Still trying, connection is CONNECTING
ebs-csi-controller-0 csi-provisioner I0214 17:43:42.378224       1 controller.go:224] Still trying, connection is TRANSIENT_FAILURE
ebs-csi-controller-0 ebs-csi-controller-0csi-attacher  I0214 17:43:42.765155       1 connection.go:113] Connected
ebs-csi-controller-0ebs-plugin  csi-attacher I0214 17:43:42.765176       1 connection.go:242] GRPC call: /csi.v1.Identity/Probe
I0214 17:43:42.771770       1 identity.go:61] Probe: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
ebs-csi-controller-0 ebs-csi-controller-0csi-attacher I0214 17:43:42.765181       1 connection.go:243] GRPC request: {}
 ebs-csi-controller-0 csi-attacher I0214 17:43:42.772187       1 connection.go:245] GRPC response: {}
ebs-pluginebs-csi-controller-0  csi-attacher I0214 17:43:42.772849       1 connection.go:246] GRPC error: <nil>
I0214 17:43:42.773643       1 identity.go:27] GetPluginInfo: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
ebs-csi-controller-0ebs-csi-controller-0  csi-attacher I0214 17:43:42.772857       1 main.go:211] Probe succeeded
ebs-csi-controller-0 ebs-plugincsi-attacher  I0214 17:43:42.772880       1 connection.go:242] GRPC call: /csi.v1.Identity/GetPluginInfo
I0214 17:43:42.775224       1 identity.go:37] GetPluginCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
ebs-csi-controller-0 csi-attacher I0214 17:43:42.772887       1 connection.go:243] GRPC request: {}
ebs-csi-controller-0 csi-attacher I0214 17:43:42.773941       1 connection.go:245] GRPC response: {"name":"ebs.csi.aws.com","vendor_version":"0.3.0-alpha"}
ebs-csi-controller-0 csi-attacher I0214 17:43:42.774549       1 connection.go:246] GRPC error: <nil>
ebs-csi-controller-0 csi-attacher I0214 17:43:42.774558       1 main.go:128] CSI driver name: "ebs.csi.aws.com"
ebs-csi-controller-0 csi-attacher I0214 17:43:42.774566       1 connection.go:242] GRPC call: /csi.v1.Identity/GetPluginCapabilities
ebs-csi-controller-0 csi-attacher I0214 17:43:42.774571       1 connection.go:243] GRPC request: {}
ebs-csi-controller-0 csi-attacher I0214 17:43:42.775700       1 connection.go:245] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
ebs-csi-controller-0 csi-attacher I0214 17:43:42.783221       1 connection.go:246] GRPC error: <nil>
ebs-csi-controller-0 csi-attacher I0214 17:43:42.783230       1 connection.go:242] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
ebs-csi-controller-0 csi-attacher I0214 17:43:42.783236       1 connection.go:243] GRPC request: {}
ebs-csi-controller-0ebs-csi-controller-0  csi-attacherebs-plugin I0214 17:43:42.784204       1 controller.go:220] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
 I0214 17:43:42.785050       1 connection.go:245] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}}]}
ebs-csi-controller-0 csi-attacher I0214 17:43:42.788303       1 connection.go:246] GRPC error: <nil>
ebs-csi-controller-0 csi-attacher I0214 17:43:42.789537       1 main.go:152] CSI driver supports ControllerPublishUnpublish, using real CSI handler
ebs-csi-controller-0 csi-attacher I0214 17:43:42.790980       1 controller.go:111] Starting CSI attacher
ebs-csi-controller-0 ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:43.081709       1 connection.go:96] Still trying, connection is CONNECTING
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:43.081811       1 connection.go:93] Connected
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:43.081820       1 main.go:88] Calling CSI driver to discover driver name.
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:43.081829       1 connection.go:137] GRPC call: /csi.v1.Identity/GetPluginInfo
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:43.081834       1 connection.go:138] GRPC request: {}
ebs-csi-controller-0ebs-plugin cluster-driver-registrar I0214 17:43:43.083106       1 identity.go:27] GetPluginInfo: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
 I0214 17:43:43.083821       1 connection.go:140] GRPC response: {"name":"ebs.csi.aws.com","vendor_version":"0.3.0-alpha"}
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:43.084433       1 connection.go:141] GRPC error: <nil>
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:43.084441       1 main.go:96] CSI driver name: "ebs.csi.aws.com"
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:43.084447       1 main.go:100] Loading kubeconfig.
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:43.084737       1 k8s_register.go:54] AttachRequired: true
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:43.084765       1 k8s_register.go:55] PodInfoOnMountVersion: 
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:43.084771       1 k8s_register.go:58] Registering csidrivers
ebs-csi-controller-0 cluster-driver-registrar I0214 17:43:43.104433       1 k8s_register.go:69] CSIDriver CRD already had been registered
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.376479       1 controller.go:224] Still trying, connection is CONNECTING
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.376585       1 controller.go:221] Connected
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.380089       1 controller.go:571] Starting provisioner controller ebs.csi.aws.com_ebs-csi-controller-0_12606c0e-3080-11e9-b3df-b28d31712265!
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.483402       1 controller.go:620] Started provisioner controller ebs.csi.aws.com_ebs-csi-controller-0_12606c0e-3080-11e9-b3df-b28d31712265!
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.483520       1 controller.go:926] provision "default/ebs-claim" class "ebs": started
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.487847       1 controller.go:188] GRPC call: /csi.v1.Identity/GetPluginCapabilities
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.488450       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-claim", UID:"e6d74eec-307b-11e9-a224-0274920412e2", APIVersion:"v1", ResourceVersion:"207179", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/ebs-claim"
ebs-csi-controller-0 ebs-plugin I0214 17:43:43.489633       1 identity.go:37] GetPluginCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.487861       1 controller.go:189] GRPC request: {}
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.492706       1 controller.go:191] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.498658       1 controller.go:192] GRPC error: <nil>
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.498671       1 controller.go:188] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.498676       1 controller.go:189] GRPC request: {}
ebs-csi-controller-0 ebs-plugin I0214 17:43:43.502479       1 controller.go:220] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.503051       1 controller.go:191] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}}]}
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.506029       1 controller.go:192] GRPC error: <nil>
ebs-csi-controller-0 ebs-csi-controller-0 csi-provisioner I0214 17:43:43.506043       1 controller.go:188] GRPC call: /csi.v1.Identity/GetPluginInfo
ebs-plugin I0214 17:43:43.507153       1 identity.go:27] GetPluginInfo: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.506047       1 controller.go:189] GRPC request: {}
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.507428       1 controller.go:191] GRPC response: {"name":"ebs.csi.aws.com","vendor_version":"0.3.0-alpha"}
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.508821       1 controller.go:192] GRPC error: <nil>
ebs-csi-controller-0 ebs-plugin I0214 17:43:43.512210       1 controller.go:49] CreateVolume: called with args {Name:pvc-e6d74eec-307b-11e9-a224-0274920412e2 CapacityRange:required_bytes:10737418240  VolumeCapabilities:[mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[type:gp2] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.508853       1 controller.go:544] CreateVolumeRequest {Name:pvc-e6d74eec-307b-11e9-a224-0274920412e2 CapacityRange:required_bytes:10737418240  VolumeCapabilities:[mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[type:gp2] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.509013       1 controller.go:188] GRPC call: /csi.v1.Controller/CreateVolume
ebs-csi-controller-0 csi-provisioner I0214 17:43:43.509022       1 controller.go:189] GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-e6d74eec-307b-11e9-a224-0274920412e2","parameters":{"type":"gp2"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]}
ebs-csi-controller-0 ebs-pluginebs-csi-controller-0  E0214 17:43:53.509489       1 driver.go:80] GRPC error: rpc error: code = Internal desc = RequestCanceled: request context canceled
csi-provisioner I0214 17:43:53.509243       1 controller.go:191] GRPC response: {}
ebs-csi-controller-0 csi-provisioner I0214 17:43:53.509800       1 controller.go:192] GRPC error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
ebs-csi-controller-0 ebs-csi-controller-0ebs-plugin caused by: context canceled
 csi-provisioner W0214 17:43:53.509816       1 controller.go:592] CreateVolume timeout: 10s has expired, operation will be retried
ebs-csi-controller-0ebs-csi-controller-0 csi-provisioner  I0214 17:43:58.509996       1 controller.go:188] GRPC call: /csi.v1.Controller/CreateVolume
ebs-plugin ebs-csi-controller-0I0214 17:43:58.513424       1 controller.go:49] CreateVolume: called with args {Name:pvc-e6d74eec-307b-11e9-a224-0274920412e2 CapacityRange:required_bytes:10737418240  VolumeCapabilities:[mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[type:gp2] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
 csi-provisioner I0214 17:43:58.510027       1 controller.go:189] GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-e6d74eec-307b-11e9-a224-0274920412e2","parameters":{"type":"gp2"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]}
ebs-csi-controller-0 ebs-plugin E0214 17:44:08.510374       1 driver.go:80] GRPC error: rpc error: code = Internal desc = RequestCanceled: request context canceled
ebs-csi-controller-0 csi-provisioner I0214 17:44:08.510168       1 controller.go:191] GRPC response: {}
ebs-csi-controller-0ebs-csi-controller-0  ebs-plugincsi-provisioner  caused by: context canceled
I0214 17:44:08.510648       1 controller.go:192] GRPC error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
ebs-csi-controller-0 csi-provisioner W0214 17:44:08.510662       1 controller.go:592] CreateVolume timeout: 10s has expired, operation will be retried
ebs-csi-controller-0ebs-csi-controller-0  ebs-plugincsi-provisioner I0214 17:44:14.510855       1 controller.go:188] GRPC call: /csi.v1.Controller/CreateVolume
 I0214 17:44:14.513327       1 controller.go:49] CreateVolume: called with args {Name:pvc-e6d74eec-307b-11e9-a224-0274920412e2 CapacityRange:required_bytes:10737418240  VolumeCapabilities:[mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[type:gp2] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
ebs-csi-controller-0 csi-provisioner I0214 17:44:14.510911       1 controller.go:189] GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-e6d74eec-307b-11e9-a224-0274920412e2","parameters":{"type":"gp2"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]}
ebs-csi-controller-0 ebs-plugin E0214 17:44:24.511174       1 driver.go:80] GRPC error: rpc error: code = Internal desc = RequestCanceled: request context canceled
ebs-csi-controller-0ebs-csi-controller-0  csi-provisionerebs-plugin  caused by: context deadline exceeded
I0214 17:44:24.511042       1 controller.go:191] GRPC response: {}
ebs-csi-controller-0 csi-provisioner I0214 17:44:24.511578       1 controller.go:192] GRPC error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
ebs-csi-controller-0 csi-provisioner W0214 17:44:24.511592       1 controller.go:592] CreateVolume timeout: 10s has expired, operation will be retried
ebs-csi-controller-0 ebs-pluginebs-csi-controller-0  I0214 17:44:31.714097       1 controller.go:49] CreateVolume: called with args {Name:pvc-e6d74eec-307b-11e9-a224-0274920412e2 CapacityRange:required_bytes:10737418240  VolumeCapabilities:[mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[type:gp2] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
csi-provisioner I0214 17:44:31.711803       1 controller.go:188] GRPC call: /csi.v1.Controller/CreateVolume
ebs-csi-controller-0 csi-provisioner I0214 17:44:31.711829       1 controller.go:189] GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-e6d74eec-307b-11e9-a224-0274920412e2","parameters":{"type":"gp2"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]}
ebs-csi-controller-0ebs-csi-controller-0  csi-provisioner I0214 17:44:41.711943       1 controller.go:191] GRPC response: {}
ebs-pluginebs-csi-controller-0 csi-provisioner  I0214 17:44:41.715233       1 controller.go:192] GRPC error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0214 17:44:41.712073       1 driver.go:80] GRPC error: rpc error: code = Internal desc = RequestCanceled: request context canceled
ebs-csi-controller-0 ebs-csi-controller-0csi-provisioner  W0214 17:44:41.715293       1 controller.go:592] CreateVolume timeout: 10s has expired, operation will be retried
ebs-plugin caused by: context deadline exceeded

What you expected to happen?

I expected either csi-provisioner to create a volume or ebs-plugin to return a helpful error message about i.e. the failing API call.

Environment

  • Kubernetes version (use kubectl version): 1.13.3
  • Driver version: latest (0.3.0alpha)

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 22 (7 by maintainers)

Commits related to this issue

Most upvoted comments

we’ve got the same issue running k8s 1.23

failed to provision volume with StorageClass "gp2": rpc error: code = Internal desc = Could not create volume "pvc-c7610d24-aa0c-42ca-bf23-f07b0b06139f": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded

+12
namgk on Sep 29, 2022

I was following this https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html, and I made two mistakes, which caused the issue -

  1. Creating the Amazon EBS CSI driver IAM role for service accounts - (https://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html), check role has been created properly. Check “Trust Relationships” of the role, click edit in AWS console and check for any validation errors. For, me, there were validation error.
  2. Creating an IAM OIDC provider for your cluster - Again, this has been created properly. It has at least one thumbprint.
+8
Kingshuk-Kubera on Mar 24, 2023

@leakingtapan Luckly or sadly we were able to identify the root cause of this issue to be fully unrelated to the aws-ebs-csi-driver. The ebs-plugin was unable to resolve the IP of the AWS API via DNS. So the plugin initialized just fine (using the metadata service), but then was unable to send out any requests to the AWS API. If I may kindly ask to add more logging of the ebs-plugin when doing API requests? A failed DNS request should be distinguishable from timeout reaching the AWS API.

The only thing left to say is: Thank you.

+4
frittentheke on Feb 15, 2019

Same problem as @shishirkh here. Anybody to help ?

+3
throrin19 on Jun 8, 2022

I think I am facing the same issue - cannot be sure as I am not seeing any DNS error in the logs.

@frittentheke How did you solve it when you faced the DNS issue?

+3
shishirkh on Nov 28, 2021

I was following this https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html, and I made two mistakes, which caused the issue -

  1. Creating the Amazon EBS CSI driver IAM role for service accounts - (https://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html), check role has been created properly. Check “Trust Relationships” of the role, click edit in AWS console and check for any validation errors. For, me, there were validation error.
  2. Creating an IAM OIDC provider for your cluster - Again, this has been created properly. It has at least one thumbprint.

Man, have I said that I love you??? 🤣🤣🤣🤣🤣 You solved my problem.

Yeah, I made a few mistakes too. Let me show you how I fixed it. Mine was a bit similar to yours.

  1. I was following this tutorial: https://repost.aws/knowledge-center/eks-persistent-storage However, I noticed that my cluster didn’t have OIDC provider. So I was following the next

  2. Then, I was following this one https://repost.aws/knowledge-center/eks-troubleshoot-oidc-and-irsa to add OIDC provider to my cluster

  3. Lastly, I follow this one https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html

After all, everything was working nicely. Thanks again, brother. 🤜🏼🤛🏼

+1
ricardorqr on Mar 25, 2023

I followed below steps and it worked - https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html

+1
Kingshuk-Kubera on Oct 27, 2022
Read more comments on GitHub
← aws-ebs-csi-driver: Cannot Run with IAM Service Account and no metadata service
aws-ebs-csi-driver: csi-resizer getting OOMKilled →