aws-ebs-csi-driver: Node can't access volumeattachments resource

/kind bug

What happened? When a pod with a persistent volume is deleted, the new pod fails to attach / mount the storage with the following error:

MountVolume.WaitForAttach failed for volume "<pvc_name>" : volume <volume_name> has GET error
 for volume attachment csi-4b2c0c56...: volumeattachments.storage.k8s.io "csi-4b2c0c56..." is forbidden: User "system:node:<node_name>" cannot get resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope: no relationship found between node "<node_name>" and this object

What you expected to happen? Volume should move to new pod and successfully mount.

How to reproduce it (as minimally and precisely as possible)?

  1. Create a deployment with a pod that mounts a PVC provisioned by AWS EBS CSI Driver
  2. Delete the pod
  3. Describe the new pod and see the message specified above. It is usually the next message after “Multi-attach failure”, which is an expected message while the original pod is being deleted.

Anything else we need to know?:

  • This error has been intermittent, and seen with both new volumes and “migrated” ones.
  • This error occurs in a cluster with the aws cloud provider running out-of-tree (which doesn’t include volume provisioning logic)

Environment

  • Kubernetes version (use kubectl version): 1.15.1
  • Driver version: commit 2aed4b5

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 3
  • Comments: 25 (7 by maintainers)

Most upvoted comments

Is there a workaround for this? Deleting the affected pod didn’t help. The volumeattachment object doesn’t exist in my case.

Is there any updates? we also see same problem

Was there any solution this ever? We ran into this just today

This does appear to fix the issue for us (so far). We upgraded to chart version 2.12.1 and app version 1.12.1.

We didn’t see the issue since, so I’d say yes.