aws-ebs-csi-driver: Node can't access volumeattachments resource
/kind bug
What happened? When a pod with a persistent volume is deleted, the new pod fails to attach / mount the storage with the following error:
MountVolume.WaitForAttach failed for volume "<pvc_name>" : volume <volume_name> has GET error
for volume attachment csi-4b2c0c56...: volumeattachments.storage.k8s.io "csi-4b2c0c56..." is forbidden: User "system:node:<node_name>" cannot get resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope: no relationship found between node "<node_name>" and this object
What you expected to happen? Volume should move to new pod and successfully mount.
How to reproduce it (as minimally and precisely as possible)?
- Create a deployment with a pod that mounts a PVC provisioned by AWS EBS CSI Driver
- Delete the pod
- Describe the new pod and see the message specified above. It is usually the next message after “Multi-attach failure”, which is an expected message while the original pod is being deleted.
Anything else we need to know?:
- This error has been intermittent, and seen with both new volumes and “migrated” ones.
- This error occurs in a cluster with the aws cloud provider running out-of-tree (which doesn’t include volume provisioning logic)
Environment
- Kubernetes version (use
kubectl version): 1.15.1 - Driver version: commit 2aed4b5
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 3
- Comments: 25 (7 by maintainers)
Is there a workaround for this? Deleting the affected pod didn’t help. The
volumeattachmentobject doesn’t exist in my case.Is there any updates? we also see same problem
Was there any solution this ever? We ran into this just today
This does appear to fix the issue for us (so far). We upgraded to chart version
2.12.1and app version1.12.1.We didn’t see the issue since, so I’d say yes.