kubernetes: FailedMount due to MountVolume.WaitForAttach timed out waiting for condition
What happened:
MountVolume ran into timeout while trying to mount a PVC to a pod on Azure.
The pod events show MountVolume.WaitForAttach ... timed out waiting for condition.
After ~6 hours the volume got attached successfully.
What you expected to happen: The volume should be mounted without timeout.
How to reproduce it (as minimally and precisely as possible):
- Delete the pod by setting replicas of custom resource to 0
- Delete the pvc
- Get a new pod created by setting replicas to 1 in the custom resource
- Wait for pod coming up
- Repeat until pod doesn’t come up and stays in ContainerCreating status showing a FailedMount event
Anything else we need to know?:
Environment:
- Kubernetes version (use
kubectl version): v1.16.14 - Cloud provider or hardware configuration: Azure
- OS (e.g:
cat /etc/os-release): SLES 15.1 (suse-chost) - Kernel (e.g.
uname -a): 4.12.14-197.37-default - Install tools:
- Network plugin and version (if this is a network-related bug):
- Others:
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 17 (11 by maintainers)
basically yes. The disk attach does not return success but fails, the second attach attempt from cloud controller is not attempted because the disk is already part of the goal state of the VM. With your fix, the second attempt is executed because now the cloud controller also looks at the VM state. @jakob-kr is from the same team I was working with to analyze the issue.