kubernetes: Unable to mount volumes for pod: timeout expired waiting for volumes to attach or mount
Is this a BUG REPORT or FEATURE REQUEST?:
Uncomment only one, leave it on its own line:
/kind bug
/kind feature
What happened: I successfully create PersistentVolume and PersistentVolumeClaim on AWS, but the pod is not able to mount to those volumes. The error message thrown by the k8s is
Warning FailedMount 58s (x19 over 41m) kubelet, ip-xxx-xxx-xxx-xxx Unable to mount volumes for pod “master-deployment-5979f94fc6-sf9br_default(28fddeda-797a-11e8-907e-0e594759e784)”: timeout expired waiting for volumes to attach or mount for pod “default”/“master-deployment-5979f94fc6-sf9br”. list of unmounted volumes=[mysql-volume]. list of unattached volumes=[mysql-volume elasticsearch-volume default-token-2tfjn]
This is my config file.
kind: PersistentVolume
apiVersion: v1
metadata:
name: master-volume
spec:
storageClassName: local-storage
capacity:
storage: 64Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/k8s-volume"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: master-volume
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 64Gi
storageClassName:
local-storage
---
kind: Deployment
...
volumes:
- name: mysql-volume
persistentVolumeClaim:
claimName: master-volume
- name: elasticsearch-volume
persistentVolumeClaim:
claimName: master-volume
...
What you expected to happen: All pods and the deployment start successfully.
How to reproduce it (as minimally and precisely as possible): Not sure how to reproduce. The images I use are private. I share part of my config file which could be related to the persistent volume issue.
Anything else we need to know?:
- Use kubeadm to create a cluster on multiple instances, and trying to create a deployment with Persistent Volume and Persistent Volume Claim.
- I am using m4.xlarge AWS instance, which should have correct support for
kubernetes.
Environment:
- Kubernetes version (use
kubectl version):
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5", GitCommit:"32ac1c9073b132b8ba18aa830f46b77dcceb0723", GitTreeState:"clean", BuildDate:"2018-06-21T11:46:00Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5", GitCommit:"32ac1c9073b132b8ba18aa830f46b77dcceb0723", GitTreeState:"clean", BuildDate:"2018-06-21T11:34:22Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
- Cloud provider or hardware configuration:
AWS
- OS (e.g. from /etc/os-release):
NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.4 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
- Kernel (e.g.
uname -a):
Linux k8s 4.4.0-1060-aws #69-Ubuntu SMP Sun May 20 13:42:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 12
- Comments: 27 (5 by maintainers)
I am also having this problem and restarting the kubelet does not fix the problem. It occurs highly inconsistently (three prometheus pods are running here but only one cannot seem to mount its storage). Why was this issue closed without being addressed?
I had the same problem with EKS
v1.16.8-eks-e16311. The following steps worked for me, maybe it would work for anyone else:in-usedetach it and wait for it to becomeavailableHave same issue for static pod which has hostpath volume, restart kubelet can solve this problem, but don’t know the reason for now.
Anyone struggling with this issue in EKS especially with statefulsets, EBS and Pending state,
a) Find the volumeid associated with PVC, then execute b) aws ec2 detach-volume --volume-id vol-1234567890abcdef0 --force c) The statefulset will now recover from Pending state
Thanks @serhatcetinkaya, I was able to resolve this issue with a similar set of steps, however when I went to attach the volume, despite being listed as
availablein the console, I received an error that it was already in use. Shortly thereafter the pod succeeded in mounting the volume by itself.I guess something is missing in the glue between EKS and the EC2 API and it gets “stuck” in an inconsistent state.
I get below error. Please help.
Warning FailedMount 95s (x4 over 8m23s) kubelet, ip-xx-xx-x-x.us-east-2.compute.internal Unable to mount volumes for pod “app1_default(xxxxx-xxxx-xxxxxx)”: timeout expired waiting for volumes to attach or mount for pod “default”/“app1”. list of unmounted volumes=[persistent-storage]. list of unattached volumes=[persistent-storage default-token-q44xg] Warning FailedAttachVolume 22s (x4 over 6m29s) attachdetach-controller AttachVolume.Attach failed for volume “efs-pv” : attachment timeout for volume fs-xxx
This is the case with some of my stateful sets (replica=5), three of them are up and running and the fourth pod is stuck on mount &attach errors. Restarting does not help. Each claim is backed by a disk volume. I don’t see multiple pods contending for the same claim.
We had the same issue recently unable to mount with nfs volume mount which was working perfectly from very long time. We had upgraded kubernetes to 1.14 version and there was a node autoscaled which had network isssue. When i removed the node with the network issue, the volume was attached again. So, check for node health and event longs of node, also check if all the daemon pods are working perfectly esp the network related daemon pods.
We were having a similar problem on AWS EKS 1.16. A stateful set’s pod wasn’t able to attach to a volume, and it was trying to come up on a specific node. That node had been recently restarted, and the EBC CSI driver pod on that node had already failed. So, I drained the node and killed it. Another node came up, and my pod started on a different node.
The above works when you don’t want to (or can’t) restart the kubelet for whatever reason
Thank you very much!
Same issue! My kubernetes version is 1.14.3, the volume type is RBD, and it was fixed after I restarted the kubelet.
Hello,
I also receive this error
Same issue here aswell.
I had the same issue with IBM Cloud Kubernetes service. I resolved with a worker node reload:
ibmcloud ks worker reload --cluster <cluster_name> --worker <worker_ID>