kubernetes: Cinder volume attach does not work
Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.2", GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean", BuildDate:"2017-04-19T20:33:11Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Unable to connect to the server: Bad Gateway
Environment:
- Cloud provider or hardware configuration: Openstack (unknown version)
- OS (e.g. from /etc/os-release): Ubuntu 16.04
- Kernel (e.g.
uname -a): 4.4.0-28-generic - Install tools: kubeadm
What happened:
Cinder volume does not mount
What you expected to happen:
Cinder volume does mount
How to reproduce it (as minimally and precisely as possible):
I’m trying to fire up sample mysql pod with Cinder PD (https://github.com/kubernetes/kubernetes/tree/master/examples/mysql-cinder-pd).
I have created the volume:
$ cinder show fa4d4dee-9f44-4191-ab88-8ef8d1d6b2dc
+------------------------------+--------------------------------------+
| Property | Value |
+------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2017-05-02T08:12:32.000000 |
| description | None |
| encrypted | False |
| id | fa4d4dee-9f44-4191-ab88-8ef8d1d6b2dc |
| metadata | {} |
| multiattach | False |
| name | k8s-test-pd |
| os-vol-tenant-attr:tenant_id | 4a7d3b5645254117af1d36c4eb5201a5 |
| replication_status | disabled |
| size | 2 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| updated_at | 2017-05-02T08:12:33.000000 |
| user_id | 11c0fd1d2a3a475aaeb31f1f637c2f91 |
| volume_type | None |
+------------------------------+--------------------------------------+
and modified the volumeID in sample yaml.
Pod does not come up and describe pod operation returns
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
13m 13m 1 {default-scheduler } Normal Scheduled Successfully assigned mysql to mm-kube-slave-1
11m 16s 6 {kubelet mm-kube-slave-1} Warning FailedMount Unable to mount volumes for pod "mysql_default(9ca3450e-2f32-11e7-b455-fa163ed6d9bd)": timeout expired waiting for volumes to attach/mount for pod "default"/"mysql". list of unattached/unmounted volumes=[mysql-persistent-storage]
11m 16s 6 {kubelet mm-kube-slave-1} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"mysql". list of unattached/unmounted volumes=[mysql-persistent-storage]
And syslog on my only minion (where the pod is scheduled) says:
May 2 12:37:29 mm-kube-slave-1 kubelet[1979]: I0502 12:37:29.573805 1979 reconciler.go:231] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/cinder/fa4d4dee-9f44-4191-ab88-8ef8d1d6b2dc" (spec.Name: "mysql-persistent-storage") pod "9ca3450e-2f32-11e7-b455-fa163ed6d9bd" (UID: "9ca3450e-2f32-11e7-b455-fa163ed6d9bd")
May 2 12:37:29 mm-kube-slave-1 kubelet[1979]: E0502 12:37:29.577353 1979 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/cinder/fa4d4dee-9f44-4191-ab88-8ef8d1d6b2dc\"" failed. No retries permitted until 2017-05-02 12:39:29.577306663 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/cinder/fa4d4dee-9f44-4191-ab88-8ef8d1d6b2dc" (spec.Name: "mysql-persistent-storage") pod "9ca3450e-2f32-11e7-b455-fa163ed6d9bd" (UID: "9ca3450e-2f32-11e7-b455-fa163ed6d9bd") is not yet attached according to node status
May 2 12:37:42 mm-kube-slave-1 kubelet[1979]: E0502 12:37:42.602080 1979 kubelet.go:1549] Unable to mount volumes for pod "mysql_default(9ca3450e-2f32-11e7-b455-fa163ed6d9bd)": timeout expired waiting for volumes to attach/mount for pod "default"/"mysql". list of unattached/unmounted volumes=[mysql-persistent-storage]; skipping pod
May 2 12:37:42 mm-kube-slave-1 kubelet[1979]: E0502 12:37:42.602697 1979 pod_workers.go:182] Error syncing pod 9ca3450e-2f32-11e7-b455-fa163ed6d9bd ("mysql_default(9ca3450e-2f32-11e7-b455-fa163ed6d9bd)"), skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"mysql". list of unattached/unmounted volumes=[mysql-persistent-storage]
Any hint what is wrong? I have set up k8s 1.6.2 using kubeadm with Openstack provider (correct openstack configuration present and --cloud-provider parameters passed to the kubelet. Ubuntu Xenial with Docker 17.04.0-ce.
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 19 (11 by maintainers)
I have the exact same Issue as mentioned by @wu105 But even if I create manually the
special deviceit does not workWe run into a similar situation: when a minion node joined the cluster, it somehow missed the cloud-provider specification for openstack. This should be considered user error. However, it might be helpful for the event to include a hint at the error, or the master could have complained when a minion like this joins. As it stands, ‘kubectl get node -o wide’ would display external IP for all nodes except the ones missed the cloud provider specification.
In any case, this lead to the following:
When deploying an application, which creates dynamic persistent volumes, the volumes can attach to the node, but the pod would register the following events:
The above are the first few events, then it keeps repeating the last two, until we delete the pod, which may get recreated on a different node, and the volume re-attaches and all is ok.
If the pod gets recreated on the original node, the error will repeat, until the spacial device is manually created with a command such as ‘mkdir -p /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/672d026d-28fa-4851-825e-f40d35903b03’, but the pod would be using local storage under that directory instead of the pv.
The situation is fixed by drain and delete the node in question, and rejoin.
A side note: initially the volumes have wrong labels about zones thus cannot attach to any nodes, and the pods cannot be scheduled. We have a cron job to fix the labels on the volumes. the volumes then got attached to some node. It would be nice to be able to specify the label/zone when creating a pv,