kubernetes: Problem rescheduling POD with GCE PD disk attached

+1

gcloud compute instances detach-disk NODE --disk DISK

It is indeed my friend https://github.com/kubernetes/kubernetes/issues/25457 Hang tight =)

We’re working on a fix. It is targeted for the next minor release v1.3.0.

@ScyDev That’s https://github.com/kubernetes/kubernetes/issues/19953 Should be fixed at the same time as this.

I think i ran into the same issue trying two deploy a pod with a GCE PD and a replication controller set to replicas=2.

Regarding the documentation that should be possible:

Using a PD on a pod controlled by a ReplicationController will fail unless the PD is read-only or the replica count is 0 or 1.

So i setup a new cluster with one minion and Kubernetes 1.2.2. Then i created a PD, PersistentVolumes and my ReplicationController:

---

apiVersion: v1 kind: PersistentVolume metadata: name: volume-nginx-data-disk spec: capacity: storage: 1Gi accessModes: - ReadOnlyMany gcePersistentDisk: pdName: nginx-data-disk fsType: ext4

---

kind: PersistentVolumeClaim apiVersion: v1 metadata: name: claim-nginx-data-disk spec: accessModes: - ReadOnlyMany resources: requests: storage: 1Gi

---

apiVersion: v1 kind: ReplicationController metadata: name: sslproxy-rc labels: name: sslproxy-rc spec: replicas: 2 selector: name: sslproxy-rc template: metadata: labels: name: sslproxy-rc spec: containers: - name: nginx image: nginx ports: - name: nginx-ssl containerPort: 443 volumeMounts: - name: nginx-data-disk mountPath: /etc/nginx/conf.d volumes: - name: nginx-data-disk persistentVolumeClaim: claimName: claim-nginx-data-disk readOnly: true

---

So everything should be mounted in “readOnly” mode. When i try to deploy my RC, only the first pod is created, the second is always stuck with status “Pending”.

kubectl describe pod xyz:

Unable to mount volumes for pod “sslproxy-rc-zxe52_default(a532eb8f-07f2-11e6-8707-42010af00108)”: Could not attach GCE PD “nginx-data-disk”. Timeout waiting for mount paths to be created.

{kubelet gke-sslproxy-cluster1-default-pool-4091b8f3-wys3} Warning FailedSync Error syncing pod, skipping: Could not attach GCE PD “nginx-data-disk”. Timeout waiting for mount paths to be created.

kubelet.log:

GCE operation failed: googleapi: Error 400: The disk resource ‘nginx-data-disk’ is already being used by ‘gke-sslproxy-cluster1-default-pool-4091b8f3-wys3’ gce_util.go:187] Error attaching PD “nginx-data-disk”: googleapi: Error 400: The disk resource ‘nginx-data-disk’ is already being used by ‘gke-sslproxy-cluster1-default-pool-4091b8f3-wys3’

I think that is because of (https://github.com/kubernetes/kubernetes/issues/21931)

When two or more pods specify the same volume (allowed for some plugins in certain cases), the second pod will fail to start because because the volume attach call for the 2nd pod will continuously fail since the volume is already attached (by the first pod).

Kilian

@saad-ali im my case the node come back up and kubelet did not detach the PD even if the pod was rescheduled in an other node. I had to detach it manually.

While we work on a permanent fix, for folks running into this issue, here are a couple workarounds:

1. Do nothing, wait for the downed node to come back up.
   • Once the original node is rebooted and kubelet comes up on it, it will realize that the pod/volume are no longer scheduled on it and will detach the PD. Then the other node will then be able to attach it.
   • This assumes that the original, downed node will come back up…
2. Manually detach the disk.
   • Manually detach the disk using the gcloud API (CLI or REST or WebUI) from the downed node, and this will allow the other node to attach it.