kubernetes: Slow binding of PV => PVC

We’re manually creating drives to contain each of our databases, as such it makes sense for us to have known bindings between each PVC and PV. Since volumeSelector is not yet available creating them one at a time has been working for us.

Recently for unknown reasons it has gotten very slow. What used to take seconds to bind the PVC=>PV now takes to 20+ minutes.

Initially it shows up as Pending with errors such as the following in the logs: PVClaimBinder could not update claim sliding-data-3c-pvc: Error updating volume: persistentvolumeclaims "sl iding-data-3c-pvc" cannot be updated: the object has been modified; please apply your changes to the latest version and try again

After a while it finally binds, but with Capacity: 0:

# kubectl describe pv sliding-data-1c-pv
Name:        sliding-data-1c-pv
Labels:        <none>
Status:        Available
Claim:        
Reclaim Policy:    Retain
Access Modes:    RWO
Capacity:    16Gi
Message:    
Source:
    Type:    GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
    PDName:    sliding-data-1c
    FSType:    ext4
    Partition:    0
    ReadOnly:    false

# kubectl describe pvc sliding-data-1c-pvc
Name:        sliding-data-1c-pvc
Namespace:    default
Status:        Bound
Volume:        sliding-data-1c-pv
Labels:        <none>
Capacity:    0
Access Modes:

Finally after around half an hour everything settles down as expected. Since we can only create one at a time, this means many hours while trying to create a group of PV/PVC pairs.

We’re running our own Kubernetes cluster on Google Compute Engine, here are some other details:

# kubectl version
Client Version: version.Info{Major:"1", Minor:"1", GitVersion:"v1.1.3", GitCommit:"6a81b50c7e97bbe0ade075de55ab4fa34f049dc2", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"1", GitVersion:"v1.1.3", GitCommit:"6a81b50c7e97bbe0ade075de55ab4fa34f049dc2", GitTreeState:"clean"}

# cat sliding-data-3c-pv.json
{
  "apiVersion": "v1",
  "kind": "PersistentVolume",
  "metadata": {
    "name": "sliding-data-3c-pv"
  },
  "spec": {
    "accessModes": [
      "ReadWriteOnce"
    ],
    "capacity": {
      "storage": "16Gi"
    },
    "gcePersistentDisk": {
      "fsType": "ext4",
      "pdName": "sliding-data-3c"
    },
    "persistentVolumeReclaimPolicy": "Retain"
  }
}

# cat sliding-data-3c-pvc.json 
{
  "apiVersion": "v1",
  "kind": "PersistentVolumeClaim",
  "metadata": {
    "name": "sliding-data-3c-pvc"
  },
  "spec": {
    "accessModes": [
      "ReadWriteOnce"
    ],
    "resources": {
      "requests": {
        "storage": "16Gi"
      }
    }
  }
}

# docker --version
Docker version 1.6.2, build 7c8fca2

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 23 (15 by maintainers)

Most upvoted comments

Expanding on @jsafrane’s suggestion. Here is an example showing how to pre-bind volumes:

Create a PV object with a ClaimRef field referencing a PVC that you will subsequently create:

$ kubectl create -f pv.yaml 
persistentvolume "pv0003" created

where pv.yaml contains:

  apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: pv0003
  spec:
    capacity:
      storage: 5Gi
    accessModes:
      - ReadWriteOnce
    persistentVolumeReclaimPolicy: Recycle
    claimRef:
      namespace: default
      name: myclaim
    nfs:
      path: /tmp
      server: 172.17.0.2

Then create the PVC with the same name:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: myclaim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

The PV and PVC should be bound immediately:

$ kubectl get pvc
NAME      STATUS    VOLUME    CAPACITY   ACCESSMODES   AGE
myclaim   Bound     pv0003    5Gi        RWO           4s
$ ./cluster/kubectl.sh get pv
NAME      CAPACITY   ACCESSMODES   STATUS    CLAIM             REASON    AGE
pv0003    5Gi        RWO           Bound     default/myclaim             57s

saad-ali on Apr 18, 2016