kubernetes: Unable to create a persistent volume with a default storage class

I am having this problem too with nfs-provisioner. What I’ve discovered is that the problem only happens if you first create a PVC and then run the provisioner. All PVCs that are created after the provisioner is deployed will succeed.

Here’s the failing PVC log:

Events:
  FirstSeen	LastSeen	Count	From				SubObjectPath	Type		Reason		Message
  ---------	--------	-----	----				-------------	--------	------		-------
  29m		8s		118	persistentvolume-controller			Normal		FailedBinding	no persistent volumes available for this claim and no storage class is set

When I deleted the PVC and recreated it using the same template:

Events:
  FirstSeen	LastSeen	Count	From									SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----									-------------	--------	------			-------
  13s		13s		2	persistentvolume-controller								Normal		ExternalProvisioning	waiting for a volume to be created, either by external provisioner "local.net/nfs" or manually created by system administrator
  13s		13s		1	local.net/nfs nfs-provisioner-0 07729f15-a8f2-11e7-9b4b-d61a662c2538			Normal		Provisioning		External provisioner is provisioning volume for claim "test/test-postgresql"
  13s		13s		1	local.net/nfs nfs-provisioner-0 07729f15-a8f2-11e7-9b4b-d61a662c2538			Normal		ProvisioningSucceeded	Successfully provisioned volume pvc-11346119-a8f3-11e7-b405-00d5a3ffd69a

In the provisioner logs, it was first failing then it succeeded:

E1004 10:58:21.872600       1 controller.go:569] Error getting claim "test/test-postgresql"'s StorageClass's fields: StorageClass "" not found
E1004 10:58:36.872758       1 controller.go:569] Error getting claim "test/test-postgresql"'s StorageClass's fields: StorageClass "" not found
E1004 10:58:51.872880       1 controller.go:569] Error getting claim "test/test-postgresql"'s StorageClass's fields: StorageClass "" not found
I1004 10:59:17.728782       1 controller.go:1052] scheduleOperation[lock-provision-test/test-postgresql[11346119-a8f3-11e7-b405-00d5a3ffd69a]]
I1004 10:59:17.744725       1 controller.go:1052] scheduleOperation[lock-provision-test/test-postgresql[11346119-a8f3-11e7-b405-00d5a3ffd69a]]
I1004 10:59:17.754203       1 leaderelection.go:154] attempting to acquire leader lease...
I1004 10:59:17.773669       1 leaderelection.go:176] successfully acquired lease to provision for pvc test/test-postgresql
I1004 10:59:17.777182       1 controller.go:1052] scheduleOperation[provision-test/test-postgresql[11346119-a8f3-11e7-b405-00d5a3ffd69a]]
I1004 10:59:17.816305       1 provision.go:411] using service SERVICE_NAME=nfs-provisioner cluster IP 10.110.0.92 as NFS server IP
I1004 10:59:17.824160       1 controller.go:786] volume "pvc-11346119-a8f3-11e7-b405-00d5a3ffd69a" for claim "test/test-postgresql" created
I1004 10:59:17.829239       1 controller.go:803] volume "pvc-11346119-a8f3-11e7-b405-00d5a3ffd69a" for claim "test/test-postgresql" saved
I1004 10:59:17.829255       1 controller.go:839] volume "pvc-11346119-a8f3-11e7-b405-00d5a3ffd69a" provisioned for claim "test/test-postgresql"
I1004 10:59:19.797022       1 leaderelection.go:196] stopped trying to renew lease to provision for pvc test/test-postgresql, task succeeded

By the way, I created a Helm chart to deploy nfs-provisioner, might be useful for testing: https://github.com/IlyaSemenov/nfs-provisioner-chart

@MagicJohnJang

1. It’s not always possible to customize third party Helm charts (for example, see the official Postgres chart which can only use a PVC and not a pre-made PV)
2. Using custom default storage provider does not necessary imply single-server environment. For example, I am using a NFS provider in one of my clusters. The pods will fail if moved across nodes.
3. In any case, it’s barely relevant to the topic, isn’t it?

I do not like filling it in either. I think this can be solved adequately with better communication. What if the admission controller wrote an event on the PVC like “the DefaultStorageClass plugin is on but no default storage class yet exists, no storage class has been set” (I am not sure if it is allowed to write events…but obviously it is allowed to write to the PVC spec.)

I think, for people familiar with kube api “default” probably carries the implication of “default at creation time”. for everybody else it is not so clear that “default” is only a creation-time thing.

I wholly disagree. Kubernetes is a state resolution engine. Define your desired state and eventually you have that state. I should not be required to create a StorageClass primitive with a specific annotation before any other primitive in order to achieve consistent behavior. I should be able to spin up a vanilla cluster, kubectl apply -f . and have a working application. This behavior is inconsistent with the rest of kubernetes and its philosophies and should be changed.

I confirm that it works if the storage class exists prior to creating a PVC. After all, I mentioned the same in my first comment: “What I’ve discovered is that the problem only happens if you first create a PVC and then run the provisioner.”

This is however I believe a typical scenario for newcomers on newly created clusters. You deploy an app (with a Helm chart or manually) which creates a PVC for itself, but it fails to come up. You then read logs and discover that you’re missing a provisioner, you create a default provisioner, but it still does not help and shows cryptic messages 🤷. It’s far from obvious that the default provisioner will not “pick up” previously created PVC which have been set to use the default provisioner, and that you need to recreate all PVCs.

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

Unbound PVCs should be safe to backfill

PVC with storageclass == nil is used to indicate the default, whereas storageclass == “” means no storageclass. But the filling in of the default storageclass is done in the admission controller on PVC create, so if you create/set your default storageclass after creating the PVC, the storageclass field won’t be updated afterwards. I think we could possibly have a long running controller do the same thing as the admission controller, filling in the storageclass name with the default storageclass, however, we need to consider backwards compatibility scenarios.

Unfortunately this dependency between storage classes and PVCs breaks the resource model of Kubernetes where everything converges after a while. I don’t know whether there are any backwards-compatibility concerns but a controller that sets the default storage class in PVCs that do not use it sounds good to me.

/remove-lifecycle stale

PVC with storageclass == nil is used to indicate the default, whereas storageclass == “” means no storageclass. But the filling in of the default storageclass is done in the admission controller on PVC create, so if you create/set your default storageclass after creating the PVC, the storageclass field won’t be updated afterwards. I think we could possibly have a long running controller do the same thing as the admission controller, filling in the storageclass name with the default storageclass, however, we need to consider backwards compatibility scenarios.

Maybe it’s ok. Who’s going to have an existing pending PVC with nil storageclass from before the release (1.6?) that default storageclasses was introduced?

same problem ! nfs-provisioner What is the new solution? thanks 💐

LASTSEEN   FIRSTSEEN   COUNT     NAME                KIND                    SUBOBJECT   TYPE      REASON             SOURCE                        MESSAGE
28s        1h          226       es-data-0           Pod                                 Warning   FailedScheduling   default-scheduler             PersistentVolumeClaim is not bound: "storage-es-data-0" (repeated 3 times)
15s        1h          286       storage-es-data-0   PersistentVolumeClaim               Normal    FailedBinding      persistentvolume-controller   no persistent volumes available for this claim and no storage class is set

UPDATE：Successfully created pvc using extended nfs mode.Solved @avagin @IlyaSemenov File1: Define nfs-client deployment

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: nfs-client-provisioner-large
  name: nfs-client-provisioner-large
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nfs-client-provisioner-large
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-client-provisioner-large
    spec:
      containers:
      - env:
        - name: PROVISIONER_NAME
          value: fuseim.io/ifs
        - name: NFS_SERVER
          value: 192.168.20.4
        - name: NFS_PATH
          value: /large_storage
        image: quay.io/external_storage/nfs-client-provisioner:latest
        imagePullPolicy: Always
        name: nfs-client-provisioner-xxl
        volumeMounts:
        - mountPath: /persistentvolumes
          name: nfs-client-root
      volumes:
      - name: nfs-client-root
        nfs:
          path: /large_storage
          server: 192.168.20.4
$ kubectl create -f nfs-client deployment

File2: Define storageclass

$ cat managed-nfs-storage-large.yaml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: managed-nfs-storage-large
provisioner: fuseim.io/ifs
$ kubectl create -f managed-nfs-storage-large.yaml

File3: example stateful Using pvctemplates

cat example.yaml
apiVersion: apps/v1beta1
kind: StatefulSet
.......

        volumeMounts:
        - name: storage
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: storage
    spec:
      storageClassName: managed-nfs-storage-large
      accessModes: [ ReadWriteOnce ]
      resources:
        requests:
          storage: 12Gi
$ kubectl create -f example.yaml

Check.

$ kubectl get pvc -n devops | grep es-data
storage-es-data-0             Bound     pvc-fa2d04f1-0109-11e8-8327-0242ac130006   12Gi         RWO           managed-nfs-storage-large   13h
storage-es-data-1             Bound     pvc-fcb0f311-0109-11e8-8327-0242ac130006   12Gi         RWO           managed-nfs-storage-large   13h
storage-es-data-2             Bound     pvc-fea2eb26-0109-11e8-8327-0242ac130006   12Gi         RWO           managed-nfs-storage-large   13h

The reason for the failure was because I did not choose to use storageClassName in volumeClaimTemplates. @verult thanks Hope to help everyone !