kubeflow: minio-pvc and mysql-pv-claim Pods stuck in Pending state although I have created 3 persistent volume

I was following Kubeflow Troubleshooting document section “Pods stuck in Pending state”. One pvc become “Bound” state. but rest two are still pending although I have created three persistent volume.

$ kubectl -n kubeflow get pvc
NAME             STATUS    VOLUME       CAPACITY   ACCESS MODES   STORAGECLASS   AGE
katib-mysql      Bound     pv-volume3   10Gi       RWO                           70m
minio-pvc        Pending                                                         69m
mysql-pv-claim   Pending                                                         69m

$ kubectl -n kubeflow describe pvc minio-pvc
Name:          minio-pvc
Namespace:     kubeflow
StorageClass:
Status:        Pending
Volume:
Labels:        app.kubernetes.io/deploy-manager=ksonnet
               ksonnet.io/component=pipeline
Annotations:   ksonnet.io/managed:
                 {"pristine":"H4sIAAAAAAAA/ySOPU/EMBBEe37G1OGzTEtBhUAUR4Eo9pwBrWLvGq9zFKf8d+Sje3oaPc0ZUvXAFuqGGad7TFjVFsx4HTY6rR88b4WPWbRgQmGXRbpgPiPLkTkGr...
               kubecfg.ksonnet.io/garbage-collect-tag: gc-tag
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Events:
  Type       Reason         Age                  From                         Message
  ----       ------         ----                 ----                         -------
  Normal     FailedBinding  21s (x303 over 75m)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set
Mounted By:  minio-5f6b9b5b7-ms5kb

$ kubectl -n kubeflow describe pvc mysql-pv-claim
Name:          mysql-pv-claim
Namespace:     kubeflow
StorageClass:
Status:        Pending
Volume:
Labels:        app.kubernetes.io/deploy-manager=ksonnet
               ksonnet.io/component=pipeline
Annotations:   ksonnet.io/managed:
                 {"pristine":"H4sIAAAAAAAA/yyOPU8DMRBEe37G1Be+SrcUVAhEEQpEsfENyDp71/H6glB0/x05Svf0dvU0Z0hNezZPpgg4PWDCknRGwNuw3ql9b3ktfMqSCiYUdpmlC8IZWQ7MP...
               kubecfg.ksonnet.io/garbage-collect-tag: gc-tag
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Events:
  Type       Reason         Age                  From                         Message
  ----       ------         ----                 ----                         -------
  Normal     FailedBinding  38s (x303 over 75m)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set
Mounted By:  mysql-657f87857d-dfxb7

Environment details: OS: Ubuntu 18.04 Platform: x86

Thanks.

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 16 (1 by maintainers)

Most upvoted comments

I’m not at my PC right now to check this out. I’ll do it first thing tomorrow!

Thanks a ton for your help hall!

AGKhalil on Nov 25, 2019

Btw I think I see something:

$ kubectl get pv -n kubeflow
NAME         CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                              STORAGECLASS    REASON   AGE
pv-volume2   10Gi       RWO            Retain           Bound    kubeflow/metadata-mysql            local-storage            65s
pv-volume3   10Gi       RWO            Retain           Bound    kubeflow/katib-mysql               local-storage            65s
pv-volume4   20Gi       RWO            Retain           Bound    ahmed/workspace-myfirst-notebook   local-storage            65s
pv-volume5   20Gi       RWO            Retain           Bound    ahmed/workspace-first-notebook     local-storage            65s

The claim on my last 2 pvs are to a notebook I created before. I’ve been testing with the same app directory. It’s possible that I just need to delete everything and create the kubeflow app from scratch. At which point there will be no trace of my notebook servers.

AGKhalil on Nov 25, 2019

Sorry – I hit enter to add a newline and it submitted my comment with half-formed thoughts; one moment.

hall on Nov 25, 2019