kubernetes: "invalid memory address or nil pointer dereference" while provisioning volumes on vsphere

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): vsphere, volumes, invalid memory address or nil pointer dereference

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Kubernetes version (use kubectl version): Client Version: version.Info{Major:“1”, Minor:“5”, GitVersion:“v1.5.3”, GitCommit:“029c3a408176b55c30846f0faedf56aae5992e9b”, GitTreeState:“clean”, BuildDate:“2017-02-17T20:49:14Z”, GoVersion:“go1.8”, Compiler:“gc”, Platform:“darwin/amd64”} Server Version: version.Info{Major:“1”, Minor:“5”, GitVersion:“v1.5.3”, GitCommit:“029c3a408176b55c30846f0faedf56aae5992e9b”, GitTreeState:“clean”, BuildDate:“2017-02-15T06:34:56Z”, GoVersion:“go1.7.4”, Compiler:“gc”, Platform:“linux/amd64”}

Environment:

• Cloud provider or hardware configuration: vSphere
• OS (e.g. from /etc/os-release): NAME=“CentOS Linux” VERSION=“7 (Core)” ID=“centos” ID_LIKE=“rhel fedora” VERSION_ID=“7” PRETTY_NAME=“CentOS Linux 7 (Core)” ANSI_COLOR=“0;31” CPE_NAME=“cpe:/o:centos:centos:7” HOME_URL=“https://www.centos.org/” BUG_REPORT_URL=“https://bugs.centos.org/” CENTOS_MANTISBT_PROJECT=“CentOS-7” CENTOS_MANTISBT_PROJECT_VERSION=“7” REDHAT_SUPPORT_PRODUCT=“centos” REDHAT_SUPPORT_PRODUCT_VERSION=“7”
• Kernel (e.g. uname -a): Linux centos73x64.cg.internal 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: Kismatic
• Others:

What happened: I tried to provision a volume using a storage class using the vSphere examples

1. Created the storage class:

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: fast
provisioner: kubernetes.io/vsphere-volume
parameters:
    diskformat: zeroedthick

1. Created the pvc

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvcsc001
  annotations:
    volume.beta.kubernetes.io/storage-class: fast
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

The volume was created in vSphere.

The PVC stayed pending and the following error was produced by the controller manager:

E0321 13:35:04.574218       1 goroutinemap.go:164] Operation for "provision-default/pvcsc001[965028ea-0e36-11e7-bc7e-0050568b8bbe]" failed. No retries permitted until 2017-03-21 13:37:04.574155789 +0000 UTC (durationBeforeRetry 2m0s). Error: recovered from panic "runtime error: invalid memory address or nil pointer dereference". (err=<nil>) Call stack:
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:72
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:153
/usr/local/go/src/runtime/asm_amd64.s:479
/usr/local/go/src/runtime/panic.go:458
/usr/local/go/src/runtime/panic.go:62
/usr/local/go/src/runtime/sigpanic_unix.go:24
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/task/wait.go:73
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/task/wait.go:120
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property/wait.go:140
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property/wait.go:67
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/task/wait.go:120
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/object/task.go:51
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/object/task.go:45
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:1252
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/vsphere_volume/vsphere_volume_util.go:88
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/vsphere_volume/vsphere_volume.go:337
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1302
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1204
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/goroutinemap/goroutinemap.go:128
/usr/local/go/src/runtime/asm_amd64.s:2086

What you expected to happen: PVC should be successfully created and a new volume should be added in vSphere.

How to reproduce it (as minimally and precisely as possible):

1. use the vSphere Cloud provider

2. Create the storage class:

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: fast
provisioner: kubernetes.io/vsphere-volume
parameters:
    diskformat: zeroedthick

3… Create the pvc

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvcsc001
  annotations:
    volume.beta.kubernetes.io/storage-class: fast
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

Anything else we need to know: Using predefined volumes works fine.

Tagging @kerneltime