kubernetes: kubelet failed to start because it failed to get zone from vsphere cloud provider

What happened:

Enabled vSphere cloud provider in Kubernetes v1.12.4 cluster. But found kubelet failed to start due error to get zone from vSphere cloud provider.

Jan 25 07:58:25 bws-01y-20190121-mst01 hyperkube[31519]: E0125 07:58:25.351818   31519 vsphere.go:1358] Cannot list attached tags. Get zone for node bws-01y-20190121-mst01: POST https://cicd-vc1.rtp.x.x.x:443/rest/com/vmware/cis/tagging/tag-association?~action=list-attached-tags: 403 Forbidden
Jan 25 07:58:25 bws-01y-20190121-mst01 hyperkube[31519]: E0125 07:58:25.361085   31519 vsphere.go:1405] Get zone for node bws-01y-20190121-mst01: POST https://cicd-vc1.rtp.x.x.x:443/rest/com/vmware/cis/tagging/tag-association?~action=list-attached-tags: 403 Forbidden
Jan 25 07:58:25 bws-01y-20190121-mst01 hyperkube[31519]: F0125 07:58:25.361109   31519 kubelet.go:1354] Kubelet failed to get node info: failed to get zone from cloud provider: POST https://cicd-vc1.rtp.x.x.x:443/rest/com/vmware/cis/tagging/tag-association?~action=list-attached-tags: 403 Forbidden

# cat /etc/systemd/system/kubelet.service
...
  --cloud-provider=vsphere \
  --cloud-config=/etc/cfc/conf/vsphere_cloud_conf \

# cat /etc/cfc/conf/vsphere_cloud_conf
[Global]
   user=username
   password=passw0rd
   server=cicd-vc1.x.x.x.x
   port=443
   insecure-flag=1
   datacenter=wlot01
   datastore=DS01
   working-dir=bws-01y
[Disk]
   scsicontrollertype=pvscsi

While the same conf works for Kubernetes v1.11.5.

What you expected to happen:

Kubelet should start successfully.

How to reproduce it (as minimally and precisely as possible):

Just build a Kubernetes v1.12.4 cluster with vSphere cloud provider enabled.

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version): v1.12.4
Cloud provider or hardware configuration: vSphere
OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS
Kernel (e.g. uname -a): 4.4.0-87-generic
Install tools:
Others:

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 25 (9 by maintainers)

Commits related to this issue

Updated privileges required for zone The VCP zone feature is based on tagging the Datacenter/Cluster/Hosts with the zone and region tags in the vCenter inventory. At startup, kubelet fetches these ta... — committed to subramanian-neelakantan/vsphere-storage-for-kubernetes by subramanian-neelakantan 5 years ago
Updated privileges required for zone (#43) The VCP zone feature is based on tagging the Datacenter/Cluster/Hosts with the zone and region tags in the vCenter inventory. At startup, kubelet fetches ... — committed to vmware-archive/vsphere-storage-for-kubernetes by subramanian-neelakantan 5 years ago

Most upvoted comments

@liqlin2015 For querying tags associated with DataCenter, Cluster and Host objects, the VC user configured in your vsphere.conf needs to have System.Read privilege on that DataCenter, Cluster and hosts on which the k8s nodes are running in vCenter.

You could verify these privileges in two ways that I can think of.

Using PowerCLI. Invoke the Get-TagAssignment cmdlet after logging into the VC as this user. Invoke this on the DataCenter, Cluster and all Host objects in which the k8s VMs are running.
Using VC Managed Object Browser (mob) https://<vc_ip>/invsvc/mob1, logging in as this VC user and invoking the QueryAttachedInventoryServiceTags() API. Invoke this on the DataCenter, Cluster and all Host objects in which the k8s VMs are running.

If the VC user does not have the necessary privilege, you should see a NotAuthorized error, which needs to be fixed by adjusting the permissions in VC for this user on that DataCenter/Cluster/Host.

subramanian-neelakantan on Jan 30, 2019

Hi @monica-wei, thanks for confirming. Really appreciate you reporting the documentation issue in https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/vcp-roles.html.

subramanian-neelakantan on Feb 18, 2019

@subramanian-neelakantan We have the privilege required for dynamic persistent volume provisioning with storage policy based volume placement. Can tell which privilege is needed for querying tags associated with DataCenter, Cluster and hosts objects?

liqlin2015 on Jan 29, 2019

This looks like a privilege issue for the kubelet’s VC user that is connecting to vCenter to list the tags associated with host/cluster/datacenter. Can you verify whether this VC user has the necessary permissions setup as described in https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/vcp-roles.html? Can this user list the tags associated with a host and a cluster and a datacenter?

There is no easy way to disable zone support in vsphere cloud provider. So, you would have to get the VC user permissions fixed for the kubelet to become active again.

subramanian-neelakantan on Jan 28, 2019