kubernetes: VSphere cloud provider settings with secret not working.

What happened:

VSphere cloud provider settings with secret not working. Kubernetes node goes into “Not-ready” state when the “/etc/cfc/conf/vsphere_cloud_conf” file is modified to use a secret (as stated in the vsphere support document here: https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/existing.html Under point number 2.). The moment a secret file is created with a base64 encoded username and password, and the details updated in the vsphere_cloud_conf file, the node status goes to “Not-ready”.

What you expected to happen:

Once the “/etc/cfc/conf/vsphere_cloud_conf” file is updated with the details of a secret file, and the kubernetes daemon and kubelet services is restarted (by running “systemctl daemon-reload” and “systemctl restart kubelet.service”), it is expected that the node comes back to a “ready” state and that all vsphere operations work fine.

How to reproduce it (as minimally and precisely as possible):

Create a vsphere secret file, secret.yaml, as follows:

apiVersion: v1
kind: Secret
metadata:
  name: vspsecret
type: Opaque
data:
  <vcenter IP 1.1.1.1>.username: <some base64 encoded username>
  1.1.1.1.password: abcdefgDBcvX==

Create the secret in the kube-system namespace:

# kubectl create -f secret.yaml -n kube-system

Modify “/etc/cfc/conf/vsphere_cloud_conf” as follows:

[Global]
secret-name="vspsecret"
secret-namespace="kube-system"
port = "443"
insecure-flag = "1"
datacenters = <Name of your datacenter>

[VirtualCenter "1.1.1.1"]
datacenters = <Name of your datacenter>

[Workspace]
server = "1.1.1.1"
default-datastore = <any datastore name>
folder = "kubernetes"


[Disk]
scsicontrollertype = pvscsi

Restart the kubelet service as follows:

# systemctl daemon-reload

# systemctl restart kubelet.service

Verify by checking the node status:

# kubectl get nodes

Here are the kubelet logs corresponding to the issue observed (Please note that the username and password have been verified with a base64 decode):

Mar  8 11:30:11 krish-icp-vsan-ubt1804-master-vm-1 hyperkube[4940]: I0308 11:30:11.404262    4940 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach                                                                                           Mar  8 11:30:12 krish-icp-vsan-ubt1804-master-vm-1 hyperkube[4940]: W0308 11:30:12.067606    4940 prober.go:103] No ref for container "docker://daf9ae0bc1331c1ab14e733790699a858d9112885e0d76c1824dc9a37d4eb7a1" (metrics-server-5bcdd5579-qtvjv_kube-system(a35f459a-3f0b-11e9-9d76-005056a814c4):metrics-server)                                                                                                                    ^C                                                                                                                                           root@krish-icp-vsan-ubt1804-master-vm-1:~# tail -f /var/log/syslog
Mar 12 05:30:32 krish-icp-vsan-ubt1804-master-vm-1 hyperkube[4940]: E0312 05:30:32.451684    4940 vsphere.go:1330] Cannot connent to vsphere. Get zone for node krish-icp-vsan-ubt1804-master-vm-1 error
Mar 12 05:30:32 krish-icp-vsan-ubt1804-master-vm-1 hyperkube[4940]: E0312 05:30:32.451700    4940 kubelet_node_status.go:66] Unable to construct v1.Node object for kubelet: failed to get zone from cloud provider: ServerFaultCode: Cannot complete login due to an incorrect user name or password.
Mar 12 05:30:34 krish-icp-vsan-ubt1804-master-vm-1 hyperkube[4940]: E0312 05:30:34.230029    4940 pod_workers.go:186] Error syncing pod a35f459a-3f0b-11e9-9d76-005056a814c4 ("metrics-server-5bcdd5579-qtvjv_kube-system(a35f459a-3f0b-11e9-9d76-005056a814c4)"), skipping: failed to "StartContainer" for "metrics-server" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=metrics-server pod=metrics-server-5bcdd5579-qtvjv_kube-system(a35f459a-3f0b-11e9-9d76-005056a814c4)"
Mar 12 05:30:39 krish-icp-vsan-ubt1804-master-vm-1 hyperkube[4940]: I0312 05:30:39.451846    4940 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Mar 12 05:30:43 krish-icp-vsan-ubt1804-master-vm-1 hyperkube[4940]: E0312 05:30:43.785103    4940 certificate_manager.go:378] Certificate request was not signed: timed out waiting for the condition
Mar 12 05:30:44 krish-icp-vsan-ubt1804-master-vm-1 hyperkube[4940]: E0312 05:30:44.470578    4940 connection.go:65] Failed to create govmomi client. err: ServerFaultCode: Cannot complete login due to an incorrect user name or password.
Mar 12 05:30:44 krish-icp-vsan-ubt1804-master-vm-1 hyperkube[4940]: E0312 05:30:44.470608    4940 nodemanager.go:382] Cannot connect to vCenter with err: ServerFaultCode: Cannot complete login due to an incorrect user name or password.
Mar 12 05:30:44 krish-icp-vsan-ubt1804-master-vm-1 hyperkube[4940]: E0312 05:30:44.470624    4940 vsphere.go:589] failed connecting to vcServer "9.42.99.254" with error ServerFaultCode: Cannot complete login due to an incorrect user name or password.
Mar 12 05:30:44 krish-icp-vsan-ubt1804-master-vm-1 hyperkube[4940]: E0312 05:30:44.470640    4940 vsphere.go:1330] Cannot connent to vsphere. Get zone for node krish-icp-vsan-ubt1804-master-vm-1 error
Mar 12 05:30:44 krish-icp-vsan-ubt1804-master-vm-1 hyperkube[4940]: E0312 05:30:44.470656    4940 kubelet_node_status.go:66] Unable to construct v1.Node object for kubelet: failed to get zone from cloud provider: ServerFaultCode: Cannot complete login due to an incorrect user name or password.
Mar 12 05:30:45 krish-icp-vsan-ubt1804-master-vm-1 hyperkube[4940]: E0312 05:30:45.228658    4940 pod_workers.go:186] Error syncing pod a35f459a-3f0b-11e9-9d76-005056a814c4 ("metrics-server-5bcdd5579-qtvjv_kube-system(a35f459a-3f0b-11e9-9d76-005056a814c4)"), skipping: failed to "StartContainer" for "metrics-server" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=metrics-server pod=metrics-server-5bcdd5579-qtvjv_kube-system(a35f459a-3f0b-11e9-9d76-005056a814c4)"

Anything else we need to know?:

The feature for using a vsphere secret was implemented in kubernetes version 1.11. I am using version 1.12.

Environment:

• Kubernetes version (use kubectl version): 1.12
• Cloud provider or hardware configuration: vSphere
• OS (e.g: cat /etc/os-release): Ubuntu 18.04
• Kernel (e.g. uname -a): Linux 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: The installation was performed using IBM Cloud Private version 3.1.2.
• Others: