cloud-provider-vsphere: cpi cannot find node - Unable to find VM by UUID

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened: CPI cannot find the VMs in vsphere. What you expected to happen: CPI to query and find all VMs, and assign a ProviderID

How to reproduce it (as minimally and precisely as possible): it happens every time in my reference setup and configuration. I presume that is either a path issue (our VMs are not in default folders), case sensitivity issue or something similar.

Anything else we need to know?: VMs are not in default folders. Datacenter has multiple clusters.

I followed the CPI part of configuration from https://github.com/kubernetes/cloud-provider-vsphere/blob/master/docs/book/tutorials/kubernetes-on-vsphere-with-kubeadm.md

on an existing RKE cluster (v1.18.3)

Environment:

• vsphere-cloud-controller-manager version: v1.2.1
• OS (e.g. from /etc/os-release): RHEL7.9
• Kernel (e.g. uname -a): 3.10.0-1160.2.2.el7.x86_64
• Others: RKE cluster (v1.18.3)

Sample log extract

` 116 17:16:42.377487 1 cloud.go:102] Starting the API Server I1116 17:16:42.377552 1 clientconn.go:105] parsed scheme: “” I1116 17:16:42.377575 1 clientconn.go:105] scheme “” not registered, fallback to default scheme I1116 17:16:42.377621 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{:43001 0 <nil>}] <nil>} I1116 17:16:42.377642 1 clientconn.go:578] ClientConn switching balancer to “pick_first” I1116 17:16:42.377795 1 balancer_conn_wrappers.go:127] pickfirstBalancer: HandleSubConnStateChange: 0xc00045f3c0, CONNECTING I1116 17:16:42.380023 1 balancer_conn_wrappers.go:127] pickfirstBalancer: HandleSubConnStateChange: 0xc00045f3c0, READY I1116 17:16:42.381505 1 search.go:76] WhichVCandDCByNodeID nodeID: e2292342-0293-1ad5-64bf-83641074bdcf I1116 17:16:42.382676 1 server.go:143] APIVersion: 0.0.1 I1116 17:16:42.382705 1 controllermanager.go:234] Starting “cloud-node” I1116 17:16:42.384597 1 node_controller.go:71] Sending events to api server. I1116 17:16:42.384677 1 controllermanager.go:244] Started “cloud-node” I1116 17:16:42.384690 1 controllermanager.go:234] Starting “cloud-node-lifecycle” I1116 17:16:42.386168 1 node_lifecycle_controller.go:77] Sending events to api server I1116 17:16:42.386254 1 controllermanager.go:244] Started “cloud-node-lifecycle” I1116 17:16:42.386267 1 controllermanager.go:234] Starting “service” W1116 17:16:42.388455 1 cloud.go:129] The vSphere cloud provider does not support load balancers E1116 17:16:42.388485 1 core.go:80] Failed to start service controller: the cloud provider does not support external load balancers W1116 17:16:42.388809 1 controllermanager.go:241] Skipping “service” I1116 17:16:42.388821 1 controllermanager.go:234] Starting “route” I1116 17:16:42.388828 1 core.go:91] Will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true. W1116 17:16:42.388849 1 controllermanager.go:241] Skipping “route” I1116 17:16:42.409970 1 search.go:76] WhichVCandDCByNodeID nodeID: dev-dev-0348 E1116 17:16:47.427581 1 connection.go:63] Failed to create govmomi client. err: ServerFaultCode: Cannot complete login due to an incorrect user name or password. I1116 17:16:47.427617 1 connectionmanager.go:152] Invalid credentials. Fetching credentials from secrets. vcServer=192.168.10.10 credentialHolder=Global E1116 17:16:47.533014 1 datacenter.go:124] Unable to find VM by UUID. VM UUID: e2292342-0293-1ad5-64bf-83641074bdcf E1116 17:16:47.533036 1 search.go:181] Error while looking for vm=e2292342-0293-1ad5-64bf-83641074bdcf(byUUID) in vc=192.168.10.10 and datacenter=NHQ_DEV: No VM found I1116 17:16:47.533055 1 search.go:186] Did not find node e2292342-0293-1ad5-64bf-83641074bdcf in vc=192.168.10.10 and datacenter=NHQ_DEV E1116 17:16:47.533086 1 nodemanager.go:144] WhichVCandDCByNodeID failed using normally formatted UUID. Err: No VM found I1116 17:16:47.533111 1 search.go:76] WhichVCandDCByNodeID nodeID: 422329e2-9302-d51a-64bf-83641074bdcf E1116 17:16:47.535879 1 datacenter.go:107] Unable to find VM by DNS Name. VM DNS Name: dev-dev-0348 E1116 17:16:47.535909 1 search.go:181] Error while looking for vm=dev-dev-0348(byName) in vc=192.168.10.10 and datacenter=NHQ_DEV: No VM found I1116 17:16:47.535921 1 search.go:186] Did not find node dev-dev-0348 in vc=192.168.10.10 and datacenter=NHQ_DEV I1116 17:16:47.535937 1 search.go:76] WhichVCandDCByNodeID nodeID: dev-dev-0348 E1116 17:16:47.553402 1 datacenter.go:90] Unable to find VM by IP. VM IP: dev-dev-0348

`

Any ideas?