vsphere-csi-driver: failed to get CsiNodeTopology for the node: no matches for kind "CSINodeTopology" in version "cns.vmware.com/v1alpha1", restarting registration container.
/kind bug
What happened: Trying to install vSphere CSI drivers v2.7.0 with RKE2 cluster v1.24.10+rke2r1.
$ cat /etc/rancher/rke2/config.yaml cloud-provider-name: external
_$ cat csi-vsphere.conf [Global] cluster-id = “${CLUSTER_NAME}” cluster-distribution = “Kubernetes”
[VirtualCenter “172.16.16.110”] insecure-flag = “true” user = “user1@vsphere.local” password = “password12345” port = “443” datacenters = “datacenter1”_
_root@urnpk8sm60:~# kubectl --namespace=vmware-system-csi get all NAME READY STATUS RESTARTS AGE pod/vsphere-csi-controller-7589ccbcf8-6w7pw 0/7 Pending 0 3m2s pod/vsphere-csi-controller-7589ccbcf8-phl5c 0/7 Pending 0 3m2s pod/vsphere-csi-controller-7589ccbcf8-wwwfc 0/7 Pending 0 3m2s pod/vsphere-csi-node-6vljg 2/3 CrashLoopBackOff 4 (79s ago) 3m2s pod/vsphere-csi-node-dpnh9 2/3 CrashLoopBackOff 5 (7s ago) 3m2s pod/vsphere-csi-node-jd4wt 2/3 CrashLoopBackOff 4 (78s ago) 3m2s pod/vsphere-csi-node-wtlp7 2/3 CrashLoopBackOff 4 (72s ago) 3m2s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/vsphere-csi-controller ClusterIP 10.43.162.210 <none> 2112/TCP,2113/TCP 3m2s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/vsphere-csi-node 4 4 0 4 0 kubernetes.io/os=linux 3m2s daemonset.apps/vsphere-csi-node-windows 0 0 0 0 0 kubernetes.io/os=windows 3m2s
NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/vsphere-csi-controller 0/3 3 0 3m2s
NAME DESIRED CURRENT READY AGE
replicaset.apps/vsphere-csi-controller-7589ccbcf8 3 3 0 3m2s
root@urnpk8sm60:~#_
root@urnpk8sm60:~# kubectl --namespace=vmware-system-csi logs pod/vsphere-csi-node-wtlp7 Defaulted container "node-driver-registrar" out of: node-driver-registrar, vsphere-csi-node, liveness-probe I0315 11:27:48.852737 1 main.go:166] Version: v2.5.1 I0315 11:27:48.852835 1 main.go:167] Running node-driver-registrar in mode=registration I0315 11:27:48.854993 1 main.go:191] Attempting to open a gRPC connection with: "/csi/csi.sock" I0315 11:27:48.855119 1 connection.go:154] Connecting to unix:///csi/csi.sock I0315 11:27:48.859495 1 main.go:198] Calling CSI driver to discover driver name I0315 11:27:48.859554 1 connection.go:183] GRPC call: /csi.v1.Identity/GetPluginInfo I0315 11:27:48.859566 1 connection.go:184] GRPC request: {} I0315 11:27:48.875719 1 connection.go:186] GRPC response: {"name":"csi.vsphere.vmware.com","vendor_version":"v2.7.0"} I0315 11:27:48.876170 1 connection.go:187] GRPC error: <nil> I0315 11:27:48.876774 1 main.go:208] CSI driver name: "csi.vsphere.vmware.com" I0315 11:27:48.877323 1 node_register.go:53] Starting Registration Server at: /registration/csi.vsphere.vmware.com-reg.sock I0315 11:27:48.878695 1 node_register.go:62] Registration Server started at: /registration/csi.vsphere.vmware.com-reg.sock I0315 11:27:48.879412 1 node_register.go:92] Skipping HTTP server because endpoint is set to: "" I0315 11:27:49.996391 1 main.go:102] Received GetInfo call: &InfoRequest{} I0315 11:27:49.998477 1 main.go:109] "Kubelet registration probe created" path="/var/lib/kubelet/plugins/csi.vsphere.vmware.com/registration" I0315 11:27:50.069958 1 main.go:120] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- plugin registration failed with err: rpc error: code = Internal desc = failed to get CsiNodeTopology for the node: "urnpk8sm60". Error: no matches for kind "CSINodeTopology" in version "cns.vmware.com/v1alpha1",} E0315 11:27:50.070058 1 main.go:122] Registration process failed with error: RegisterPlugin error -- plugin registration failed with err: rpc error: code = Internal desc = failed to get CsiNodeTopology for the node: "urnpk8sm60". Error: no matches for kind "CSINodeTopology" in version "cns.vmware.com/v1alpha1", restarting registration container. root@urnpk8sm60:~#
After changing improved-volume-topology: ‘true’ to false in vsphere-csi-driver.yaml, pod/vsphere-csi-node are running but pod/vsphere-csi-controller are still in Pending state due to node affinity/selector.
Warning FailedScheduling 26s default-scheduler 0/4 nodes are available: 4 node(s) didn’t match Pod’s node affinity/selector. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.
What you expected to happen: Same steps are working fine with vanilla Kubernetes but not working with RKE2.
Environment:
- csi-vsphere version: v2.7.0
- vsphere-cloud-controller-manager version: 1.24
- Kubernetes version: v1.24.10+rke2r1
- vSphere version: 7.0.3
- OS (e.g. from /etc/os-release): Ubuntu 22.04
- Kernel (e.g.
uname -a): 5.15.0-60-generic - Install tools: NA
- Others: NA
About this issue
- Original URL
- State: open
- Created a year ago
- Reactions: 1
- Comments: 35 (15 by maintainers)
@qdrddr After looking at labels on your nodes, I think you need to change nodeSelector in the deployment yaml file. Current nodeSelector is:
Try to change it as mentioned below and see if it works:
As per my observation, on RKE k8s cluster’s master node we get label as “node-role.kubernetes.io/control-plane=true”, whereas normal on-prem k8s cluster’s master node has label “node-role.kubernetes.io/control-plane=”.
Please let us know if pod scheduling works after making this change. I will recommend deleting old deployment first and then re-deploy CSI driver after making this change.
guys I have the same problem, I am using however version 3.0.0 I have the pods the CrashLoopBackOff:
and going into the logs in one of the pods I get this:
I’m following step by step this guide: https://docs.vmware.com/en/VMware-vSphere-Container-Storage-Plug-in/2.0/vmware-vsphere-csp-getting-started/GUID-54BB79D2-B13F-4673-8CC2-63A772D17B3C.html
My env consists of: k8s cluster 1.26.3 1 master node 2 worker node esxi 7.0.3 vcenter 7.0.3