kubevirt: registration request from device plugin failing with kubeadm + containerd
I’m trying to get a kubernetes cluster up to test kubevirt however I’m running into the same issue with CentOS8 and Ubuntu22.
For the kubelet this is what I see:
Sep 17 22:08:16 virt-node01 kubelet[1618]: I0917 22:08:16.940489 1618 server.go:142] "Got registration request from device plugin with resource" resourceName="devices.kubevirt.io/kvm"
Sep 17 22:08:16 virt-node01 kubelet[1618]: I0917 22:08:16.999311 1618 server.go:142] "Got registration request from device plugin with resource" resourceName="devices.kubevirt.io/tun"
Sep 17 22:08:17 virt-node01 kubelet[1618]: I0917 22:08:17.105519 1618 server.go:142] "Got registration request from device plugin with resource" resourceName="devices.kubevirt.io/vhost-net"
Sep 17 22:08:17 virt-node01 kubelet[1618]: I0917 22:08:17.105519 1618 server.go:142] "Got registration request from device plugin with resource" resourceName="devices.kubevirt.io/vhost-net"
Sep 17 22:08:26 virt-node01 kubelet[1618]: E0917 22:08:26.941467 1618 client.go:65] "Unable to connect to device plugin client with socket path" err="failed to dial device plugin: context deadline exceeded" path="/var/lib/kubelet/device-plugins/k>
Sep 17 22:08:26 virt-node01 kubelet[1618]: E0917 22:08:26.941520 1618 handler.go:73] "Failed to connect to new client" err="failed to dial device plugin: context deadline exceeded" resource="devices.kubevirt.io/kvm"
Sep 17 22:08:26 virt-node01 kubelet[1618]: I0917 22:08:26.941560 1618 server.go:158] "Error connecting to device plugin client" err="failed to dial device plugin: context deadline exceeded"
Sep 17 22:08:27 virt-node01 kubelet[1618]: E0917 22:08:26.999929 1618 client.go:65] "Unable to connect to device plugin client with socket path" err="failed to dial device plugin: context deadline exceeded" path="/var/lib/kubelet/device-plugins/k>
Sep 17 22:08:27 virt-node01 kubelet[1618]: E0917 22:08:26.999987 1618 handler.go:73] "Failed to connect to new client" err="failed to dial device plugin: context deadline exceeded" resource="devices.kubevirt.io/tun"
Sep 17 22:08:27 virt-node01 kubelet[1618]: I0917 22:08:27.000014 1618 server.go:158] "Error connecting to device plugin client" err="failed to dial device plugin: context deadline exceeded"
Sep 17 22:08:27 virt-node01 kubelet[1618]: E0917 22:08:27.106404 1618 client.go:65] "Unable to connect to device plugin client with socket path" err="failed to dial device plugin: context deadline exceeded" path="/var/lib/kubelet/device-plugins/k>
Sep 17 22:08:27 virt-node01 kubelet[1618]: E0917 22:08:27.106456 1618 handler.go:73] "Failed to connect to new client" err="failed to dial device plugin: context deadline exceeded" resource="devices.kubevirt.io/vhost-net"
Sep 17 22:08:27 virt-node01 kubelet[1618]: I0917 22:08:27.106472 1618 server.go:158] "Error connecting to device plugin client" err="failed to dial device plugin: context deadline exceeded"
I also see errors on virt-handler pod:
{"component":"virt-handler","level":"error","msg":"Error starting kvm device plugin","pos":"device_controller.go:69","reason":"error registering with device plugin manager: rpc error: code = Unknown desc = failed to dial device plugin: context deadline exceeded","timestamp":"2022-09-17T22:17:00.038220Z"}
{"component":"virt-handler","level":"error","msg":"Error starting tun device plugin","pos":"device_controller.go:69","reason":"error registering with device plugin manager: rpc error: code = Unknown desc = failed to dial device plugin: context deadline exceeded","timestamp":"2022-09-17T22:17:00.121864Z"}
{"component":"virt-handler","level":"error","msg":"Error starting vhost-net device plugin","pos":"device_controller.go:69","reason":"error registering with device plugin manager: rpc error: code = Unknown desc = failed to dial device plugin: context deadline exceeded","timestamp":"2022-09-17T22:17:00.388707Z"}
I’ve tested my system and it looks ok:
[root@virt-node01 centos]# virt-host-validate
QEMU: Checking for hardware virtualization : PASS
QEMU: Checking if device /dev/kvm exists : PASS
QEMU: Checking if device /dev/kvm is accessible : PASS
QEMU: Checking if device /dev/vhost-net exists : PASS
QEMU: Checking if device /dev/net/tun exists : PASS
QEMU: Checking for cgroup 'cpu' controller support : PASS
QEMU: Checking for cgroup 'cpuacct' controller support : PASS
QEMU: Checking for cgroup 'cpuset' controller support : PASS
QEMU: Checking for cgroup 'memory' controller support : PASS
QEMU: Checking for cgroup 'devices' controller support : PASS
QEMU: Checking for cgroup 'blkio' controller support : PASS
QEMU: Checking for device assignment IOMMU support : WARN (No ACPI DMAR table found, IOMMU either disabled in BIOS or not supported by this hardware platform)
QEMU: Checking for secure guest support : WARN (Unknown if this platform has Secure Guest support)
[root@virt-node01 centos]#
I see the files popping up on /var/lib/kubelet/device-plugins/, however they come and go…
I’ve also tried adding --feature-gates=DevicePlugins=true to my Kubelet.
This is my node info:
[root@virt-node01 centos]# kubectl describe node virt-node01
Name: virt-node01
Roles: control-plane
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
cpu-feature.node.kubevirt.io/aes=true
cpu-feature.node.kubevirt.io/apic=true
cpu-feature.node.kubevirt.io/arat=true
cpu-feature.node.kubevirt.io/arch-capabilities=true
cpu-feature.node.kubevirt.io/avx=true
cpu-feature.node.kubevirt.io/clflush=true
cpu-feature.node.kubevirt.io/cmov=true
cpu-feature.node.kubevirt.io/cx16=true
cpu-feature.node.kubevirt.io/cx8=true
cpu-feature.node.kubevirt.io/de=true
cpu-feature.node.kubevirt.io/erms=true
cpu-feature.node.kubevirt.io/f16c=true
cpu-feature.node.kubevirt.io/fpu=true
cpu-feature.node.kubevirt.io/fsgsbase=true
cpu-feature.node.kubevirt.io/fxsr=true
cpu-feature.node.kubevirt.io/hypervisor=true
cpu-feature.node.kubevirt.io/lahf_lm=true
cpu-feature.node.kubevirt.io/lm=true
cpu-feature.node.kubevirt.io/mca=true
cpu-feature.node.kubevirt.io/mce=true
cpu-feature.node.kubevirt.io/mmx=true
cpu-feature.node.kubevirt.io/msr=true
cpu-feature.node.kubevirt.io/mtrr=true
cpu-feature.node.kubevirt.io/nx=true
cpu-feature.node.kubevirt.io/pae=true
cpu-feature.node.kubevirt.io/pat=true
cpu-feature.node.kubevirt.io/pcid=true
cpu-feature.node.kubevirt.io/pclmuldq=true
cpu-feature.node.kubevirt.io/pdcm=true
cpu-feature.node.kubevirt.io/pge=true
cpu-feature.node.kubevirt.io/pni=true
cpu-feature.node.kubevirt.io/popcnt=true
cpu-feature.node.kubevirt.io/pschange-mc-no=true
cpu-feature.node.kubevirt.io/pse=true
cpu-feature.node.kubevirt.io/pse36=true
cpu-feature.node.kubevirt.io/rdrand=true
cpu-feature.node.kubevirt.io/rdtscp=true
cpu-feature.node.kubevirt.io/sep=true
cpu-feature.node.kubevirt.io/skip-l1dfl-vmentry=true
cpu-feature.node.kubevirt.io/smep=true
cpu-feature.node.kubevirt.io/ss=true
cpu-feature.node.kubevirt.io/sse=true
cpu-feature.node.kubevirt.io/sse2=true
cpu-feature.node.kubevirt.io/sse4.1=true
cpu-feature.node.kubevirt.io/sse4.2=true
cpu-feature.node.kubevirt.io/ssse3=true
cpu-feature.node.kubevirt.io/syscall=true
cpu-feature.node.kubevirt.io/tsc=true
cpu-feature.node.kubevirt.io/tsc-deadline=true
cpu-feature.node.kubevirt.io/tsc_adjust=true
cpu-feature.node.kubevirt.io/umip=true
cpu-feature.node.kubevirt.io/vme=true
cpu-feature.node.kubevirt.io/vmx=true
cpu-feature.node.kubevirt.io/x2apic=true
cpu-feature.node.kubevirt.io/xsave=true
cpu-feature.node.kubevirt.io/xsaveopt=true
cpu-model-migration.node.kubevirt.io/IvyBridge=true
cpu-model-migration.node.kubevirt.io/Nehalem=true
cpu-model-migration.node.kubevirt.io/Opteron_G1=true
cpu-model-migration.node.kubevirt.io/Opteron_G2=true
cpu-model-migration.node.kubevirt.io/Penryn=true
cpu-model-migration.node.kubevirt.io/SandyBridge=true
cpu-model-migration.node.kubevirt.io/Westmere=true
cpu-model.node.kubevirt.io/IvyBridge=true
cpu-model.node.kubevirt.io/Nehalem=true
cpu-model.node.kubevirt.io/Opteron_G1=true
cpu-model.node.kubevirt.io/Opteron_G2=true
cpu-model.node.kubevirt.io/Penryn=true
cpu-model.node.kubevirt.io/SandyBridge=true
cpu-model.node.kubevirt.io/Westmere=true
cpu-vendor.node.kubevirt.io/Intel=true
cpumanager=false
host-model-cpu.node.kubevirt.io/IvyBridge=true
host-model-required-features.node.kubevirt.io/arat=true
host-model-required-features.node.kubevirt.io/arch-capabilities=true
host-model-required-features.node.kubevirt.io/hypervisor=true
host-model-required-features.node.kubevirt.io/pcid=true
host-model-required-features.node.kubevirt.io/pdcm=true
host-model-required-features.node.kubevirt.io/pschange-mc-no=true
host-model-required-features.node.kubevirt.io/skip-l1dfl-vmentry=true
host-model-required-features.node.kubevirt.io/ss=true
host-model-required-features.node.kubevirt.io/tsc_adjust=true
host-model-required-features.node.kubevirt.io/umip=true
host-model-required-features.node.kubevirt.io/vmx=true
host-model-required-features.node.kubevirt.io/xsaveopt=true
hyperv.node.kubevirt.io/base=true
hyperv.node.kubevirt.io/frequencies=true
hyperv.node.kubevirt.io/ipi=true
hyperv.node.kubevirt.io/reenlightenment=true
hyperv.node.kubevirt.io/reset=true
hyperv.node.kubevirt.io/runtime=true
hyperv.node.kubevirt.io/synic=true
hyperv.node.kubevirt.io/synic2=true
hyperv.node.kubevirt.io/synictimer=true
hyperv.node.kubevirt.io/time=true
hyperv.node.kubevirt.io/tlbflush=true
hyperv.node.kubevirt.io/vpindex=true
kubernetes.io/arch=amd64
kubernetes.io/hostname=virt-node01
kubernetes.io/os=linux
kubevirt.io/schedulable=false
node-role.kubernetes.io/control-plane=
node.kubernetes.io/exclude-from-external-load-balancers=
Annotations: csi.volume.kubernetes.io/nodeid: {"csi.tigera.io":"virt-node01 "}
kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
kubevirt.io/heartbeat: 2022-09-17T22:20:43Z
node.alpha.kubernetes.io/ttl: 0
projectcalico.org/IPv4Address: 172.16.230.51/24
projectcalico.org/IPv4VXLANTunnelAddr: 10.244.11.0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Sat, 17 Sep 2022 21:42:25 +0000
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: virt-node01
AcquireTime: <unset>
RenewTime: Sat, 17 Sep 2022 22:21:18 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Sat, 17 Sep 2022 22:07:22 +0000 Sat, 17 Sep 2022 22:07:22 +0000 CalicoIsUp Calico is running on this node
MemoryPressure False Sat, 17 Sep 2022 22:16:55 +0000 Sat, 17 Sep 2022 21:42:25 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Sat, 17 Sep 2022 22:16:55 +0000 Sat, 17 Sep 2022 21:42:25 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Sat, 17 Sep 2022 22:16:55 +0000 Sat, 17 Sep 2022 21:42:25 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Sat, 17 Sep 2022 22:16:55 +0000 Sat, 17 Sep 2022 21:48:21 +0000 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 172.16.230.51
Hostname: virt-node01
Capacity:
cpu: 8
ephemeral-storage: 157275116Ki
hugepages-2Mi: 0
memory: 14142284Ki
pods: 110
Allocatable:
cpu: 8
ephemeral-storage: 144944746666
hugepages-2Mi: 0
memory: 14039884Ki
pods: 110
System Info:
Machine ID: 4e10552abb674974b4778ed76a2c2b2e
System UUID: 4e10552a-bb67-4974-b477-8ed76a2c2b2e
Boot ID: 0d1d8885-c569-44a2-be53-ecf00a94f839
Kernel Version: 4.18.0-358.el8.x86_64
OS Image: CentOS Stream 8
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.6.8
Kubelet Version: v1.25.1
Kube-Proxy Version: v1.25.1
PodCIDR: 10.244.0.0/24
PodCIDRs: 10.244.0.0/24
Non-terminated Pods: (21 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
calico-apiserver calico-apiserver-7644b49445-2ttm5 0 (0%) 0 (0%) 0 (0%) 0 (0%) 32m
calico-apiserver calico-apiserver-7644b49445-cv4dd 0 (0%) 0 (0%) 0 (0%) 0 (0%) 32m
calico-system calico-kube-controllers-85666c5b94-v4xdp 0 (0%) 0 (0%) 0 (0%) 0 (0%) 34m
calico-system calico-node-8nhx6 0 (0%) 0 (0%) 0 (0%) 0 (0%) 34m
calico-system calico-typha-7b4bc5cd58-hv55p 0 (0%) 0 (0%) 0 (0%) 0 (0%) 34m
calico-system csi-node-driver-s2pkj 0 (0%) 0 (0%) 0 (0%) 0 (0%) 33m
kube-system coredns-565d847f94-ghn76 100m (1%) 0 (0%) 70Mi (0%) 170Mi (1%) 38m
kube-system coredns-565d847f94-mpv9s 100m (1%) 0 (0%) 70Mi (0%) 170Mi (1%) 38m
kube-system etcd-virt-node01 100m (1%) 0 (0%) 100Mi (0%) 0 (0%) 38m
kube-system kube-apiserver-virt-node01 250m (3%) 0 (0%) 0 (0%) 0 (0%) 38m
kube-system kube-controller-manager-virt-node01 200m (2%) 0 (0%) 0 (0%) 0 (0%) 38m
kube-system kube-proxy-ngndc 0 (0%) 0 (0%) 0 (0%) 0 (0%) 38m
kube-system kube-scheduler-virt-node01 100m (1%) 0 (0%) 0 (0%) 0 (0%) 38m
kube-system metrics-server-696f4844bb-dshxd 100m (1%) 0 (0%) 200Mi (1%) 0 (0%) 25m
kubevirt virt-api-68d4b788f7-2fd5q 5m (0%) 0 (0%) 250Mi (1%) 0 (0%) 23m
kubevirt virt-controller-887fd7878-8khwq 10m (0%) 0 (0%) 250Mi (1%) 0 (0%) 22m
kubevirt virt-controller-887fd7878-smd5x 10m (0%) 0 (0%) 250Mi (1%) 0 (0%) 22m
kubevirt virt-handler-lcf9v 10m (0%) 0 (0%) 230Mi (1%) 0 (0%) 22m
kubevirt virt-operator-7bfd844447-8hd6g 10m (0%) 0 (0%) 250Mi (1%) 0 (0%) 24m
kubevirt virt-operator-7bfd844447-cgg5j 10m (0%) 0 (0%) 250Mi (1%) 0 (0%) 24m
tigera-operator tigera-operator-6675dc47f4-k7fjn 0 (0%) 0 (0%) 0 (0%) 0 (0%) 34m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 1005m (12%) 0 (0%)
memory 1920Mi (14%) 340Mi (2%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 14m kube-proxy
Normal Starting 38m kube-proxy
Normal NodeHasSufficientPID 39m (x7 over 39m) kubelet Node virt-node01 status is now: NodeHasSufficientPID
Normal NodeHasSufficientMemory 39m (x8 over 39m) kubelet Node virt-node01 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 39m (x7 over 39m) kubelet Node virt-node01 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 38m kubelet Node virt-node01 status is now: NodeHasSufficientPID
Normal Starting 38m kubelet Starting kubelet.
Warning InvalidDiskCapacity 38m kubelet invalid capacity 0 on image filesystem
Normal NodeHasSufficientMemory 38m kubelet Node virt-node01 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 38m kubelet Node virt-node01 status is now: NodeHasNoDiskPressure
Normal NodeAllocatableEnforced 38m kubelet Updated Node Allocatable limit across pods
Normal RegisteredNode 38m node-controller Node virt-node01 event: Registered Node virt-node01 in Controller
Normal NodeReady 33m kubelet Node virt-node01 status is now: NodeReady
Normal RegisteredNode 21m node-controller Node virt-node01 event: Registered Node virt-node01 in Controller
Warning InvalidDiskCapacity 15m kubelet invalid capacity 0 on image filesystem
Normal Starting 15m kubelet Starting kubelet.
Normal NodeHasSufficientMemory 14m (x8 over 15m) kubelet Node virt-node01 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 14m (x7 over 15m) kubelet Node virt-node01 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 14m (x7 over 15m) kubelet Node virt-node01 status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 14m kubelet Updated Node Allocatable limit across pods
Normal RegisteredNode 13m node-controller Node virt-node01 event: Registered Node virt-node01 in Controller
and this is my cluster info
[root@virt-node01 centos]# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.1", GitCommit:"e4d4e1ab7cf1bf15273ef97303551b279f0920a9", GitTreeState:"clean", BuildDate:"2022-09-14T19:49:27Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.1", GitCommit:"e4d4e1ab7cf1bf15273ef97303551b279f0920a9", GitTreeState:"clean", BuildDate:"2022-09-14T19:42:30Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"linux/amd64"}
[root@virt-node01 centos]# kubectl cluster-info
Kubernetes control plane is running at https://172.16.230.51:6443
CoreDNS is running at https://172.16.230.51:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
[root@virt-node01 centos]#
Any help is much appreciated.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17 (10 by maintainers)
+1 fyi @fabiand @aburdenthehand
Hi @vasiliy-ul , just validated on 1.24… you’re correct, I don’t see the error anymore. I’ll try to create a few VMs to make sure everything looks good. Thanks!
Same as in https://github.com/kubevirt/kubevirt/issues/8478
Hi @feitnomore, this is a known issue that currently KubeVirt does not work with Kubernetes
1.25because of the device plugin registration failure. There is already a fix for that, but it is not yet released. You can try with Kubernetes1.24, that should solve the problem