kubevirt: Missing tcg support

What happened: I was trying to start kubevirt on my AWS instance.

What you expected to happen: I was expection virt-handler to start on my AWS instance.

How to reproduce it (as minimally and precisely as possible): Try to start kubevirt on AWS.

Additional context:

virt-launcher container in virt-handler pod is failing on detecting emulator capabilities. tcg libraries are probably missing

kubectl -n kubevirt logs -f virt-handler-27s6r -c virt-launcher
error: failed to get emulator capabilities
error: internal error: Failed to start QEMU binary /usr/libexec/qemu-kvm for probing: Could not access KVM kernel module: No such file or directory
qemu-kvm: failed to initialize kvm: No such file or directory
qemu-kvm: falling back to tcg
**
ERROR:../accel/accel-softmmu.c:82:accel_init_ops_interfaces: assertion failed: (ops != NULL)

Environment:

KubeVirt version (use virtctl version): v0.54.0
Kubernetes version (use kubectl version): v1.21.7
VM or VMI specifications: ami-05e5abbfdd4424640
Cloud provider or hardware configuration: AWS EC2 instance
OS (e.g. from /etc/os-release):

CentOS Linux release 7.7.1908 (Core)
Derived from Red Hat Enterprise Linux 7.7 (Source)
NAME="CentOS Linux"
VERSION="7.2003.13 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7.2003.13 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
OSTREE_VERSION=7.2003.13
CentOS Linux release 7.7.1908 (Core)
CentOS Linux release 7.7.1908 (Core)
cpe:/o:centos:centos:7

- Kernel (e.g. `uname -a`): `Linux ip-192-168-1-33.eu-central-1.compute.internal 4.18.0-147.5.1.ves4.el7.x86_64 #1 SMP Mon Mar 16 08:47:16 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux`
- Install tools: N/A
- Others: N/A


I'll be grateful for any suggestion.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 46 (5 by maintainers)

Most upvoted comments

Meanwhile, what do you think about applying a workaround in KubeVirt? Pre-creating the directory with proper permissions seems to solve the issue:

diff --git a/cmd/virt-launcher/BUILD.bazel b/cmd/virt-launcher/BUILD.bazel
index d9cc5f252..4190794fb 100644
--- a/cmd/virt-launcher/BUILD.bazel
+++ b/cmd/virt-launcher/BUILD.bazel
@@ -154,6 +154,15 @@ pkg_tar(
     package_dir = "/etc",
 )
 
+pkg_tar(
+    name = "qemu-kvm-modules-dir-tar",
+    empty_dirs = [
+        "usr/lib64/qemu-kvm",
+    ],
+    mode = "0755",
+    owner = "0.0",
+)
+
 container_image(
     name = "version-container",
     directory = "/",
@@ -169,6 +178,7 @@ container_image(
             ":libvirt-config",
             ":passwd-tar",
             ":nsswitch-tar",
+            ":qemu-kvm-modules-dir-tar",
             "//rpm:launcherbase_x86_64",
         ],
     }),

vasiliy-ul on Sep 7, 2022

@tomkukral, what is your Kubernetes distro? Especially what container runtime is it based on (I assume docker is just for local testing)?

Using docker container runtime, custom kubernetes installation.

tomkukral on Sep 5, 2022

@tomkukral, @poojaghumre, could you check on your side the permissions of the directory (using docker run ... with the upstream image)?

# ls -la /usr/lib64/qemu-kvm

After that run

# chmod +rx /usr/lib64/qemu-kvm
# /usr/bin/node-labeler.sh

vasiliy-ul on Sep 2, 2022

Yeah, that now looks more like an issue with qemu ‘modularization’. I guess some libraries (i.e. for tcg) are now packaged in a separate RPM which is not pulled as dependency into the virt-launcher container. Therefore the dynamic loading of tcg fails. Just a thought…

Ping @andreabolognani , @rmohr , WDYT?

… on the other hand, emulation mode should be tested in CI. therefore not sure…

@vasiliy-ul thanks a lot for the heads up!

I don’t think the issue is related to QEMU modularization, as TCG support is part of the qemu-kvm-core package which we include in the image.

From inside a virt-launcher pod (HEAD points to f77d50591ddd0f74c0c876e38fdf14ca3fe54be8 here):

sh-4.4# ls -l /usr/lib64/qemu-kvm/accel-tcg-x86_64.so
-rwxr-xr-x. 1 root root 24832 Jan  1  1970 /usr/lib64/qemu-kvm/accel-tcg-x86_64.so
sh-4.4# /usr/libexec/qemu-kvm -M q35 -accel tcg
VNC server running on ::1:5900

So TCG support is present and appears to be working.

@tomkukral are you running an upstream build of KubeVirt or a downstream one? If the latter, there might be some downstream packaging decision affecting the behavior.

andreabolognani on Jul 28, 2022