kubevirt: SRIOV lane flakes: The guest VM kernel/driver hangs to configure the SRIOV device network interface

Update

the root cause for this issue is a bug in sriov-cni which we use to attach SR-IOV VFs to VMs in tests, more details at https://github.com/k8snetworkplumbingwg/sriov-cni/issues/219.

---

What happened: Various SR-IOV tests randomly fails because the guest SR-IOV network interface gets unexpected MAC address which causing cloud-init to fail leaving the VM SR-IOV interface not configured.

It doesn’t seem like Kubevirt bug as all argument are passed correctly to the domain XML. The VFs that been attached to the VMs are valid; each VF we attach is a “child” of the PF we attach to the node that VM is scheduled on.

And according to the guest dmesg log the SRIOV devices are detected by the kernel: dmesg log1, dmesg log2 (8086:154c vendor PCI ID is Intel’s x710 SRIOV card VF).

Examples:

1. Tests Suite: [Serial]SRIOV VirtualMachineInstance with sriov plugin interface With VLAN should be able to ping between two VMIs with the same VLAN over SRIOV network

One of the VMs was left with the SR-IOV interface not configured with the expected IP address causing ping to fail. It seems that there is a race between cloud-init and what seems to be the kernel / driver which handle the VF configuration. The SRIOV network interface is not created or configured on time, cloud-init looks for that interface by its MAC address but wont find it.

• On some jobs where cloud init log is visible: [5], [6] we see cloud-init[641]: RuntimeError: Not all expected physical devices present: {'02:00:00:cc:f3:d5'} As soon as cloud init finishes we see the message: iavf 0000:06:00.0 eth1: NIC Link is Up 40 Gbps Full Duplex This message may indicate that the network interface is created and up.

• On other jobs where cloud-int output of the problematic VM is not visible: [7], [8] we see systemd error message regarding cloud-init finished with an error Failed Units: 1 ... cloud-init-local.service due to cloud-init failed to configure the SRIOV network interface. Also ip a output shows the expected IP address is not configured.

---

1. [Serial]SRIOV VirtualMachineInstance with sriov plugin interface [test_id:1755]should create a virtual machine with two sriov interfaces referring the same resource On [9] the test fails on the assertion that checks that there are two SRIOV VF’s attached. The guest VM boot and cloud-init output shows that the kernel detected both VF’s devices, but there is only one SRIOV network interface. Looks like the iavf driver did not configured the missing device on time

---

1. Tests Suite: [Serial]SRIOV VirtualMachineInstance with sriov plugin interface migration should be successful with a running VMI on the target The test fails due to the assertion that checks that the SR-IOV interface status is reflected on the VMI status timeouts.

• On this job [1] we can see that according to the VM dmesg log, the SR-IOV device is plugged back and recognized by the iavf driver link but the VMI status doesn’t reflect the new state just yet.

• On other jobs [2], [3], [4] it seems that the devices failed to attach. The following errors were reported to the target virt-launcher pod log:

{"component":"virt-launcher","level":"warning","msg":"Cannot start job (modify, none, none) for domain kubevirt-test-default1_testvmi-4tcv8; current job is (none, none, migration in) owned by (0 \u003cnull\u003e, 0 \u003cnull\u003e, 0 remoteDispatchDomainMigratePrepare3Params (flags=0x8b)) for (0s, 0s, 31s)","pos":"qemuDomainObjBeginJobInternal:955","subcomponent":"libvirt","thread":"47","timestamp":"2021-11-03T12:50:37.749000Z"}
{"component":"virt-launcher","level":"error","msg":"Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainMigratePrepare3Params)","pos":"qemuDomainObjBeginJobInternal:977","subcomponent":"libvirt","thread":"47","timestamp":"2021-11-03T12:50:37.749000Z"}
{"component":"virt-launcher","kind":"","level":"error","msg":"failed to hot-plug host-devices","name":"testvmi-4tcv8","namespace":"kubevirt-test-default1","pos":"live-migration-target.go:42","reason":"failed to attach host-device \u003chostdev type=\"pci\" managed=\"no\"\u003e\u003csource\u003e\u003caddress type=\"pci\" domain=\"0x0000\" bus=\"0xaf\" slot=\"0x08\" function=\"0x0\"\u003e\u003c/address\u003e\u003c/source\u003e\u003calias name=\"ua-sriov-sriov\"\u003e\u003c/alias\u003e\u003c/hostdev\u003e, err: virError(Code=68, Domain=10, Message='Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainMigratePrepare3Params)')\n","timestamp":"2021-11-03T12:50:37.749449Z","uid":"7849661e-9497-4dfb-8a1b-65c1d611c817"}
{"component":"virt-launcher","kind":"","level":"info","msg":"migration finalized successfully","name":"testvmi-4tcv8","namespace":"kubevirt-test-default1","pos":"server.go:329","timestamp":"2021-11-03T12:50:37.749794Z","uid":"7849661e-9497-4dfb-8a1b-65c1d611c817"}