kvm-guest-drivers-windows: viostor: cause qemu virtio_error when an uncompleted request times out

Describe the bug I found an virtio_error() in QEMU when handling virtio block request with windows11 guest.

Two uncompleted blk request(req1 and reqN) has the same indirect descriptor table address. The length of indirect descriptor table in req1 is 48, but the lenth in reqN is 160. So, it cause error when handling req1 in QEMU.

The stack trace:

(gdb) bt
#0  virtio_error (vdev=0xaaaaad604420, fmt=0xaaaaabb54ad0 "Desc next is %u") at ../hw/virtio/virtio.c:3568
#1  0x0000aaaaab642868 in virtqueue_split_read_next_desc (vdev=0xaaaaad604420, desc=0xffffefff7d00, desc_cache=0xffffefff7d10, max=3, next=0xffffefff7cb4) at ../hw/virtio/virtio.c:1057
#2  0x0000aaaaab643c8c in virtqueue_split_pop (vq=0xaaaaad60f740, sz=240) at ../hw/virtio/virtio.c:1591
#3  0x0000aaaaab644584 in virtqueue_pop (vq=0xaaaaad60f740, sz=240) at ../hw/virtio/virtio.c:1768
#4  0x0000aaaaab5d8d3c in virtio_blk_get_request (s=0xaaaaad604420, vq=0xaaaaad60f740) at ../hw/block/virtio-blk.c:238
#5  0x0000aaaaab5da1fc in virtio_blk_handle_vq (s=0xaaaaad604420, vq=0xaaaaad60f740) at ../hw/block/virtio-blk.c:772
#6  0x0000aaaaab5da304 in virtio_blk_handle_output (vdev=0xaaaaad604420, vq=0xaaaaad60f740) at ../hw/block/virtio-blk.c:806
#7  0x0000aaaaab645b00 in virtio_queue_notify_vq (vq=0xaaaaad60f740) at ../hw/virtio/virtio.c:2249
#8  0x0000aaaaab648fb4 in virtio_queue_host_notifier_read (n=0xaaaaad60f7b4) at ../hw/virtio/virtio.c:3529
#9  0x0000aaaaab951690 in aio_dispatch_handler (ctx=0xaaaaac82f9a0, node=0xfffdb0002ea0) at ../util/aio-posix.c:369
#10 0x0000aaaaab951794 in aio_dispatch_ready_handlers (ctx=0xaaaaac82f9a0, ready_list=0xffffefffe040) at ../util/aio-posix.c:399
#11 0x0000aaaaab9522d0 in aio_poll (ctx=0xaaaaac82f9a0, blocking=true) at ../util/aio-posix.c:722
#12 0x0000aaaaab7a158c in iothread_run (opaque=0xaaaaac495e30) at ../iothread.c:67
#13 0x0000aaaaab956ae0 in qemu_thread_start (args=0xaaaaac82ff40) at ../util/qemu-thread-posix.c:505
#14 0x0000fffff654ef60 in  () at /usr/lib64/libc.so.6
#15 0x0000fffff65b489c in  () at /usr/lib64/libc.so.6

To Reproduce

I use gdb to attach qemu process, and debug the virtqueue_pop() in virtio block request step by step. Using gdb command ‘next/continue/print’ in the debug process which may cause the request timeout for windows wdk.

Expected behavior

Not cause virtio_error() in QEMU.

Screenshots If applicable, add screenshots to help explain your problem.

Host:

  • Disto: [e.g. Fedora, Ubuntu, Proxmox]
  • Kernel version: 5.10.0
  • QEMU version: v7.2.0
  • QEMU command line: ./build/qemu-system-aarch64 -m 8G -object memory-backend-memfd,id=mem1,size=8G,share=on -numa node,memdev=mem1 -machine virt,virtualization=off,its=off,gic-version=host -accel kvm -cpu host -smp 8 -bios …/QEMU_EFI.fd -device nec-usb-xhci -device usb-kbd -device usb-tablet -object iothread,id=disk-iothread -device virtio-blk-pci,drive=win11,iothread=disk-iothread,disable-legacy=on -drive if=none,id=win11,format=raw,file=/home/win11.raw,cache=unsafe -device ramfb -vnc :2 -serial stdio -monitor telnet:localhost:4444,server,nowait -qmp unix:/home/qemu.sock,server,nowait
  • libvirt version
  • libvirt XML file

VM:

  • Windows version: windows11
  • Which driver has a problem: viostor
  • Driver version or commit hash that was used to build the driver: virtio-win-0.1.229.iso, virtio-win-0.1.215.iso and the compiled viostor by the newest kvm-guest-drivers-windows.

Additional context

I use dbgview to capature the debug log, and found the following RESET log before calling function virtqueue_get_buf_split.

00004734    102.45079803     RESET (FFFFA60432D88740) Function 20 Cnt 0 InQueue 1

In the windows driver doc “Handling SRB_FUNCTION_RESET_DEVICE”, it said “the port driver requests a device reset when an uncompleted request times out”. https://learn.microsoft.com/en-us/windows-hardware/drivers/storage/handling-srb-function-reset-device

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Comments: 36 (22 by maintainers)

Commits related to this issue

Most upvoted comments

@benyamin-codez

Yeah, I can use the new viostor and qemu to reproduce the current problem. It may take some time.

I use the newest master(29fc692) branch code with the upper pr to compile the viostor driver. The current issue has not been fixed.

Hi Yan Wang, That’s very telling. It looks like it could be in the reset / completion pathway(s). I can probably look at this in greater depth next week. I’ve had to defer my gdb work to focus on other priorities for the next few days.

I use a new reproduce method: Add sleep 20s when handle read/write IO for the virtio-blk in qemu after the windows has been booted.

I just wanted to check that your previous method still produces the issue as well. It would be interesting if they had different backtraces… ;^d

In the meantime, can you please post updated information regarding the following here?

  1. A backtrace of the issue using the version you referenced above

Thread 3 “qemu-system-aar” hit Breakpoint 2, virtio_blk_handle_request (req=req@entry=0xffff9800b920, mrb=mrb@entry=0xffffa74798f0) at …/hw/block/virtio-blk.c:618 618 { (gdb) p sign = 1 $1 = 1 (gdb) p sign $2 = 1 (gdb) c Continuing.

Thread 3 “qemu-system-aar” hit Breakpoint 1, virtio_error (vdev=vdev@entry=0xaaaaeffe2ba0, fmt=fmt@entry=0xaaaadc911a10 “Desc next is %u”) at …/hw/virtio/virtio.c:3609 3609 { (gdb) bt #0 virtio_error (vdev=vdev@entry=0xaaaaeffe2ba0, fmt=fmt@entry=0xaaaadc911a10 “Desc next is %u”) at …/hw/virtio/virtio.c:3609 #1 0x0000aaaadc520e1c in virtqueue_split_read_next_desc (next=<synthetic pointer>, max=3, desc_cache=0xffffa7473830, desc=0xffffa7473810, vdev=0xaaaaeffe2ba0) at …/hw/virtio/virtio.c:971 #2 virtqueue_split_read_next_desc (next=<synthetic pointer>, max=3, desc_cache=0xffffa7473830, desc=0xffffa7473810, vdev=0xaaaaeffe2ba0) at …/hw/virtio/virtio.c:956 #3 virtqueue_split_pop (vq=0xaaaaeffed0b0, sz=240) at …/hw/virtio/virtio.c:1503 #4 0x0000aaaadc521098 in virtqueue_pop (vq=vq@entry=0xaaaaeffed0b0, sz=sz@entry=240) at …/hw/virtio/virtio.c:1680 #5 0x0000aaaadc4ca1c8 in virtio_blk_get_request (vq=0xaaaaeffed0b0, s=0xaaaaeffe2ba0) at …/hw/block/virtio-blk.c:258 #6 virtio_blk_handle_vq (s=0xaaaaeffe2ba0, vq=0xaaaaeffed0b0) at …/hw/block/virtio-blk.c:790 #7 0x0000aaaadc51aa78 in virtio_queue_notify_aio_vq (vq=0xaaaaeffed0b0) at …/hw/virtio/virtio.c:2311 #8 0x0000aaaadc768cf0 in aio_dispatch_handler (ctx=ctx@entry=0xaaaaeedf7410, node=0xfffd600061d0) at …/util/aio-posix.c:329 #9 0x0000aaaadc7695b8 in aio_dispatch_ready_handlers (ready_list=<optimized out>, ctx=<optimized out>) at …/util/aio-posix.c:359 #10 aio_poll (ctx=0xaaaaeedf7410, blocking=blocking@entry=true) at …/util/aio-posix.c:662 #11 0x0000aaaadc5d7cf8 in iothread_run (opaque=opaque@entry=0xaaaaeeb8f800) at …/iothread.c:66 #12 0x0000aaaadc76c858 in qemu_thread_start (args=<optimized out>) at …/util/qemu-thread-posix.c:556 #13 0x0000ffffb38cef60 in start_thread (arg=0x0) at pthread_create.c:442 #14 0x0000ffffb393489c in thread_start () at …/sysdeps/unix/sysv/linux/aarch64/clone.S:79

  1. The QEMU version
  2. The QEMU cmdline (using qm showcmd <VMID> --pretty would be good)

QEMU version is tag v6.2.0. Cmdline: build/qemu-system-aarch64 -m 8G -object memory-backend-memfd,id=mem1,size=8G,share=on -numa node,memdev=mem1 -machine virt,virtualization=off,its=off,gic-version=host -accel kvm -cpu host -smp 8 -bios ./QEMU_EFI.fd -device nec-usb-xhci -device usb-kbd -device usb-tablet -object iothread,id=disk-iothread -device virtio-blk-pci,drive=win11,iothread=disk-iothread,disable-legacy=on -drive if=none,id=win11,format=qcow2,file=win11.qcow2,cache=unsafe -device ramfb -vnc :0 -serial stdio -monitor telnet:localhost:4444,server,nowait -qmp unix:/home/qemu.sock,server,nowait

  1. The value of TimeoutValue at HKLM\System\CurrentControlSet\Services\Disk (global value)
  2. The value of IoTimeoutValue at HKLM\System\CurrentControlSet\Services\Disk (global value)
  3. The value of IoTimeoutValue at HKLM\System\CurrentControlSet\Services\viostor\Parameters
  4. The value of IoLatencyCap at HKLM\System\CurrentControlSet\Services\viostor\Parameters> > If the registry values do not exist, that is sufficient information, but I note you previously discussed this with Vadim.

I set 6 for TimeoutValue in step 4. I don’t add IoTimeoutValue/IoLatencyCap for step 5-7.

The patch for test the qemu:

diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index f139cd7cc9..145c03bc79 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -613,6 +613,7 @@ err:
     return err_status;
 }
 
+int sign = 0;
 static int virtio_blk_handle_request(VirtIOBlockReq *req, MultiReqBuffer *mrb)
 {
     uint32_t type;
@@ -624,6 +625,7 @@ static int virtio_blk_handle_request(VirtIOBlockReq *req, MultiReqBuffer *mrb)
     VirtIODevice *vdev = VIRTIO_DEVICE(s);
 
     if (req->elem.out_num < 1 || req->elem.in_num < 1) {
+        sign = 1;
         virtio_error(vdev, "virtio-blk missing headers");
         return -1;
     }
@@ -659,6 +661,9 @@ static int virtio_blk_handle_request(VirtIOBlockReq *req, MultiReqBuffer *mrb)
     switch (type & ~(VIRTIO_BLK_T_OUT | VIRTIO_BLK_T_BARRIER)) {
     case VIRTIO_BLK_T_IN:
     {
+        if (sign) {
+            sleep(20);
+        }
         bool is_write = type & VIRTIO_BLK_T_OUT;
         req->sector_num = virtio_ldq_p(vdev, &req->out.sector);