kvm-guest-drivers-windows: viostor: cause qemu virtio_error when an uncompleted request times out
Describe the bug I found an virtio_error() in QEMU when handling virtio block request with windows11 guest.
Two uncompleted blk request(req1 and reqN) has the same indirect descriptor table address. The length of indirect descriptor table in req1 is 48, but the lenth in reqN is 160. So, it cause error when handling req1 in QEMU.
The stack trace:
(gdb) bt
#0 virtio_error (vdev=0xaaaaad604420, fmt=0xaaaaabb54ad0 "Desc next is %u") at ../hw/virtio/virtio.c:3568
#1 0x0000aaaaab642868 in virtqueue_split_read_next_desc (vdev=0xaaaaad604420, desc=0xffffefff7d00, desc_cache=0xffffefff7d10, max=3, next=0xffffefff7cb4) at ../hw/virtio/virtio.c:1057
#2 0x0000aaaaab643c8c in virtqueue_split_pop (vq=0xaaaaad60f740, sz=240) at ../hw/virtio/virtio.c:1591
#3 0x0000aaaaab644584 in virtqueue_pop (vq=0xaaaaad60f740, sz=240) at ../hw/virtio/virtio.c:1768
#4 0x0000aaaaab5d8d3c in virtio_blk_get_request (s=0xaaaaad604420, vq=0xaaaaad60f740) at ../hw/block/virtio-blk.c:238
#5 0x0000aaaaab5da1fc in virtio_blk_handle_vq (s=0xaaaaad604420, vq=0xaaaaad60f740) at ../hw/block/virtio-blk.c:772
#6 0x0000aaaaab5da304 in virtio_blk_handle_output (vdev=0xaaaaad604420, vq=0xaaaaad60f740) at ../hw/block/virtio-blk.c:806
#7 0x0000aaaaab645b00 in virtio_queue_notify_vq (vq=0xaaaaad60f740) at ../hw/virtio/virtio.c:2249
#8 0x0000aaaaab648fb4 in virtio_queue_host_notifier_read (n=0xaaaaad60f7b4) at ../hw/virtio/virtio.c:3529
#9 0x0000aaaaab951690 in aio_dispatch_handler (ctx=0xaaaaac82f9a0, node=0xfffdb0002ea0) at ../util/aio-posix.c:369
#10 0x0000aaaaab951794 in aio_dispatch_ready_handlers (ctx=0xaaaaac82f9a0, ready_list=0xffffefffe040) at ../util/aio-posix.c:399
#11 0x0000aaaaab9522d0 in aio_poll (ctx=0xaaaaac82f9a0, blocking=true) at ../util/aio-posix.c:722
#12 0x0000aaaaab7a158c in iothread_run (opaque=0xaaaaac495e30) at ../iothread.c:67
#13 0x0000aaaaab956ae0 in qemu_thread_start (args=0xaaaaac82ff40) at ../util/qemu-thread-posix.c:505
#14 0x0000fffff654ef60 in () at /usr/lib64/libc.so.6
#15 0x0000fffff65b489c in () at /usr/lib64/libc.so.6
To Reproduce
I use gdb to attach qemu process, and debug the virtqueue_pop() in virtio block request step by step. Using gdb command ‘next/continue/print’ in the debug process which may cause the request timeout for windows wdk.
Expected behavior
Not cause virtio_error() in QEMU.
Screenshots If applicable, add screenshots to help explain your problem.
Host:
- Disto: [e.g. Fedora, Ubuntu, Proxmox]
- Kernel version: 5.10.0
- QEMU version: v7.2.0
- QEMU command line: ./build/qemu-system-aarch64 -m 8G -object memory-backend-memfd,id=mem1,size=8G,share=on -numa node,memdev=mem1 -machine virt,virtualization=off,its=off,gic-version=host -accel kvm -cpu host -smp 8 -bios …/QEMU_EFI.fd -device nec-usb-xhci -device usb-kbd -device usb-tablet -object iothread,id=disk-iothread -device virtio-blk-pci,drive=win11,iothread=disk-iothread,disable-legacy=on -drive if=none,id=win11,format=raw,file=/home/win11.raw,cache=unsafe -device ramfb -vnc :2 -serial stdio -monitor telnet:localhost:4444,server,nowait -qmp unix:/home/qemu.sock,server,nowait
- libvirt version
- libvirt XML file
VM:
- Windows version: windows11
- Which driver has a problem: viostor
- Driver version or commit hash that was used to build the driver: virtio-win-0.1.229.iso, virtio-win-0.1.215.iso and the compiled viostor by the newest kvm-guest-drivers-windows.
Additional context
I use dbgview to capature the debug log, and found the following RESET log before calling function virtqueue_get_buf_split.
00004734 102.45079803 RESET (FFFFA60432D88740) Function 20 Cnt 0 InQueue 1
In the windows driver doc “Handling SRB_FUNCTION_RESET_DEVICE”, it said “the port driver requests a device reset when an uncompleted request times out”. https://learn.microsoft.com/en-us/windows-hardware/drivers/storage/handling-srb-function-reset-device
About this issue
- Original URL
- State: open
- Created 4 years ago
- Comments: 36 (22 by maintainers)
Commits related to this issue
- [viostor] Fix returning SUCCESS when really -ENOSPC (regression) 1. Refactored to only return TRUE when add_buffer_req_status == VQ_ADD_BUFFER_SUCCESS. 2. Pro-actively refactors calls to Interlocked... — committed to benyamin-codez/kvm-guest-drivers-windows by benyamin-codez 2 years ago
- [viostor] Fix returning SUCCESS when really -ENOSPC (regression) 1. Refactored to only return TRUE when add_buffer_req_status == VQ_ADD_BUFFER_SUCCESS. 2. Pro-actively refactors calls to Interlocked... — committed to virtio-win/kvm-guest-drivers-windows by benyamin-codez 2 years ago
- [viostor] Fix returning SUCCESS when really -ENOSPC (regression) 1. Refactored to only return TRUE when add_buffer_req_status == VQ_ADD_BUFFER_SUCCESS. 2. Pro-actively refactors calls to Interlocked... — committed to MartinDrab/kvm-guest-drivers-windows by benyamin-codez 2 years ago
@benyamin-codez
Yeah, I can use the new viostor and qemu to reproduce the current problem. It may take some time.
Thread 3 “qemu-system-aar” hit Breakpoint 2, virtio_blk_handle_request (req=req@entry=0xffff9800b920, mrb=mrb@entry=0xffffa74798f0) at …/hw/block/virtio-blk.c:618 618 { (gdb) p sign = 1 $1 = 1 (gdb) p sign $2 = 1 (gdb) c Continuing.
Thread 3 “qemu-system-aar” hit Breakpoint 1, virtio_error (vdev=vdev@entry=0xaaaaeffe2ba0, fmt=fmt@entry=0xaaaadc911a10 “Desc next is %u”) at …/hw/virtio/virtio.c:3609 3609 { (gdb) bt #0 virtio_error (vdev=vdev@entry=0xaaaaeffe2ba0, fmt=fmt@entry=0xaaaadc911a10 “Desc next is %u”) at …/hw/virtio/virtio.c:3609 #1 0x0000aaaadc520e1c in virtqueue_split_read_next_desc (next=<synthetic pointer>, max=3, desc_cache=0xffffa7473830, desc=0xffffa7473810, vdev=0xaaaaeffe2ba0) at …/hw/virtio/virtio.c:971 #2 virtqueue_split_read_next_desc (next=<synthetic pointer>, max=3, desc_cache=0xffffa7473830, desc=0xffffa7473810, vdev=0xaaaaeffe2ba0) at …/hw/virtio/virtio.c:956 #3 virtqueue_split_pop (vq=0xaaaaeffed0b0, sz=240) at …/hw/virtio/virtio.c:1503 #4 0x0000aaaadc521098 in virtqueue_pop (vq=vq@entry=0xaaaaeffed0b0, sz=sz@entry=240) at …/hw/virtio/virtio.c:1680 #5 0x0000aaaadc4ca1c8 in virtio_blk_get_request (vq=0xaaaaeffed0b0, s=0xaaaaeffe2ba0) at …/hw/block/virtio-blk.c:258 #6 virtio_blk_handle_vq (s=0xaaaaeffe2ba0, vq=0xaaaaeffed0b0) at …/hw/block/virtio-blk.c:790 #7 0x0000aaaadc51aa78 in virtio_queue_notify_aio_vq (vq=0xaaaaeffed0b0) at …/hw/virtio/virtio.c:2311 #8 0x0000aaaadc768cf0 in aio_dispatch_handler (ctx=ctx@entry=0xaaaaeedf7410, node=0xfffd600061d0) at …/util/aio-posix.c:329 #9 0x0000aaaadc7695b8 in aio_dispatch_ready_handlers (ready_list=<optimized out>, ctx=<optimized out>) at …/util/aio-posix.c:359 #10 aio_poll (ctx=0xaaaaeedf7410, blocking=blocking@entry=true) at …/util/aio-posix.c:662 #11 0x0000aaaadc5d7cf8 in iothread_run (opaque=opaque@entry=0xaaaaeeb8f800) at …/iothread.c:66 #12 0x0000aaaadc76c858 in qemu_thread_start (args=<optimized out>) at …/util/qemu-thread-posix.c:556 #13 0x0000ffffb38cef60 in start_thread (arg=0x0) at pthread_create.c:442 #14 0x0000ffffb393489c in thread_start () at …/sysdeps/unix/sysv/linux/aarch64/clone.S:79
QEMU version is tag v6.2.0. Cmdline: build/qemu-system-aarch64 -m 8G -object memory-backend-memfd,id=mem1,size=8G,share=on -numa node,memdev=mem1 -machine virt,virtualization=off,its=off,gic-version=host -accel kvm -cpu host -smp 8 -bios ./QEMU_EFI.fd -device nec-usb-xhci -device usb-kbd -device usb-tablet -object iothread,id=disk-iothread -device virtio-blk-pci,drive=win11,iothread=disk-iothread,disable-legacy=on -drive if=none,id=win11,format=qcow2,file=win11.qcow2,cache=unsafe -device ramfb -vnc :0 -serial stdio -monitor telnet:localhost:4444,server,nowait -qmp unix:/home/qemu.sock,server,nowait
I set 6 for TimeoutValue in step 4. I don’t add IoTimeoutValue/IoLatencyCap for step 5-7.
The patch for test the qemu: