kvm-guest-drivers-windows: NetKVM: Windows Server 2025 VM hang w/ NdisPoll enabled
Describe the bug
On Alibaba Cloud Elastic Compute Service, a Windows Server 2025 VM w/ latest NetKVM driver will hang, due to NdisPoll feature, which was enabled by default in 2k25 variant.
If the driver was slipstreamed into sysprep-ed image, it will hang during specialize pass (showing Getting ready or Getting devices ready).
If the driver was manually installed via devmgmt.msc, it will hang immediately and Windows cannot boot normally. Modifying NdisPoll to 0 in registry allow Windows to boot normally and NIC works properly.
The 2k22 variant from the same build does not have any problem.
To Reproduce Steps to reproduce the behaviour:
- Create an Alibaba Cloud ECS instance w/ Windows Server 2025
- Update NIC driver to build
0.1.262-2or later, choose2k25variant - Windows hang immediately and cannot boot normally afterwards
Expected behavior If the VM is not compatible with NDIS polling mode, this should be automatically detected and work without NDIS polling.
VM:
- Windows Server 2025 (Build
10.0.26100.3194) - NetKVM
2k25variant - build
0.1.262-2
Additional context
Alibaba Cloud has a proprietary virtio-net-pci implementation. It seems to be quite legacy and bizarre (no VIRTIO_F_VERSION_1, a legacy configuration structure in memory-mapped BAR0).
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 67 (4 by maintainers)
@ybendito Okay, I didn’t notice the comment above before. I investigated this issue with our colleagues from the hypervisor team and found that it is a bug in the Ali hypervisor (for certain specifications). There is a logical bug related to the new RX layout that causes the tail part to be treated as the next packet. In other words, the latest code in the master branch of the NetKVM driver is fine @zhangyoufu
@ybendito @zhangyoufu Hello. I have made progress on this issue(tx=0bytes). I have pinpointed it to the introduction of a new RX memory layout. This means that if we revert the commit 886047541e9a890d593bbbc97e2a07bb1644cb1b, the issue disappears. It is worth noting that the problems caused by the new layout are not only present in 2025 but also in other systems such as Server 2019, and there are issues in certain specifications on Alibaba Cloud (different specifications have varied backend implementations). However, I still need to spend some time understanding why the new layout causes this issue. I will update you both later."
@ybendito Okay, thank you for the reminder. I have currently reproduced the situation of tx=0 bytes using the latest version of master on Alibaba Cloud
@ybendito rc11 driver is working perfectly on AliCloud VM. The 30s delay in rc10 has gone.
@zhangyoufu I succeeded to reproduce the third issue on qemu. I think this does not depend on qemu but on the fact that on the AliCloud-VM you have Wireshark or NPCAP driver installed. With settings you mentioned I also get BSOD in ndisDisableUdpRsc. I did not dig too deep but it looks like there is some incompatibility between NPCAP and NDIS drivers supporting NDIS 6.89 (Win10 driver declares 6.85). Need to investigate more.
@zhangyoufu I succeeded to reproduce the first issue on qemu. The first difference between AliCloud VM and QEMU VM is that on QEMU you use virtio-net with “-netdev user” with 1 queue. I’m not sure whether it supports multiple queues (I never use it), I supprose it does not. I’ve succeeded to reproduce the problem on 2025 build 3476 with “tap” netdev, 2 queues, 2 cpus and virtio-net-pci with mq=on. The rc1 fix works. I’m not able yet to reproduce 2 other problems (TX and BSOD).
@zhangyoufu Now we know which commits cause the problem, but we do not know what the problem is. The commits are https://github.com/virtio-win/kvm-guest-drivers-windows/commit/f5e5b234c6bd0b462e5a3d1a85ac99bc46a10f62 (that creates 30 sec timeout somewhere in the OS) and https://github.com/virtio-win/kvm-guest-drivers-windows/commit/886047541e9a890d593bbbc97e2a07bb1644cb1b (that somehow makes TX to stop working). Both commits are related to RX path and according to all the test did not raise any problem with typical qemu builds. For educated decision it would be good to try reproducing the problem on some old qemu build (like 2.12/2.11), probably using the old Fedora/Ubuntu release. If we succeed to reproduce it on something other than proprietary ALI hypervisor, we can understand the problem and find a best way to solve (in driver or in hypervisor). What do you think about it?
@ybendito
@zhangyoufu Reproduction of hang on QEMU with 266w11, Ndis Poll enabled: