freebsd-wifibox: wlan0 interface inside VM no longer comes up after a few times doing suspend/resume

Hi, I just bought an X1 Carbon Gen 7 and managed to configure wifibox. Only thing left was the suspend / resume issue. The only setting that works for me is the following one, based on cthe preceding omment h(ttps://github.com//issues/31#issuecomment-1386305573):

(/usr/local/etc/devd/wifibox.conf)

notify 11 {
		match "system" "ACPI";
		match "subsystem" "Suspend";
		action "logger 'Stopping wifibox before suspend' && /usr/local/sbin/wifibox stop guest && /etc/rc.suspend acpi $notify";
};

notify 11 {
		match "system" "ACPI";
		match "subsystem" "Resume";
		action "/etc/rc.resume acpi $notify && logger 'Starting wifibox after resume and getting IP via DHCP' && /sbin/kldunload vmm && /sbin/kldload vmm && /usr/local/sbin/wifibox start guest && /sbin/dhclient wifibox0";
};

Thanks to your devd edits, my ghostbsd thinkpad is reconnecting at waking from sleep.

Hi, I just bought an X1 Carbon Gen 7 and managed to configure wifibox. Only thing left was the suspend / resume issue. The only setting that works for me is the following one, based on cthe preceding omment h(ttps://github.com/pgj/freebsd-wifibox/issues/31#issuecomment-1386305573):

(/usr/local/etc/devd/wifibox.conf)

notify 11 {
		match "system" "ACPI";
		match "subsystem" "Suspend";
		action "logger 'Stopping wifibox before suspend' && /usr/local/sbin/wifibox stop guest && /etc/rc.suspend acpi $notify";
};

notify 11 {
		match "system" "ACPI";
		match "subsystem" "Resume";
		action "/etc/rc.resume acpi $notify && logger 'Starting wifibox after resume and getting IP via DHCP' && /sbin/kldunload vmm && /sbin/kldload vmm && /usr/local/sbin/wifibox start guest && /sbin/dhclient wifibox0";
};

I ran a few more experiments making just the wifibox suspend go before the suspend notify. If I do this, the system is able to survive a few suspend/resumes but it is non deterministic. Usually within 2-5 resumes it will not be able to bring up the interface anymore. Same applies if I do the full stop and start of wifibox. I started looking a little bit at the latest code to see if I can narrow it down. At the moment I’m looking at the differences between the G and GN flags and also what was happening in the 0.10.0 release.

@pgj That’s correct, HEAD here refers to main. Primarily since there isn’t an official released port containing the new changes, I’ll want to clone it and then do my testing from that side.

I’ve switched from my Thinkpad X1C7 back to my Framework Laptop (Batch 6). I was able to reproduce the same issue on this machine, and my workaround also worked as well.

Small update on the wireguard routing situation. My initial configuration was correct (regarding slightly shifting the wifibox addresses from their default of 10.0.0.0/24 -> 10.1.0.0/24), however, I had forgotten that on my server, when I re-architected it to use jails, I locked down the host ListenAddress to 192.168.1.100, so that meant the previous 10.0.0.1 address that is the IP for that machine on my wireguard network, was no longer listened to. I added ListenAddress 10.0.0.1 in addition to the 192.168.1.100 and it’s all working. Wifibox has no issues communicating with any of my 192.168.1.X services and doesn’t have issues communicating with 10.0.0.X wireguard VPN services as well 😄.

I’ve opened up this ticket and emailed the freebsd-virtualization mailing list.

I’ll be experimenting with your suggestions regarding the restarting only the guest and not needing to run dhclient.

I’ve automated the suspend/resume for stopping/starting/assigning ip of wifibox and it seems to be working (I’ll need more time to test of course). I added the suspend section (not sure if the notify number is correct but it seems to be functioning correctly when doing multiple suspend/resumes):

root@leslie:~ # cat /usr/local/etc/devd/wifibox.conf

# This is a `devd(8)` configuration file to run the resume action of
# wifibox on the ACPI resume event.  Review the contents and create a
# copy of it without the `.sample` extension to use it.  Restart the
# `devd` service once the file has been created.

notify 11 {
        match "system"          "ACPI";
        match "subsystem"       "Suspend";
        action "logger 'Stopping wifibox before suspend' && /usr/local/sbin/wifibox stop && /etc/rc.suspend acpi $notify";
};

notify 11 {
        match "system"          "ACPI";
        match "subsystem"       "Resume";
        action "/etc/rc.resume acpi $notify && logger 'Starting wifibox after resume and getting IP via DHCP' && /usr/local/sbin/wifibox start && /sbin/dhclient wifibox0";
};

and this is how the /var/log/messages | grep wifibox looks like after a suspend/resume cycle:

Jul 28 14:14:08 leslie root[3910]: Stopping wifibox before suspend
Jul 28 14:14:11 leslie kernel: wifibox0: link state changed to DOWN
Jul 28 14:14:12 leslie dhclient[3177]: receive_packet failed on wifibox0: Device not configured
Jul 28 14:14:12 leslie dhclient[3177]: ioctl(SIOCGIFFLAGS) on wifibox0: Operation not permitted
Jul 28 14:14:12 leslie dhclient[3177]: Interface wifibox0 no longer appears valid.
Jul 28 14:14:38 leslie root[4098]: Starting wifibox after resume and getting IP via DHCP
Jul 28 14:14:38 leslie kernel: bridge0: changing name to 'wifibox0'
Jul 28 14:14:38 leslie kernel: wifibox0: link state changed to DOWN
Jul 28 14:14:38 leslie kernel: wifibox0: link state changed to UP
Jul 28 14:14:55 leslie dhclient[4412]: New IP Address (wifibox0): 10.1.0.2
Jul 28 14:14:55 leslie dhclient[4416]: New Subnet Mask (wifibox0): 255.255.255.0
Jul 28 14:14:55 leslie dhclient[4420]: New Broadcast Address (wifibox0): 10.1.0.255
Jul 28 14:14:55 leslie dhclient[4424]: New Routers (wifibox0): 10.1.0.1

Notice that the /etc/rc.suspend acpi $notify call happens at the end of our action event. I originally tried it at the beginning but when I resumed the PCI passthrough issue immediately popped up. I’m guessing is because since I called suspend before, the system didn’t have enough time to finish stopping the wifibox VM, so we got the call, but not a clean shutdown of the VM, thus triggering my theory of the PCI devices not being successfully cleaned up when the VM is running. When I flipped it and put the /etc/rc.suspend call at the end, I noticed that my sleep took a little bit longer but that would mean that it was just cleanly shutting down the VM before sleeping. It also didn’t matter if I ran zzz as root or as the user.

Lastly, there was one time I resumed where one of the machine lights lit up but it actually never resumed (The power button still was flashing as if it was sleeping, but the F1 (Mute Key Light) was on - which only happens when the machine is powered back on). I don’t necessarily think this is related to the above changes since I’ve experienced these sort of things on different machines before. It may just be because I’m sleeping and resuming multiple times in relatively quick succession (10 seconds ~ maybe more maybe less)… but I wanted to mention this for the record just in case.

Let me know what you think 😃.

@pgj I’ve noticed (still need more testing though) a possible workaround to this issue. It seems that if I stop wifibox before I put the computer to sleep, it will allow the machine to resume and still keep the PCI interface in the VM. This makes me believe that there is some resources not cleanly being handled when a zzz operation is triggered with an actively running bhyve VM using a PCI passthrough device.

I’ve also updated to FreeBSD 13.1-STABLE #0 stable/13-n251972-fb8e858c69f, no major changes but just wanted to keep you up to date.

This is the latest dmesg output (with me suspend/resume but manually stopping the wifibox before I run zzz):

messages-2.txt

These may be of interest (in a correct flow):

bridge0: Ethernet address: 58:9c:fc:10:ff:99
bridge0: changing name to 'wifibox0'
tap0: Ethernet address: 58:9c:fc:10:c0:3f
tap0: promiscuous mode enabled
wifibox0: link state changed to DOWN
ppt0 mem 0xea238000-0xea23bfff at device 20.3 on pci0
tap0: link state changed to UP
wifibox0: link state changed to UP
lo0: link state changed to UP


...

tap0: link state changed to DOWN
wifibox0: link state changed to DOWN
ppt0: detached
pci0: <network> at device 20.3 (no driver attached)
ue0: link state changed to UP
tap0: promiscuous mode disabled

...

ppt0: detached
pci0: <network> at device 20.3 (no driver attached)
ppt0 mem 0xea238000-0xea23bfff at device 20.3 on pci0
bridge0: Ethernet address: 58:9c:fc:10:ff:99
bridge0: changing name to 'wifibox0'
tap0: Ethernet address: 58:9c:fc:10:c0:3f
tap0: promiscuous mode enabled
wifibox0: link state changed to DOWN
tap0: link state changed to UP
wifibox0: link state changed to UP
tap0: link state changed to DOWN
wifibox0: link state changed to DOWN
ppt0: detached
pci0: <network> at device 20.3 (no driver attached)
ppt0 mem 0xea238000-0xea23bfff at device 20.3 on pci0
tap0: link state changed to UP
wifibox0: link state changed to UP
tap0: link state changed to DOWN
wifibox0: link state changed to DOWN
ppt0: detached
pci0: <network> at device 20.3 (no driver attached)
tap0: promiscuous mode disabled

I’ll be testing this for a few days to see what happens and report back. I’ll probably add a little bit of automation (example, when I suspend my screen via shortcuts on i3, I’ll do a wifibox stop && zzz command, and then on resume I’ll have the devfs resume event do a wifibox start && dhclient wifibox0). I’ll let you know what happens.

@pgj I compiled 13.1-RELEASE from source (releng/13.1) and re-tested my steps. I was still able to reproduce it on:

FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC amd64

For a second I thought that there may have been some weird behavior happening between my Anker Dongle (HDMI/Ethernet/USB Ports/SD/Micro SD/Power all over Type C) and my Thinkpad (maybe something about going to sleep and resuming with all of that attached). However, after about 5-6 consecutive working sleep/resume, it finally dropped the 00:06.0 Class 0280: 8086:02f0 slot from the VM. So it seems the same issue has been there and I don’t think there is a regression, just a bug somewhere.

I’ve attached a longer /var/log/messages below (contains -STABLE and then -RELEASE logs). Things related to ppt0 mem 0xea238000-0xea23bfff at device 20.3 on pci0 seem interesting.

messages-1.txt

@pgj Sounds good. I guess I should hope it fails for you too on STABLE and it isn’t just me 😂.

And yea, you could checkout 488f9d85278 (n251926) on the stable/13 branch.