v86: Kernel hang with CONFIG_DEBUG_NMI_SELFTEST (Alpine Linux)

• Compile a kernel without the nmi selftest

I recompiled Alpine Linux edge kernel without CONFIG_DEBUG_NMI_SELFTEST (virt: iso, package, lts: package), and it’s worked in Firefox 122.0.1 (Windows) on non-https copy.sh/v86:

Notes:

• To install kernel to already installed system: apk add --allow-untrusted ./linux-<virt/lts>-6.6.16-r0.apk
• Networking (ne2k-pci) and sound (sb16) working only on linux-lts, because linux-virt don’t have drivers
• Instruction for building custom kernel: https://wiki.alpinelinux.org/wiki/Custom_Kernel (compiles lts and virt once together, needed installed Alpine Linux and 20GB+ storage)

It’s used roughly three places:

• rdtsc returns it directly (multiplied by TSC_RATE): https://github.com/copy/v86/blob/21cf9ad/src/rust/cpu/cpu.rs#L3966
• main_loop uses it to yield to the browser (for redrawing and receiving mouse/keyboard/fetch events) every 1ms: https://github.com/copy/v86/blob/21cf9ad/src/rust/cpu/cpu.rs#L3056 This could explain why 16ms is much slower, but not why 1ms is slower. We could switch to yielding after a fixed number of cycles (at least as an option). Note that every yield has a fair overhead (there’s no good browser api, see https://github.com/copy/v86/blob/21cf9ad/src/main.js#L98) and cycles per second aren’t stable
• To run the pic and rtc. This condition is probably never going to return true on 16ms (and possibly on 1ms), but I don’t think it’s used by Linux. The pit typically runs at 10ms (configurable by the guest), so that doesn’t really explain any performance differences. The logic looks fine to me, even for 16ms.

setTimeout might also be affected. v86 uses setTimeout instead of postMessage when the cpu is executing HLT and there are no timer interrupts coming up. If setTimeout is throttled to 16ms, it may sleep much longer than intended.

To narrow it down, I’d suggest:

• Replacing microtick with a throttled version (function throttled_microtick() { return 16.67 * Math.round(microtick() / 16.67); }) in selected places
• Disabling idling by commenting out setTimeout here

You can run this code to determine the resolution of performance.now(): x = new Set(); while(x.size < 5) x.add(performance.now()); Array.from(x).sort((a,b) => a-b).slice(-2).reduce((a,b) => b-a)

On Firefex (developer edition), I get 1ms on http sites and 0.02ms on https (with COOP/COEP).

The NMI test suite calls udelay 1000000 times: https://github.com/torvalds/linux/blob/28b8235/arch/x86/kernel/nmi_selftest.c#L82-L83 udelay on x86 is here: https://github.com/torvalds/linux/blob/28b8235/arch/x86/lib/delay.c#L207 udelay will repeatedly call rdtsc until it changes (some amount that should amount to 1us), but with the reduced resolution this will take (worst case) 16 minutes (1ms * 1000000) on Firefox on http and 20 seconds on https (0.02ms * 1000000). I can experimentally confirm the 20 second delay.

On Chromium, the resolution is higher (0.1ms and 0.005ms respectively). Maybe your Firefox has a different resolution or doesn’t accept the COOP/COEP headers on https://copy.sh?

Now regarding fixes, there are a couple of options:

• Implement NMIs in the apic (so the test passes instead of timing out). Currently apic support is experimental and doesn’t boot Alpine at all. In any case, it would be nice if the timer issue could be fixed properly, as other OSes might also be affected (I believe KolibriOS has a similar loop in its boot process).
• Do nothing and wait until Firefox gets a better timer resolution
• Hack rdtsc to return higher values if called multiple times and the browser time hasn’t changed. There is some prototype that was enabled a long time ago (see https://github.com/copy/v86/blob/bfb59f9/src/rust/cpu/cpu.rs#L3972). However, this would mess with other uses of rdtsc. Maybe behind a flag.
• Implement tpause (although it’s not entirely clear how, if the timer resolution sucks)
• Change cpuid flags so that Linux stops using the tsc and uses the default delay_loop instead. In my experiments, setting cpuid_level to 0x14 makes Linux try to calibrate the tsc against the pic and fail (“Marking TSC unstable due to could not calculate TSC khz”), at least on Firefox where the timer resolution is 1ms
• Compile a kernel without the nmi selftest

Or, since this is not really v86’s fault:

• Ask Firefox developers to improve their timer resolution (do two browsers really need four different values?)
• Send a patch to Alpine to disable the NMI selftest
• Send a patch to Linux to only run the NMI selftest on systems that have an apic

When I create an image using the latest Alpine Virtual 3.19.1 it outputs a message similar to “watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [swapper/0:1]” from the OP over and over with the seconds number growing each time and never gets to anything usable. Maybe I missed something?

@spetterman66 also seemed to confirm that only old versions work in an earlier message…