deno: thread 'main' has overflowed its stack when file has over 250 nested if statements
Hi. I’m trying to get my library to support deno.
For reasons I don’t understand, when I import my library in deno, it overflows the stack. It’s fine in the browser, though, or in node if you use dynamic import and change the extension.
Also happens with direct binding (that is, no * as.)
It’s immediate and there are no apparent diagnostics.
I recognize that I might sound a little extreme when I say this, but I’m nervously concerned there might be a portability bug in deno. If not, alternately, there might be a resource ceiling on the stack that is significantly lower than in browsers and node. There are a number of other people claiming this is happening, and every single one of them is a Windows user. See #13196, #11356, #9752, and #5848.
Particularly interestingly, #9752 appears to treat this as a bug that was fixed upstream seven months ago.
On the other hand, there’s a solid chance that I’m doing it wrong and making some stupid mistake I just don’t understand.
You can see a reproduction, if you’re interested, by pulling the branch ReattemptDenoSupport from https://github.com/StoneCypher/jssm/tree/ReattemptDenoSupport and running npm install && npm run build, then opening deno and issuing import * as jssm from "./dist/deno/jssm.deno-esm.js".
Sorry, I know it’s not cool to put npm instructions in here. I’m trying to fix it to be on deno.land and I’ll update this as soon as I get what’s going on. It’s hard to know if I’m using deno.land correctly when I haven’t got my library working as expected.
How does someone go about diagnosing something like this?
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 30 (8 by maintainers)
I may have generated a reproduction for this issue, see https://github.com/swc-project/swc/issues/6813.
We already increase the stack size on Windows but only for
debugbuilds. Maybe we should do this onreleasebuilds too.https://github.com/denoland/deno/blob/17271532d4dcf8dfee1c7b1ac9dbdacb0a04deeb/.cargo/config.toml#L9-L10
Actionable items:
While fixing swc to not use this much stack for such parsers is the arguable correct fix, I believe Deno can can avoid this (and other stack related crashes) on Windows pretty easily.
On Windows the main thread has a maximum stack size of 1 MiB by default, while Linux and MacOS will let main thread to grow up to 8MiB by default (controlled by ulimit). There are some differences with respect to spawning other threads on each platform, but all let you specify a custom maximum stack size. Threads spawned by Rust code will use a maximum stack size of 2MiB by default, regardless of the OS default, but this does not apply to the main thread.
Increasing the default stack size on Windows is possible as a linker flag. Increasing the default to 8MiB would bring Windows in line with Linux and MacOS’s default ulimit and would not be unreasonable. While window’s strict memory accounting often makes it act differently than Linux, for this purpose Windows actually acts more like most Linux allocations do (explained later), and thus this bigger size will not make the program use more memory, unless/until the stack is used enough to grow to that size.
The only real downside here is that changing the default stack size will also change the default maximum stack size for threads created by non-rust code, but that really is not an issue, as it is not memory but only reserved virtual address space. For x64 this really only means a few extra page table entries are created per natively created thread. In a 32 bit program the extra address space reserved could potentially be a concern, but Deno only supports x64 on windows anyway, and even if it did, it would need many dozens of threads to realistically be an issue.
Lastly Deno is already increasing the stack size for debug builds on Windows (see
.cargo/config.toml), so switching to an 8MiB default stack reservation size for all Windows builds instead of only debug builds is actually a simplification, and would avoid a bunch of cases where things can crash from stack limits on Windows while working on Linux and MacOS.More about how stack works on Windows:
While Windows’ strict memory accounting normally makes it work differently from Linux, for stack size windows actually behaves similarly to most allocations in Linux, where a larger maximum stack size does not make it use more memory unless the stack grows that big.
This is because Windows only reserves the address space, for the full stack, but does not initially map it to memory (which requires “committing” it). Instead it commits memory as the stack grows, which does mean that the program can crash with a stack overflow before reaching the max stack size, but only if Windows does not have enough physical memory plus swap to allow for more committed memory. To allow programs to be sure they won’t crash from not enough stack, Windows offers a second value that determines the minimum committed stack size.
When creating other threads on windows, you can specify a minimum committed stack size to override the default, and if this minimum is greater than the default maximum, the thread will be created with a larger maximum to accommodate. You can instead use the
STACK_SIZE_PARAM_IS_A_RESERVATIONflag when creating a thread to set the maximum stack size for a new thread, without forcing that memory to be fully committed. (Rust does this when creating threads.)Hope this helps.
We use a Rust library to do dependency analysis on code, which parses the JavaScript, and appears to have challenges with deeply nested if statements. So while V8 can run this, the dependency analysis cannot parse it without overflowing the stack. It is arguable that it is an issue with swc, as while somewhat incredible, there are clearly instances of >250 nested if statements in the wild, as well as the behaviour is different between windows and other situations. Ideally swc shouldn’t overflow the stack in such an unsafe way.