go: cmd/compile, runtime: investigate Windows stack overflows calling into system C libraries
This came up with 1.8.3 when working on GUI Windows programs using lxn\walk library.
Relevant issues:
The problem is that Go linker sets very small stack size (I think 128 kB) in PE executables. The standard on Windows is more like 1-2 MB.
This is fine for code that only lightly uses system C libraries but when writing code that talks to win32 UI APIs, it’s very likely to hit the stack limit and silently crash.
I encountered it because I tried to use webview (mshtml.dll) control and it crashed on 64-bit when rendering my (not very complicated) website. Other people seen such crashes as well.
There are work-arounds: one can edit PE header after the exe is built using e.g. editbin.exe , but such tool is not necessarily available to the programmer (it’s part of Visual Studio).
It would be much easier on Windows developers if there was a linker flag to set custom stack size so that it could be done directly with go build. A library like lxn\walk could then document the need to increase stack size for Go Windows programs and recommend a
I don’t know if such option would be relevant/needed for other OSes/exe formats.
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 4
- Comments: 60 (27 by maintainers)
Commits related to this issue
- runtime: always use 2MB stacks on 64-bit Windows Currently, Windows stacks are either 128kB or 2MB depending on whether the binary uses cgo. This is because we assume that Go system stacks and the sm... — committed to golang/go by aclements 7 years ago
- runtime: always use 1MB stacks on 32-bit Windows Commit c2c07c7989 (CL 49331) changed the linker and runtime to always use 2MB stacks on 64-bit Windows. This is the corresponding change to make 32-bi... — committed to golang/go by aclements 7 years ago
- runtime: query thread stack size from OS on Windows Currently, on Windows, the thread stack size is set or assumed in many different places. In non-cgo binaries, both the Go linker and the runtime ha... — committed to golang/go by aclements 6 years ago
I don’t think 512 kb is enough. It’s where the simplest program stopped crashing on the simplest website. I haven’t done much testing beyond that but it stands to reason other websites (or other programs) might require more than 512 kb.
I think the value should be what the standard for windows linker is i.e. 1 MB (https://msdn.microsoft.com/en-us/library/windows/desktop/ms686774(v=vs.85).aspx), which seems to be the same for 32bit and 64bit (https://blogs.technet.microsoft.com/markrussinovich/2009/07/05/pushing-the-limits-of-windows-processes-and-threads/).
Less than that and Go code will run into this crash more often than C code.
And I would still like the linker option to bump it to larger value if I determine that my particular application needs that, especially on 64bit where larger reserved stack has essentially no penalty because address space is aplenty.
I get the desire that make it “just work” but you can’t in this particular case.
I had quick look at how we set reserved stack size.
For Cgo program, it is 1M for 386 and 2M for amd64. Search for oh64.SizeOfStackReserve and oh.SizeOfStackReserve in cmd/link/internal/ld/pe.go for first thread. All other threads start with
in runtime/cgo/gcc_windows_386.c and runtime/cgo/gcc_windows_amd64.c, and _beginthread defaults to what first thread does.
For non-Cgo program, it is 128K. Search for oh64.SizeOfStackReserve and oh.SizeOfStackReserve in cmd/link/internal/ld/pe.go for first thread. All other threads start with
So Cgo programs run with standard (by Windows standards) stacks. But non-Cgo programs run with small stacks. CL 2237 doubled the stack size (from 64K to 128K) already. @kjk can you, please, try make it 256K for non-Cgo - see if it fixes your problem. Maybe we can increase stack size again. I don’t think it matters on amd64, and we have less and less of 386 computers.
Thank you
Alex
The discussion in https://github.com/lxn/walk/issues/261 claims that adding a dummy
import _ "runtime/cgo"to the program would force the large stack but it doesn’t seem to be the case in my testing. Maybe that changed sometime before 1.8.3.The crash always happen on the main thread.
If you read earlier comments I think I narrowed it down to
morestackcrashing because it detects we’re on scheduler stack even though it’s being called viasystemstack()whose purpose seems to be ensuring that doesn’t happen.So my best guess is that
systemstackon amd64 doesn’t always do the job (the issue doesn’t happen on i386 which is not unexpected because it’s different assembly code).I would not bother. Once CL 49610 is submitted, Go stacks are the same as C programs compiled by Miscrosoft compilers. It is good enough.
Alex
CL https://golang.org/cl/49610 mentions this issue.
There is no doubt that I can. I’ll just allocate more than 512 kb on the stack.
I broke 256 kb on literally one of the first real programs I wrote that used mshtml. There’s infinite number of websites and infinite number of programs that use other potentially stack hungry libraries.
If I’m intentional about it then I’ll break any limit. If I’m not intentional and just writing random windows programs then I have infinite search space.
The question is: what is a reasonably safe value.
And the answer is: what everyone else is using i.e. 1 MB.
Those are the constraints under which Window OS developers and other WIndows developers operate. If they write code that uses more than 1 MB of stack, their programs will crash, so they have incentive to stay under that limit.
The lower the limit the higher probability that Go programs will crash because they’ll hit the limit.
I’ll also note that there might more bad going on than just stack size limit. When I use the repro program I linked to and visit https://vox.com, it crashes even if I set very large limit, like 2 MB. Sometimes it prints “fatal: more gc” sometimes it doesn’t and it crashes it a very bad way (not an orderly panic that prints a crashing callstack but a vanishing act where process is wiped out without a trace).
Unfortunately delve doesn’t debug this kind of code at all and windbg doesn’t understand Go symbols and I know nothing about debugging runtime so that’s as deep as I can go on this.
But I do have a consistent repro of a bad crash.
It is plenty for me. But other people might think differently.
I really do not know. I will let @aclements decide.
GUI does not have to run on the main thread. You can even run multiple GUI threads.
I think these different thread sizes for different environments are already complicated enough. I do not want to make things even more complicated.
Alex
1000 real OS threads on 32 bit windows should be more than enough for anybody. Any program with that many threads would trash the CPU anyway - there’s really no point launching more than ~NUMCPU os threads.
It’s really mostly about main GUI thread, which is the thread pumping the message loop which ends up calling wndproc of windows and the code called from there.
That’s the main thread that is created outside of control of Go runtime and uses the stack size from PE header.
I imagine other threads created by the Go runtime can have a stack size independent of PE value (given to CreateThread) and there’s argument that those threads are not expected to have such heavy nesting as main GUI thread so could get away with less stack.
For easy reproduction, I’ve created https://github.com/kjk/go20975
For what it’s worth, here’s a callstack of the crash. It’s on main thread:
Go code doing the windows message pump https://github.com/lxn/walk/blob/master/form.go#L336