wazero: Programs that succeed to execute with v1.6.0 sometimes hang indefinitely with v1.8.1 during instantiation.
Describe the bug Programs that succeed to execute with v1.6.0 sometimes hang indefinitely with v1.8.1 during instantiation. The same program crashes with v1.7.x
To Reproduce Clone the example repository:
git clone https://github.com/davidmdm/wzbug.git
cd ./wzbug
./test.sh
Expected behavior Program should behave the same as with the v1.6.0 runtime.
Environment (please complete the relevant information):
- Go version: 1.23.3
- wazero Version: v1.8.1
- Host architecture: Arm64 (M2 Mac)
- Runtime mode: Compiler
Additional context
Incredibly rare and difficult to reproduce. If I tweak the example program slightly, the bug disappears. If I don’t use stdin the bug disappears. However I don’t want to write programs fearing that the perfect storm will hang my execution.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 29 (29 by maintainers)
this function is given all of the function offsets (
refToBinaryOffset), len is 8404, i.e. 168_080 bytes, causing it to overwrite the start of the next function (which has been already written toexecutable).The issue is that
refToBinaryOffsetis indexed with the Wasm index, that includes all imported functions, thus the size is importedFn+localFn, while here it is only given thelocalFnsif we fix the count in the loop above (in this case, we just
len(refToBinaryOffset)-27, but any number >27 works, since it’s not used anyway lol) then the bug is resolved, or, we could also:Anyway:
indeed, Bryan, indeed
thanks @ncruces for the help brainstorming.
I think this is it, but I am too tired to verify with proper tests @mathetake calling it a day 👋 but the repro is definitely fixed 🎉
WELL WELL WELL WE MEET AGAIN #2158 Not 100% sure TBH 😅, but if I change the
maxUnconditionalBranchOffsetto e.g.1<<26; i.e.(1<<24)*4instead of(1<<25 - 1)*4I no longer get the issue, so I suspect it is still an integer overflow related tohttps://github.com/tetratelabs/wazero/blob/d25ce10a01152b4c5f76c56b0c4a18ec5fa25cf1/internal/engine/wazevo/backend/isa/arm64/machine_relocation.go#L56-L58
@evacchi,
I just ran it using the interpreter and it ran successfully in all versions that I am testing: 1.6.0, 1.7.3, and 1.8.1.
Compiler it is?
Glad to be of service! Maybe I can be given an honorary QA position for this miracle? 😆
can’t believe we actually hit the corner case though - it sounds like almost miracle happening lol
yeah, but it’s not necessarily the module size, but possibly the size of the single function 😃 e.g. there is a function that apparently is
78859 bytesand if I read it right, it gets trimmed down to71014 bytesI also pushed a new branch to the repository called
segfault.I modified the example program in what should be harmless ways… Just removing certain fields from the structs that I was going to json.Marshal, and this had the effect that both 1.6.0 and 1.7.3 worked as expected but 1.8.1 went from hanging to a segfault.
I am unsure what about my code is cursed, but hopefully it helps find the issue?
@evacchi here is an incomplete list of weird things which have made the program succeed.
However if I parse stdin using the yaml decoder from k8s and use that data it hangs.