wazero: Programs that succeed to execute with v1.6.0 sometimes hang indefinitely with v1.8.1 during instantiation.

Describe the bug Programs that succeed to execute with v1.6.0 sometimes hang indefinitely with v1.8.1 during instantiation. The same program crashes with v1.7.x

To Reproduce Clone the example repository:

git clone https://github.com/davidmdm/wzbug.git
cd ./wzbug
./test.sh

Expected behavior Program should behave the same as with the v1.6.0 runtime.

Environment (please complete the relevant information):

  • Go version: 1.23.3
  • wazero Version: v1.8.1
  • Host architecture: Arm64 (M2 Mac)
  • Runtime mode: Compiler

Additional context

Incredibly rare and difficult to reproduce. If I tweak the example program slightly, the bug disappears. If I don’t use stdin the bug disappears. However I don’t want to write programs fearing that the perfect storm will hang my execution.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 29 (29 by maintainers)

Most upvoted comments

func encodeCallTrampolineIsland(refToBinaryOffset []int, islandOffset int, executable []byte) {
	for i := 0; i < len(refToBinaryOffset); i++ {
		trampolineOffset := islandOffset + trampolineCallSize*i

this function is given all of the function offsets (refToBinaryOffset), len is 8404, i.e. 168_080 bytes, causing it to overwrite the start of the next function (which has been already written to executable).

The issue is that refToBinaryOffset is indexed with the Wasm index, that includes all imported functions, thus the size is importedFn+localFn, while here it is only given the localFns

	trampolineInterval, callTrampolineIslandSize, err := machine.CallTrampolineIslandInfo(localFns)

if we fix the count in the loop above (in this case, we just len(refToBinaryOffset)-27, but any number >27 works, since it’s not used anyway lol) then the bug is resolved, or, we could also:

	trampolineInterval, callTrampolineIslandSize, err := machine.CallTrampolineIslandInfo(localFns+importedFns)

Anyway:

indeed, Bryan, indeed

diff --git a/internal/engine/wazevo/engine.go b/internal/engine/wazevo/engine.go
index f02b905f..2dd180ad 100644
--- a/internal/engine/wazevo/engine.go
+++ b/internal/engine/wazevo/engine.go
@@ -283,7 +283,7 @@ func (e *engine) compileModule(ctx context.Context, module *wasm.Module, listene

                if needCallTrampoline {
                        // If the total size exceeds the trampoline interval, we need to add a trampoline island.
-                       if totalSize/trampolineInterval > len(callTrampolineIslandOffsets) {
+                       if totalSize/trampolineInterval >= len(callTrampolineIslandOffsets) {
                                callTrampolineIslandOffsets = append(callTrampolineIslandOffsets, totalSize)
                                totalSize += callTrampolineIslandSize
                        }

thanks @ncruces for the help brainstorming.

I think this is it, but I am too tired to verify with proper tests @mathetake calling it a day 👋 but the repro is definitely fixed 🎉

WELL WELL WELL WE MEET AGAIN #2158 Not 100% sure TBH 😅, but if I change the maxUnconditionalBranchOffset to e.g. 1<<26; i.e. (1<<24)*4 instead of (1<<25 - 1)*4 I no longer get the issue, so I suspect it is still an integer overflow related to

https://github.com/tetratelabs/wazero/blob/d25ce10a01152b4c5f76c56b0c4a18ec5fa25cf1/internal/engine/wazevo/backend/isa/arm64/machine_relocation.go#L56-L58

@evacchi,

I just ran it using the interpreter and it ran successfully in all versions that I am testing: 1.6.0, 1.7.3, and 1.8.1.

Compiler it is?

Glad to be of service! Maybe I can be given an honorary QA position for this miracle? 😆

can’t believe we actually hit the corner case though - it sounds like almost miracle happening lol

yeah, but it’s not necessarily the module size, but possibly the size of the single function 😃 e.g. there is a function that apparently is 78859 bytes and if I read it right, it gets trimmed down to 71014 bytes

I also pushed a new branch to the repository called segfault.

I modified the example program in what should be harmless ways… Just removing certain fields from the structs that I was going to json.Marshal, and this had the effect that both 1.6.0 and 1.7.3 worked as expected but 1.8.1 went from hanging to a segfault.

I am unsure what about my code is cursed, but hopefully it helps find the issue?

@evacchi here is an incomplete list of weird things which have made the program succeed.

  • using json.Unmarshal instead of K8s/yaml Decoder
  • hard coding the data instead of parsing it from stdin
  • parsing the data as is but removing any further code related to json encoding of the various k8 APIs

However if I parse stdin using the yaml decoder from k8s and use that data it hangs.