wasmtime: macos: "failed to set thread exception port" in forked process
Hello,
Avid Wasmtime users in our Nginx embedding, we are facing an assertion failure when running it with Wasmtime on macos (x86 and arm64):
thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
left: `268435466`,
right: `0`: failed to set thread exception port', crates/runtime/src/traphandlers/macos.rs:474:9
This assertion systematically fails with Wasmtime 8.0.1 (and probably earlier) to 11.0.1 when Nginx is configured with daemon on (which ends here in Nginx forking into a background process). It seems it occurs in our call to wasmtime_linker_instantiate after having initialized an engine and a single store.
Not forking the master process (daemon off) seems to be working fine, even in forked worker processes (managed by the master process in foreground); instances are created and behave as expected in the Nginx worker processes.
The assertion failure specifically occurs once the master process has forked itself into a background process with daemon on.
In summary:
- Works with
daemon off: master process (foreground) ->wasmtime_linker_instantiate(temporary instance) -> fork() -> worker processes ->wasmtime_linker_instantiate(worker instances processing requests). - Fails with
daemon on: master process (foreground) -> fork() (background daemon) ->wasmtime_linker_instantiate(temporary instance) -> assertion failure, no instance created, and no chance to fork() into worker processes.
In the past we used to have a CI/CD pipeline with macos targets and Wasmtime used to work fine; but this CI/CD pipeline was taken down, and this bit us in the last few days. It seems like older macos work on Wasmtime has something to do with it, but I know nothing of the macos system interface…
So far the root cause of this assertion failure has eluded us; probably state that aren’t being carried over to the forked process or something like this. Could someone shed some light on what may be at cause here?
Thanks!
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 18 (17 by maintainers)
Commits related to this issue
- Fix posix-signals-on-macos on aarch64-apple-darwin Currently the `libc` crate has an incorrect definition of `ucontext_t` for this platform which is causing alignment issues when it's used. This fixe... — committed to alexcrichton/wasmtime by alexcrichton a year ago
- Fix posix-signals-on-macos on aarch64-apple-darwin (#6793) Currently the `libc` crate has an incorrect definition of `ucontext_t` for this platform which is causing alignment issues when it's used. T... — committed to bytecodealliance/wasmtime by alexcrichton a year ago
- Configure Mach ports vs signals via `Config` This commit adds a `Config::macos_use_mach_ports` configuration option to replace the old `posix-signals-on-macos` compile-time Cargo feature. This'll mak... — committed to alexcrichton/wasmtime by alexcrichton a year ago
- Configure Mach ports vs signals via `Config` (#6807) * Configure Mach ports vs signals via `Config` This commit adds a `Config::macos_use_mach_ports` configuration option to replace the old `posix-s... — committed to bytecodealliance/wasmtime by alexcrichton a year ago
- Fix size/align of `ucontext_t` on aarch64-apple-darwin This commit effectively reverts #2817. Currently `ucontext_t` has both the wrong size and the wrong alignment for aarch64-apple-darwin which cau... — committed to rust-lang/libc by alexcrichton a year ago
- Auto merge of #3312 - alexcrichton:fix-aarch64-darwin-ucontext-t, r=JohnTitor Fix size/align of `ucontext_t` on aarch64-apple-darwin This commit effectively reverts #2817. Currently `ucontext_t` has... — committed to rust-lang/libc by bors a year ago
- Auto merge of #3312 - alexcrichton:fix-aarch64-darwin-ucontext-t, r=JohnTitor Fix size/align of `ucontext_t` on aarch64-apple-darwin This commit effectively reverts #2817. Currently `ucontext_t` has... — committed to rust-lang/libc by bors 10 months ago
- Fix posix-signals-on-macos on aarch64-apple-darwin (#6793) Currently the `libc` crate has an incorrect definition of `ucontext_t` for this platform which is causing alignment issues when it's used. T... — committed to eduardomourar/wasmtime by alexcrichton a year ago
- Configure Mach ports vs signals via `Config` (#6807) * Configure Mach ports vs signals via `Config` This commit adds a `Config::macos_use_mach_ports` configuration option to replace the old `posix-s... — committed to eduardomourar/wasmtime by alexcrichton a year ago
- Fix size/align of `ucontext_t` on aarch64-apple-darwin This commit effectively reverts #2817. Currently `ucontext_t` has both the wrong size and the wrong alignment for aarch64-apple-darwin which cau... — committed to xevor11/libc by alexcrichton a year ago
If you don’t want to modify the source
--features wasmtime/posix-signals-on-macosmay also work.I think everything is now handled here with a combo of what I mentioned above:
pthread_atforkto fail-fast in child processes trying to do wasm thingsI’m going to close this now but @thibaultcha if there’s anything else please comment here!
Yeah, it appears to be missing from the
wasmtime-c-apicrate, just needs to forward the feature to thewasmtimecrate:crates/c-api/Cargo.toml: