wasm-pack: Segmentation fault after small patch

I’m seeing what’s probably the same segfault. It looks like a race condition within openssl triggered by exiting (running atexit handlers) while the thread spawned by background_check_for_updates is initializing openssl.

Even if openssl handled shutdown mid-initialization, I suspect you’d still be in trouble if the version check thread gets as far as actually using openssl while the main thread exits the process. It’d be nice if wasm-pack could wait for that other thread before exiting, but unless it can interrupt the check in flight I’m guessing that’d be a bit slow… Haven’t investigated what it would take to interrupt the check and exit cleanly.

I’m also seeing this with a locally built wasm-pack (from revision 30a95a42f14abb5b7d6e50f98256da7400046199, built with Rust 1.35.0, with a local change that shouldn’t matter if we exit this early). I haven’t investigated if official builds of wasm-pack are less crashy (or why that might be).

#653 (and any other changes that rate-limit the version check) might mostly hide this issue.

The exact stack of the crashing thread varies, but usually looks like:

Thread 1 (Thread 0x7f9ca167f700 (LWP 17184)):
#0  __pthread_rwlock_wrlock_full (abstime=0x0, rwlock=0x0) at pthread_rwlock_common.c:581
#1  __GI___pthread_rwlock_wrlock (rwlock=0x0) at pthread_rwlock_wrlock.c:27
#2  0x00007f6461bb6109 in CRYPTO_THREAD_write_lock (lock=<optimized out>) at crypto/threads_pthread.c:66
#3  0x00007f9ca196c245 in OBJ_NAME_add (name=0x7f9ca19e0500 "md5", type=type@entry=1, data=data@entry=0x7f9ca1a5ef20 <md5_md> "\004") at crypto/objects/o_names.c:230
...
#5  0x00007f9ca207aa4f in ossl_init_ssl_base () at ssl/ssl_init.c:77
...
#10 0x00007f646227abdb in OPENSSL_init_ssl (opts=2097152, settings=<optimized out>) at ssl/ssl_init.c:201
...
#12 0x000055af94bf51e0 in openssl_sys::init ()
#13 0x000055af94bf1eb5 in std::sync::once::Once::call_once::{{closure}} ()
#14 0x000055af94d4cf16 in call_inner () at src/libstd/sync/once.rs:387
...
#18 0x000055af94b5d2c3 in wasm_pack::manifest::Crate::return_wasm_pack_latest_version ()
...

There’s another thread that looks like:

Thread 2 (Thread 0x7f9ca1680980 (LWP 17183)):
#0  0x00007f9ca1c690f8 in _fini () from /usr/lib64/libsmime3.so
#1  0x00007f9ca216a605 in _dl_fini () at dl-fini.c:143
#2  0x00007f9ca1ebd8fc in __run_exit_handlers (status=1, listp=0x7f9ca2049718 <__exit_funcs>, run_list_atexit=run_list_a
texit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
#3  0x00007f9ca1ebda4a in __GI_exit (status=<optimized out>) at exit.c:139
#4  0x00005565384311b7 in std::sys::unix::os::exit () at src/libstd/sys/unix/os.rs:541
#5  0x000055653842fc3f in std::process::exit () at src/libstd/process.rs:1485
#6  0x000055653830c1cd in clap::app::App::get_matches ()
#7  0x00005565381e84a1 in wasm_pack::main ()
...

or:

Thread 2 (Thread 0x7f6461880980 (LWP 17749)):
#0  0x00007f6462369995 in _dl_fixup (l=0x7f64620408b0, reloc_arg=<optimized out>) at ../elf/dl-runtime.c:93
#1  0x00007f646237035a in _dl_runtime_resolve_xsave () at ../sysdeps/x86_64/dl-trampoline.h:126
#2  0x00007f6461d358a3 in __do_global_dtors_aux () from /usr/lib64/libnss3.so
#3  0x00007ffccfedee00 in ?? ()
#4  0x00007f646236a5e3 in _dl_fini () at dl-fini.c:138
Backtrace stopped: frame did not save the PC

Looking at openssl (I’m using 1.1.0k), it uses pthread_once (or equivalent) to run its initialization function once, and registers an atexit handler that tears everything down quite early in that initialization. We’re crashing trying to use a static lock in o_names.c that got nulled out by OBJ_NAME_cleanup, called from evp_cleanup_int, called by OPENSSL_cleanup, which is registered as an atexit handler by ossl_init_base, which runs early in OPENSSL_init_crypto. which gets called by OPENSSL_init_ssl before the call to ossl_init_ssl_base the crashing thread is in.

The backtraces from @derekdreery also contain openssl initialization functions and also occur when exiting immediately, so I think it’s the same crash.

The curl-sys assert was curl failing to initialize. I haven’t confirmed it has a similar race (and handles it better by failing initialization instead of crashing) but it seems plausible.

The segfault only happens when vendored OpenSSL is enabled, though, so running cargo install --path . --no-default-features will get you a working version of wasm-pack.

I was able to boil the segfault down to this minimal repro:

# Cargo.toml
[package]
name = "wasm-pack-segfault-repro"
version = "0.1.0"
edition = "2021"

[dependencies]
curl = "0.4.44"
openssl-sys = { version = "0.9.80", features = ["vendored"] }
reqwest = { version = "0.11.14", features = ["blocking"] }

// main.rs
use curl::easy;

fn main() {
    // This code comes from `wasm_pack::manifest::Crate::check_wasm_pack_latest_version`.
    struct NoopHandler;

    impl easy::Handler for NoopHandler {
        fn write(&mut self, data: &[u8]) -> Result<usize, easy::WriteError> {
            Ok(data.len())
        }
    }

    let mut easy = easy::Easy2::new(NoopHandler);
    easy.url("https://example.com").unwrap();
    easy.get(true).unwrap();
    // The segfault happens here.
    easy.perform().unwrap();

    // This code comes from `wasm_pack::install::krate::Krate::new`.
    // It's never reached, but the segfault doesn't happen if it's not here.
    let _ = reqwest::Client::builder();
}

This is the backtrace:

#0  0x00007ffff7c8bd66 in pthread_rwlock_wrlock () from /usr/lib/libc.so.6
#1  0x00007ffff77ca0ee in CRYPTO_THREAD_write_lock () from /usr/lib/libcrypto.so.3
#2  0x00007ffff7837ee5 in ?? () from /usr/lib/libcrypto.so.3
#3  0x00007ffff7837f81 in X509_STORE_add_cert () from /usr/lib/libcrypto.so.3
#4  0x00007ffff781c540 in X509_load_cert_crl_file_ex () from /usr/lib/libcrypto.so.3
#5  0x00007ffff781c696 in ?? () from /usr/lib/libcrypto.so.3
#6  0x00007ffff7837a4a in X509_STORE_load_file_ex () from /usr/lib/libcrypto.so.3
#7  0x00007ffff7f62012 in ?? () from /usr/lib/libcurl.so.4
#8  0x00007ffff7f63357 in ?? () from /usr/lib/libcurl.so.4
#9  0x00007ffff7f5ac0c in ?? () from /usr/lib/libcurl.so.4
#10 0x00007ffff7f3641b in ?? () from /usr/lib/libcurl.so.4
#11 0x00007ffff7f3944e in curl_multi_perform () from /usr/lib/libcurl.so.4
#12 0x00007ffff7f10044 in curl_easy_perform () from /usr/lib/libcurl.so.4
#13 0x0000555555654038 in curl::easy::handler::Easy2<wasm_pack_segfault_repro::main::NoopHandler>::perform<wasm_pack_segfault_repro::main::NoopHandler> (self=0x7fffffffdd20)
    at /home/liam/.cargo/registry/src/github.com-1ecc6299db9ec823/curl-0.4.44/src/easy/handler.rs:3163
#14 0x0000555555652a65 in wasm_pack_segfault_repro::main () at src/main.rs:16

I’m still not really sure why it’s happening, though, and particularly why including the bit of code referencing reqwest that’s never reached is necessary. The segfault happens to also come from libcrypto, but it doesn’t seem particularly related to the original issue.

If you’re segfaulting only on specific wasm-pack invocations, you’re probably hitting a different bug. This bug covers a race condition that can be hit by running wasm-pack with no arguments (or other wasm-pack invocations that are fast). If you’re only segfaulting when building something, it’s probably a different bug.

Please share a backtrace for the segfault (and if it doesn’t look like this bug, probably best to file a new one and CC me if you want me to look at the backtraces).

I had a quick look to see if there’s an obvious difference between LibreSSL and OpenSSL, or if it’s just more timing changes hiding the problem. There is a real difference, but I don’t know if it’s intentional or not.

Comparing OpenSSL 1.1.1d and LibreSSL 3.0.2, the important difference is that I see no atexit() handlers in LibreSSL.

However, that might be because LibreSSL currently does not need them, rather than by design. The lock whose cleanup triggered the segfault I saw before is also not present in libressl. It looks like this lock was introduced in openssl relatively recently (https://github.com/openssl/openssl/pull/3525), and (assuming https://github.com/libressl-portable/openbsd/commits/master/src/lib/libcrypto/objects/o_names.c is current) libressl has not yet picked up those changes.

So it’s possible libressl still has that race condition, and that if they choose to fix it the same way openssl did it’ll start crashing wasm-pack the same way. (Or if they start using atexit() for cleanup for other reasons.) Unless someone confirms libressl intentionally does not use atexit handlers I wouldn’t recommend LibreSSL as the fix for this one.

I would love to see this documented somewhere, but maybe it’s a bit too early to say LibreSSL fixes the segfaults before others could confirm the same?

Thanks for the great work on wasm-pack, by the way! ❤️

I’m happy to report that with LibreSSL 3.0.2, the race conditions I experienced are gone in both release and debug modes.