miri: Epoll deadlock on tokio's test

Description

When running one of the tokio test with cargo +nightly miri test --features full --test spawn_pinned, miri reported deadlock on epoll_wait syscall. This is likely caused by epoll shim not receiving an epoll event notification to unblock the thread when it should have received one in real world system.

Version

Rustc:

rustc 1.83.0-nightly (7042c269c 2024-09-23)
binary: rustc
commit-hash: 7042c269c166191cd5d8daf0409890903df7af57
commit-date: 2024-09-23
host: x86_64-unknown-linux-gnu
release: 1.83.0-nightly
LLVM version: 19.1.0

Tokio: Tokio repo master branch commit 27539ae3

Full trace
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.06s
     Running tests/spawn_pinned.rs (target/miri/x86_64-unknown-linux-gnu/debug/deps/spawn_pinned-5d9d1b31504e33c5)

running 9 tests
test callback_panic_does_not_kill_worker ... error: deadlock: the evaluated program deadlocked
   --> /home/byt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-1.0.2/src/sys/unix/selector/epoll.rs:56:9
    |
56  | /         syscall!(epoll_wait(
57  | |             self.ep.as_raw_fd(),
58  | |             events.as_mut_ptr(),
59  | |             events.capacity() as i32,
60  | |             timeout,d
61  | |         ))
    | |__________^ the evaluated program deadlocked
    |
    = note: BACKTRACE on thread `unnamed-2`:
    = note: inside `mio::sys::unix::selector::Selector::select` at /home/byt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-1.0.2/src/sys/unix/mod.rs:8:48: 8:49
    = note: inside `mio::poll::Poll::poll` at /home/byt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-1.0.2/src/poll.rs:435:9: 435:61
note: inside `tokio::runtime::io::driver::Driver::turn`
   --> /home/byt/Documents/tokio/tokio/src/runtime/io/driver.rs:149:15
    |
149 |         match self.poll.poll(events, max_wait) {
    |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::io::driver::Driver::park`
   --> /home/byt/Documents/tokio/tokio/src/runtime/io/driver.rs:122:9
    |
122 |         self.turn(handle, None);
    |         ^^^^^^^^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::signal::Driver::park`
   --> /home/byt/Documents/tokio/tokio/src/runtime/signal/mod.rs:92:9
    |
92  |         self.io.park(handle);
    |         ^^^^^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::process::Driver::park`
   --> /home/byt/Documents/tokio/tokio/src/runtime/process.rs:32:9
    |
32  |         self.park.park(handle);
    |         ^^^^^^^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::driver::IoStack::park`
   --> /home/byt/Documents/tokio/tokio/src/runtime/driver.rs:175:40
    |
175 |                 IoStack::Enabled(v) => v.park(handle),
    |                                        ^^^^^^^^^^^^^^
note: inside `tokio::runtime::time::Driver::park_internal`
   --> /home/byt/Documents/tokio/tokio/src/runtime/time/mod.rs:247:21
    |
247 |                     self.park.park(rt_handle);
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::time::Driver::park`
   --> /home/byt/Documents/tokio/tokio/src/runtime/time/mod.rs:173:9
    |
173 |         self.park_internal(handle, None);
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::driver::TimeDriver::park`
   --> /home/byt/Documents/tokio/tokio/src/runtime/driver.rs:332:55
    |
332 |                 TimeDriver::Enabled { driver, .. } => driver.park(handle),
    |                                                       ^^^^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::driver::Driver::park`
   --> /home/byt/Documents/tokio/tokio/src/runtime/driver.rs:71:9
    |
71  |         self.inner.park(handle);
    |         ^^^^^^^^^^^^^^^^^^^^^^^
note: inside closure
   --> /home/byt/Documents/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:382:17
    |
382 |                 driver.park(&handle.driver);
    |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::scheduler::current_thread::Context::enter::<(), {closure@tokio::runtime::scheduler::current_thread::Context::park::{closure#1}}>`
   --> /home/byt/Documents/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:423:19
    |
423 |         let ret = f();
    |                   ^^^
note: inside `tokio::runtime::scheduler::current_thread::Context::park`
   --> /home/byt/Documents/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:381:27
    |
381 |               let (c, ()) = self.enter(core, || {
    |  ___________________________^
382 | |                 driver.park(&handle.driver);
383 | |                 self.defer.wake();
384 | |             });
    | |______________^
note: inside closure
   --> /home/byt/Documents/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:724:33
    |
724 | ...                   context.park(core, handle)
    |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside closure
   --> /home/byt/Documents/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:774:68
    |
774 |         let (core, ret) = context::set_scheduler(&self.context, || f(core, context));
    |                                                                    ^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::context::scoped::Scoped::<tokio::runtime::scheduler::Context>::set::<{closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::enter<{closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::block_on<std::pin::Pin<&mut {async fn body of tokio::task::LocalSet::run_until<{async block@tokio_util::task::spawn_pinned::LocalWorkerHandle::run::{closure#0}}>()}>>::{closure#0}}, std::option::Option<()>>::{closure#0}}, (std::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, std::option::Option<()>)>`
   --> /home/byt/Documents/tokio/tokio/src/runtime/context/scoped.rs:40:9
    |
40  |         f()
    |         ^^^
note: inside closure
   --> /home/byt/Documents/tokio/tokio/src/runtime/context.rs:180:26
    |
180 |         CONTEXT.with(|c| c.scheduler.set(v, f))
    |                          ^^^^^^^^^^^^^^^^^^^^^
    = note: inside `std::thread::LocalKey::<tokio::runtime::context::Context>::try_with::<{closure@tokio::runtime::context::set_scheduler<(std::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, std::option::Option<()>), {closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::enter<{closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::block_on<std::pin::Pin<&mut {async fn body of tokio::task::LocalSet::run_until<{async block@tokio_util::task::spawn_pinned::LocalWorkerHandle::run::{closure#0}}>()}>>::{closure#0}}, std::option::Option<()>>::{closure#0}}>::{closure#0}}, (std::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, std::option::Option<()>)>` at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:283:12: 283:27
    = note: inside `std::thread::LocalKey::<tokio::runtime::context::Context>::with::<{closure@tokio::runtime::context::set_scheduler<(std::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, std::option::Option<()>), {closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::enter<{closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::block_on<std::pin::Pin<&mut {async fn body of tokio::task::LocalSet::run_until<{async block@tokio_util::task::spawn_pinned::LocalWorkerHandle::run::{closure#0}}>()}>>::{closure#0}}, std::option::Option<()>>::{closure#0}}>::{closure#0}}, (std::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, std::option::Option<()>)>` at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:260:9: 260:25
note: inside `tokio::runtime::context::set_scheduler::<(std::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, std::option::Option<()>), {closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::enter<{closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::block_on<std::pin::Pin<&mut {async fn body of tokio::task::LocalSet::run_until<{async block@tokio_util::task::spawn_pinned::LocalWorkerHandle::run::{closure#0}}>()}>>::{closure#0}}, std::option::Option<()>>::{closure#0}}>`
   --> /home/byt/Documents/tokio/tokio/src/runtime/context.rs:180:9
    |
180 |         CONTEXT.with(|c| c.scheduler.set(v, f))
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::scheduler::current_thread::CoreGuard::<'_>::enter::<{closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::block_on<std::pin::Pin<&mut {async fn body of tokio::task::LocalSet::run_until<{async block@tokio_util::task::spawn_pinned::LocalWorkerHandle::run::{closure#0}}>()}>>::{closure#0}}, std::option::Option<()>>`
   --> /home/byt/Documents/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:774:27
    |
774 |         let (core, ret) = context::set_scheduler(&self.context, || f(core, context));
    |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside closure
   --> /home/byt/Documents/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:191:28
    |
191 |                     return core.block_on(future);
    |                            ^^^^^^^^^^^^^^^^^^^^^
note: inside `tokio_util::task::spawn_pinned::LocalWorkerHandle::run`
   --> /home/byt/Documents/tokio/tokio-util/src/task/spawn_pinned.rs:396:9
    |
396 | /         local_set.block_on(&runtime, async {
397 | |             while let Some(spawn_task) = task_receiver.recv().await {
398 | |                 // Calls spawn_local(future)
399 | |                 (spawn_task)();
400 | |             }
401 | |         });
    | |__________^
note: inside closure
   --> /home/byt/Documents/tokio/tokio-util/src/task/spawn_pinned.rs:381:31
    |
381 |         std::thread::spawn(|| Self::run(runtime, receiver, task_count_clone));
    |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    = note: this error originates in the macro `syscall` (in Nightly builds, run with -Z macro-backtrace for more info)

error: deadlock: the evaluated program deadlocked
  --> /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/pal/unix/futex.rs:75:21
   |
75 |                     )
   |                     ^ the evaluated program deadlocked
   |
   = note: BACKTRACE:
   = note: inside `std::sys::pal::unix::futex::futex_wait` at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/pal/unix/futex.rs:75:21: 75:22
   = note: inside `std::sys::sync::thread_parking::futex::Parker::park` at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/sync/thread_parking/futex.rs:55:13: 55:50
   = note: inside `std::thread::Thread::park` at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:1404:18: 1404:53
   = note: inside `std::thread::park` at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:1110:9: 1110:25
   = note: inside `std::sync::mpmc::context::Context::wait_until` at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/mpmc/context.rs:138:17: 138:31
   = note: inside closure at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/mpmc/list.rs:447:27: 447:50
   = note: inside closure at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/mpmc/context.rs:49:13: 49:18
   = note: inside closure at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/mpmc/context.rs:57:31: 57:37
   = note: inside `std::thread::LocalKey::<std::cell::Cell<std::option::Option<std::sync::mpmc::context::Context>>>::try_with::<{closure@std::sync::mpmc::context::Context::with<{closure@std::sync::mpmc::list::Channel<test::event::CompletedTest>::recv::{closure#1}}, ()>::{closure#1}}, ()>` at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:283:12: 283:27
   = note: inside `std::sync::mpmc::context::Context::with::<{closure@std::sync::mpmc::list::Channel<test::event::CompletedTest>::recv::{closure#1}}, ()>` at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/mpmc/context.rs:52:9: 61:15
   = note: inside `std::sync::mpmc::list::Channel::<test::event::CompletedTest>::recv` at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/mpmc/list.rs:437:13: 458:15
   = note: inside `std::sync::mpmc::Receiver::<test::event::CompletedTest>::recv` at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/mpmc/mod.rs:308:43: 308:58
   = note: inside `std::sync::mpsc::Receiver::<test::event::CompletedTest>::recv` at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/mpsc/mod.rs:848:9: 848:26
   = note: inside `test::run_tests::<{closure@test::run_tests_console::{closure#2}}>` at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:379:38: 379:47
   = note: inside `test::run_tests_console` at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/console.rs:322:5: 322:70
   = note: inside `test::test_main` at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:137:19: 137:59
   = note: inside `test::test_main_static` at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:159:5: 159:40
   = note: inside `main`

error: deadlock: the evaluated program deadlocked
   --> /home/byt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-1.0.2/src/sys/unix/selector/epoll.rs:56:9
    |
56  | /         syscall!(epoll_wait(
57  | |             self.ep.as_raw_fd(),
58  | |             events.as_mut_ptr(),
59  | |             events.capacity() as i32,
60  | |             timeout,
61  | |         ))
    | |__________^ the evaluated program deadlocked
    |
    = note: BACKTRACE on thread `callback_panic_`:
    = note: inside `mio::sys::unix::selector::Selector::select` at /home/byt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-1.0.2/src/sys/unix/mod.rs:8:48: 8:49
    = note: inside `mio::poll::Poll::poll` at /home/byt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-1.0.2/src/poll.rs:435:9: 435:61
note: inside `tokio::runtime::io::driver::Driver::turn`
   --> /home/byt/Documents/tokio/tokio/src/runtime/io/driver.rs:149:15
    |
149 |         match self.poll.poll(events, max_wait) {
    |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::io::driver::Driver::park`
   --> /home/byt/Documents/tokio/tokio/src/runtime/io/driver.rs:122:9
    |
122 |         self.turn(handle, None);
    |         ^^^^^^^^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::signal::Driver::park`
   --> /home/byt/Documents/tokio/tokio/src/runtime/signal/mod.rs:92:9
    |
92  |         self.io.park(handle);
    |         ^^^^^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::process::Driver::park`
   --> /home/byt/Documents/tokio/tokio/src/runtime/process.rs:32:9
    |
32  |         self.park.park(handle);
    |         ^^^^^^^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::driver::IoStack::park`
   --> /home/byt/Documents/tokio/tokio/src/runtime/driver.rs:175:40
    |
175 |                 IoStack::Enabled(v) => v.park(handle),
    |                                        ^^^^^^^^^^^^^^
note: inside `tokio::runtime::time::Driver::park_internal`
   --> /home/byt/Documents/tokio/tokio/src/runtime/time/mod.rs:247:21
    |
247 |                     self.park.park(rt_handle);
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::time::Driver::park`
   --> /home/byt/Documents/tokio/tokio/src/runtime/time/mod.rs:173:9
    |
173 |         self.park_internal(handle, None);
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::driver::TimeDriver::park`
   --> /home/byt/Documents/tokio/tokio/src/runtime/driver.rs:332:55
    |
332 |                 TimeDriver::Enabled { driver, .. } => driver.park(handle),
    |                                                       ^^^^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::driver::Driver::park`
   --> /home/byt/Documents/tokio/tokio/src/runtime/driver.rs:71:9
    |
71  |         self.inner.park(handle);
    |         ^^^^^^^^^^^^^^^^^^^^^^^
note: inside closure
   --> /home/byt/Documents/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:382:17
    |
382 |                 driver.park(&handle.driver);
    |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::scheduler::current_thread::Context::enter::<(), {closure@tokio::runtime::scheduler::current_thread::Context::park::{closure#1}}>`
   --> /home/byt/Documents/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:423:19
    |
423 |         let ret = f();
    |                   ^^^
note: inside `tokio::runtime::scheduler::current_thread::Context::park`
   --> /home/byt/Documents/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:381:27
    |
381 |               let (c, ()) = self.enter(core, || {
    |  ___________________________^
382 | |                 driver.park(&handle.driver);
383 | |                 self.defer.wake();
384 | |             });
    | |______________^
note: inside closure
   --> /home/byt/Documents/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:724:33
    |
724 | ...                   context.park(core, handle)
    |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside closure
   --> /home/byt/Documents/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:774:68
    |
774 |         let (core, ret) = context::set_scheduler(&self.context, || f(core, context));
    |                                                                    ^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::context::scoped::Scoped::<tokio::runtime::scheduler::Context>::set::<{closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::enter<{closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::block_on<std::pin::Pin<&mut std::pin::Pin<&mut dyn futures::Future<Output = ()>>>>::{closure#0}}, std::option::Option<()>>::{closure#0}}, (std::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, std::option::Option<()>)>`
   --> /home/byt/Documents/tokio/tokio/src/runtime/context/scoped.rs:40:9
    |
40  |         f()
    |         ^^^
note: inside closure
   --> /home/byt/Documents/tokio/tokio/src/runtime/context.rs:180:26
    |
180 |         CONTEXT.with(|c| c.scheduler.set(v, f))
    |                          ^^^^^^^^^^^^^^^^^^^^^
    = note: inside `std::thread::LocalKey::<tokio::runtime::context::Context>::try_with::<{closure@tokio::runtime::context::set_scheduler<(std::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, std::option::Option<()>), {closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::enter<{closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::block_on<std::pin::Pin<&mut std::pin::Pin<&mut dyn futures::Future<Output = ()>>>>::{closure#0}}, std::option::Option<()>>::{closure#0}}>::{closure#0}}, (std::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, std::option::Option<()>)>` at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:283:12: 283:27
    = note: inside `std::thread::LocalKey::<tokio::runtime::context::Context>::with::<{closure@tokio::runtime::context::set_scheduler<(std::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, std::option::Option<()>), {closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::enter<{closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::block_on<std::pin::Pin<&mut std::pin::Pin<&mut dyn futures::Future<Output = ()>>>>::{closure#0}}, std::option::Option<()>>::{closure#0}}>::{closure#0}}, (std::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, std::option::Option<()>)>` at /home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:260:9: 260:25
note: inside `tokio::runtime::context::set_scheduler::<(std::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, std::option::Option<()>), {closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::enter<{closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::block_on<std::pin::Pin<&mut std::pin::Pin<&mut dyn futures::Future<Output = ()>>>>::{closure#0}}, std::option::Option<()>>::{closure#0}}>`
   --> /home/byt/Documents/tokio/tokio/src/runtime/context.rs:180:9
    |
180 |         CONTEXT.with(|c| c.scheduler.set(v, f))
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside `tokio::runtime::scheduler::current_thread::CoreGuard::<'_>::enter::<{closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::block_on<std::pin::Pin<&mut std::pin::Pin<&mut dyn futures::Future<Output = ()>>>>::{closure#0}}, std::option::Option<()>>`
   --> /home/byt/Documents/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:774:27
    |
774 |         let (core, ret) = context::set_scheduler(&self.context, || f(core, context));
    |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside closure
   --> /home/byt/Documents/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:191:28
    |
191 |                     return core.block_on(future);
    |                            ^^^^^^^^^^^^^^^^^^^^^
note: inside `callback_panic_does_not_kill_worker`
   --> tokio-util/tests/spawn_pinned.rs:120:5
    |
120 |     assert_eq!(result.unwrap(), "test");
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside closure
   --> tokio-util/tests/spawn_pinned.rs:100:47
    |
98  | #[tokio::test]
    | -------------- in this procedural macro expansion
99  | #[cfg(panic = "unwind")]
100 | async fn callback_panic_does_not_kill_worker() {
    |                                               ^
    = note: this error originates in the macro `syscall` which comes from the expansion of the attribute macro `tokio::test` (in Nightly builds, run with -Z macro-backtrace for more info)

note: some details are omitted, run with `MIRIFLAGS=-Zmiri-backtrace=full` for a verbose backtrace

error: aborting due to 3 previous errors

error: test failed, to rerun pass `-p tokio-util --test spawn_pinned`

Caused by:
  process didn't exit successfully: `/home/byt/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo-miri runner /home/byt/Documents/tokio/target/miri/x86_64-unknown-linux-gnu/debug/deps/spawn_pinned-5d9d1b31504e33c5` (exit status: 1)
note: test exited abnormally; to see the full output pass --nocapture to the harness.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 60 (60 by maintainers)

Commits related to this issue

Most upvoted comments

Adding VClock to the epoll ready_list fixes the tokio reproducer.

Sorry, yes, I meant the ready list whenever I said “interest list”.

Okay in that case it’s almost certainly a lack of proper vector clock synchronization as part of the epoll-induced wakeup. You can look e.g. at the clock field in eventfd to see how this must be done, though figuring out the details for epoll could be a bit tricky since it is a more complicated primitive.

This also means there is likely a way to write a program where Miri reports UB but it doesn’t actually have UB. Something like

  • set up some file descriptor and an epoll watching it (let’s say a pipe)
  • spawn a thread A
  • thread A first mutates some static mut, then writes to the pipe
  • meanwhile, the main thread waits with epoll until the pipe is ready, and then reads from the static mut. (it never reads from the pipe!)

This is basically the epoll version of this test.

So likely every interest list also needs to track a vector clock, and every time an event gets added to the interest list, the current threads’ clock is joined into the interest list clock. And then when a wakeup bappens, the woken-up thread acquires the interest list clock.

Maybe this helps understand the failing code a little better. There are three threads being reported as deadlocked. I think this means there are three threads with no chance of making further progress in the simulator. Likely they aren’t actually deadlocked with each other. But the design would have called for a clean exit on the main thread after the nop future was polled on the worker thread.

So it appears an event was lost or an event didn’t get created.

Of the three threads, two are blocked on epoll, and one is blocked on a futex.

The two epolls being blocked on are actually in different tokio runtimes. There is the multi_thread tokio runtime/scheduler that was responsible for the futex blocked thread and the main thread. And there is a current_thread (aka single threaded tokio) scheduler that is also blocked on an epoll.

The output I’m looking at shows the three thread backtraces, I’ll number then 1, 2 and 3, in the order they are printed.

1 - shows a thread name of “unnamed-2” and is the current_thread tokio blocked on epoll.

2 - shows no thread name, is blocked on a futex and condvar, and deeper down the stack shows is part of the tokio multi_thread runtime.

3 - shows a thread name of “tokio-runtime-w”, and is the multi-thread runtime blocked on epoll.

I’m going to go out on a limb and guess that thread 1 above is the sole worker thread that was created for the pool and it is in its normal state of having nothing to do so it waits. The logic in Miri should have signaled that the work it was given has completed.

I am wondering if there are API things we can do to reduce the likelihood of this. Thread unblocking almost always occurs because of some other event an another thread that we should register a causal link with, so maybe the thread unblocking function should take a clock. It’s just we’ve so far kept those subsystems largely orthogonal.

Thanks for debugging this, finding a fix and opening a PR! This was definitely very unexpected from just looking at the original issue

Yes, the tokio reproducible is one of my goals. Just getting it to compile is satisfying but not that satisfying.

I kind of hope I can reproduce the deadlock issue purely using syscalls to check whether it is fixed.

As far as I am concerned, the above is a better test. But if you want to keep looking I won’t stop you. 😃

Fences on their own don’t do synchronization, they only help in combination with a read-write pair relating two threads. But in epoll these reads/writes of the interest list are working on extra machine state, not regular memory. So no, just adding some fence won’t help.

However, there is no doubt that returning from a blocked epoll should be an acquire operation (some thing happened in another thread, which woke up this thread – there’s a causal link here, and every such causal link should be reflected in the vector clocks). And currently I don’t think there’s any vector clock management in epoll. So even if we ignore this bug, this is something that must be fixed. And given the fact that the issue disappears with -Zmiri-disable-weak-memory-emulation, I am fairly confident that that will also fix the deadlock.

Just an update - I am making progress in understanding where the passing path diverges from the failing path in the “best” case. Best defined by the longest common path created by the Miri PRNG. Thank goodness for the INFO messages from the miri::machine module. I clearly see a difference in the miri shim use of the first futex. On the failing side, there is never a futex_wake called because the tokio code seems to have already thought there is no-one left on the other side of an unbounded mpsc channel (funny I’m not sure yet if the sender or the receiver was thought to be gone - I just see on the passing path that tokio::sync::maps::UnboundedSender code kicks in while on the failing side, std::rt::panic_count::count_is_zero kicks in). But even this, take with a grain of salt. I need to reproduce the results, and hopefully somehow reduce the diff noise that came before. I haven’t looked at how Miri uses its PRNG from the seed - maybe there is already a way to do this, but it would be nice if there were a second level of seed that could be kicked in after ‘n’ calls to the first PRNG - ‘n’ could then be determined manually with another binary-search. So when either a good seed or a bad seed was found, one could play with when to diverge down a new path. As it is, I have a passing seed of 167 and a failing seed of 178 that show remarkable congruency for well over a quarter of the extensive trace outputs (33,000 some lines are the same). But before interesting things happen about lines 105,000, lots of small and medium differences - and the significance of those differences ahead of the large divergence I haven’t figured out. I also have ideas of reducing the noise by running tokio with features features enabled and even hard-wiring some cfg macros like for coop and MetricsBatch, that one wouldn’t expect to affect the final results, but because of their use of the cell module, certainly cause a lot of variation possibilities for Miri from seed to seed.

If someone finds a quicker solution, I won’t mind but this investigation is certainly fun so far.

The interplay of the code running in the worker that would use condvar to indicate the work is done and then presumably call notify_one, and the condvar loop running in the main thread which is waiting to see some shared state change is where I want to get further visibility. I’m still looking for the tokio call that does these two sides, and then I’ll look at the Miri shim for those pieces. (I think shim is the right term but I could be wrong.)

Generally, each Tokio runtime will have one thread blocked on epoll, and all other threads blocked on a futex/condvar. In the case of the main thread, that’s a block_on call which is always in the futex/condvar category.

I think deadlock is the correct term. If no thread can make progress, that means there’s a subset of threads that are in a dependency cycle.