tokio: tokio@0.2.14 + futures::executor::block_on causes hang
Version
tokio@0.2.14 and tokio@0.2.15 exhibit this issue, tokio@0.2.13 does not.
tokio-hang-reproduction v0.1.0 (/Users/bryan/personal/tokio-hang-reproduction)
│ │ │ ├── tokio v0.2.14
│ │ │ │ └── tokio-macros v0.2.5
│ │ │ └── tokio-util v0.3.1
│ │ │ └── tokio v0.2.14 (*)
│ │ ├── tokio v0.2.14 (*)
│ ├── tokio v0.2.14 (*)
└── tokio v0.2.14 (*)
Platform
Currently, I can get this to happen on AWS Lambda.
I can not reproduce on MacOS (Darwin Bryans-MacBook-Pro.local 19.3.0 Darwin Kernel Version 19.3.0: Thu Jan 9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64 x86_64) or in Docker-for-Mac.
Description
For our GraphQL server, we use AWS Lambda + Juniper + Tokio. Juniper is currently a synchronous library, but the Lambda library we use and our model layer are fully tokio@0.2.
In order to get Juniper to call our models, we use futures::executor::block_on, based largely on this comment: https://github.com/tokio-rs/tokio/issues/2042#issuecomment-569919258
Starting with tokio@0.2.14, this approach has started hanging and causing timeouts. Downgrading to tokio@0.2.13 solved the issue.
https://github.com/bryanburgers/tokio-hang-reproduction
#![allow(dead_code)]
use futures::stream::{self, StreamExt};
use tokio::time::delay_for;
use std::time::Duration;
use serde_json::{Value, json};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync + 'static>> {
let fun = lambda::handler_fn(handler);
lambda::run(fun).await?;
Ok(())
}
async fn handler(_event: Value) -> Result<Value, String> {
let result = invoke_juniper();
// If we don't use juniper and .await the futures normally, it works fine.
// let result = test1(100).await;
println!("result={:?}", result);
Ok(json!({}))
}
// Juniper (https://crates.io/crates/juniper) is a synchronous library.
fn invoke_juniper() -> Vec<usize> {
// And from it, we call into our async code using block_on
futures::executor::block_on(test1(100))
}
// All three of these functions fail after executing ~60 futures.
async fn test1(n: usize) -> Vec<usize> {
stream::iter(0..n).map(pause_and_return).buffer_unordered(10).collect().await
}
async fn test2(n: usize) -> Vec<usize> {
stream::iter(0..n).map(pause_and_return).buffered(10).collect().await
}
async fn test3(n: usize) -> Vec<usize> {
let mut vec = Vec::new();
for i in 0..n {
vec.push(pause_and_return(i).await);
}
vec
}
async fn pause_and_return(n: usize) -> usize {
println!("Starting {}", n);
delay_for(Duration::from_millis(100)).await;
println!("Finished {}", n);
n
}
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 17 (14 by maintainers)
Use
spawn_blockingwhen you go from async -> sync. If you do that, you 100% can usefutures::executors::block_onto go from sync -> async.@bartlomieju Using
block_onin a single-threaded executor sounds like a recipe for deadlock disaster. I very strongly recommend finding another solution.Calling
futures::executor::block_on(...)should never be done from within the context of a tokio task. Unfortunately, we do not control that function as it is fromfutures. If we did, we would make it panic.The solution is indeed to wrap blocking calls w/
spawn_blocking.Blocking the underlying runtime thread is allowed to cause a deadlock. Don’t do it.
One cause can be the cooperative scheduling feature. If Tokio decides that the task has been running for too long, it will make all futures in the task return pending until the task yields back to the Tokio runtime. Another cause could be if you request a task dump.
Additional sources of deadlocks may be added in future Tokio releases. So don’t do it, even if it works today.
I understand that I shouldn’t be using
futures::executor::block_on.I understand that
spawn_blockingcan get me from an async context to a sync context so I can call the third party code (in this case, Juniper).What I don’t understand is when Juniper calls code I provided to it (assuming a synchronous context), how can I get back into an async context to get data that I can return to it Juniper?
That’s why
futures::executor::block_onis appealing (even though it’s wrong). It has the right signature.I need something that is roughly
fn _(fut: impl Future<Output=T>) -> T.I can’t use
runtime.block_onbecause In this context it’s not possible to have a mutable reference to the runtime.Handledoesn’t have ablock_on. If theJoinHandlereturned from ahandle.spawncan be waited for from a synchronous context, I don’t see it.I really wish I didn’t have to do the async -> sync -> async dance, but at this point I don’t have a choice. I’m really looking for any reliable way to make that second jump.
So, the code inside the
futures::executor::block_onis not really running on the tokio executor. If a future inside of theblock_onyields, it only yields to the executor created by the call toblock_on, not back to the Tokio runtime created bytokio::main.I think that the reason this code is breaking in 0.2.14 is because 0.2.14 introduced automatic cooperative task yielding to improve the scheduler’s fairness. After your stream iterates so many times, in 0.2.14+ the task’s budget is exhausted, and it yields to the executor to give other tasks a turn. When that happens, it should notify itself so that it goes to the end of the queue to continue executing after other tasks have been allowed to run. My guess is that for some reason related to an interaction between
futures::executor::block_onand running on a Tokio worker, your task is not being polled again after it yields.Using
spawn_blockingfixes this, because the blocking code is now running on a separate blocking threadpool, rather than on a Tokio worker thread. However, it’s not ideal, because when you useblock_onto start executing async code again, it’s no longer running on the Tokio runtime. I think what you want to be doing is running the blocking call withspawn_blocking, and when you need to call back into your async code, you spawn it on the main Tokio runtime, and only usefutures::executor::block_on(from the blocking thread) to await the spawned future’sJoinHandle. I mean, ideally, you wouldn’t be doing that, either — the blocking code would just return, and then the async work would continue, rather than having the blocking code call back into asynchronous code, but if the third-party library doesn’t work that way, I guess this would be the closest you can get.What happens if you put the blocking call (
invoke_juniper) in atokio::task::spawn_blockingortokio::task::block_in_placecall?