tokio: tokio@0.2.14 + futures::executor::block_on causes hang

Version

tokio@0.2.14 and tokio@0.2.15 exhibit this issue, tokio@0.2.13 does not.

tokio-hang-reproduction v0.1.0 (/Users/bryan/personal/tokio-hang-reproduction)
│   │   │   ├── tokio v0.2.14
│   │   │   │   └── tokio-macros v0.2.5
│   │   │   └── tokio-util v0.3.1
│   │   │       └── tokio v0.2.14 (*)
│   │   ├── tokio v0.2.14 (*)
│   ├── tokio v0.2.14 (*)
└── tokio v0.2.14 (*)

Platform

Currently, I can get this to happen on AWS Lambda.

I can not reproduce on MacOS (Darwin Bryans-MacBook-Pro.local 19.3.0 Darwin Kernel Version 19.3.0: Thu Jan 9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64 x86_64) or in Docker-for-Mac.

Description

For our GraphQL server, we use AWS Lambda + Juniper + Tokio. Juniper is currently a synchronous library, but the Lambda library we use and our model layer are fully tokio@0.2.

In order to get Juniper to call our models, we use futures::executor::block_on, based largely on this comment: https://github.com/tokio-rs/tokio/issues/2042#issuecomment-569919258

Starting with tokio@0.2.14, this approach has started hanging and causing timeouts. Downgrading to tokio@0.2.13 solved the issue.

https://github.com/bryanburgers/tokio-hang-reproduction

#![allow(dead_code)]
use futures::stream::{self, StreamExt};
use tokio::time::delay_for;
use std::time::Duration;
use serde_json::{Value, json};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync + 'static>> {
    let fun = lambda::handler_fn(handler);
    lambda::run(fun).await?;
    Ok(())
}

async fn handler(_event: Value) -> Result<Value, String> {
    let result = invoke_juniper();
    // If we don't use juniper and .await the futures normally, it works fine.
    // let result = test1(100).await;
    println!("result={:?}", result);
    Ok(json!({}))
}

// Juniper (https://crates.io/crates/juniper) is a synchronous library.
fn invoke_juniper() -> Vec<usize> {
    // And from it, we call into our async code using block_on
    futures::executor::block_on(test1(100))
}


// All three of these functions fail after executing ~60 futures.
async fn test1(n: usize) -> Vec<usize> {
    stream::iter(0..n).map(pause_and_return).buffer_unordered(10).collect().await
}

async fn test2(n: usize) -> Vec<usize> {
    stream::iter(0..n).map(pause_and_return).buffered(10).collect().await
}

async fn test3(n: usize) -> Vec<usize> {
    let mut vec = Vec::new();
    for i in 0..n {
        vec.push(pause_and_return(i).await);
    }
    vec
}

async fn pause_and_return(n: usize) -> usize {
    println!("Starting {}", n);
    delay_for(Duration::from_millis(100)).await;
    println!("Finished {}", n);
    n
}

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 17 (14 by maintainers)

Most upvoted comments

Use spawn_blocking when you go from async -> sync. If you do that, you 100% can use futures::executors::block_on to go from sync -> async.

@bartlomieju Using block_on in a single-threaded executor sounds like a recipe for deadlock disaster. I very strongly recommend finding another solution.

Calling futures::executor::block_on(...) should never be done from within the context of a tokio task. Unfortunately, we do not control that function as it is from futures. If we did, we would make it panic.

The solution is indeed to wrap blocking calls w/ spawn_blocking.

Blocking the underlying runtime thread is allowed to cause a deadlock. Don’t do it.

One cause can be the cooperative scheduling feature. If Tokio decides that the task has been running for too long, it will make all futures in the task return pending until the task yields back to the Tokio runtime. Another cause could be if you request a task dump.

Additional sources of deadlocks may be added in future Tokio releases. So don’t do it, even if it works today.

I understand that I shouldn’t be using futures::executor::block_on.

I understand that spawn_blocking can get me from an async context to a sync context so I can call the third party code (in this case, Juniper).

What I don’t understand is when Juniper calls code I provided to it (assuming a synchronous context), how can I get back into an async context to get data that I can return to it Juniper?

That’s why futures::executor::block_on is appealing (even though it’s wrong). It has the right signature.

I need something that is roughly fn _(fut: impl Future<Output=T>) -> T.

I can’t use runtime.block_on because In this context it’s not possible to have a mutable reference to the runtime. Handle doesn’t have a block_on. If the JoinHandle returned from a handle.spawn can be waited for from a synchronous context, I don’t see it.

I really wish I didn’t have to do the async -> sync -> async dance, but at this point I don’t have a choice. I’m really looking for any reliable way to make that second jump.

So, the code inside the futures::executor::block_on is not really running on the tokio executor. If a future inside of the block_on yields, it only yields to the executor created by the call to block_on, not back to the Tokio runtime created by tokio::main.

I think that the reason this code is breaking in 0.2.14 is because 0.2.14 introduced automatic cooperative task yielding to improve the scheduler’s fairness. After your stream iterates so many times, in 0.2.14+ the task’s budget is exhausted, and it yields to the executor to give other tasks a turn. When that happens, it should notify itself so that it goes to the end of the queue to continue executing after other tasks have been allowed to run. My guess is that for some reason related to an interaction between futures::executor::block_on and running on a Tokio worker, your task is not being polled again after it yields.

Using spawn_blocking fixes this, because the blocking code is now running on a separate blocking threadpool, rather than on a Tokio worker thread. However, it’s not ideal, because when you use block_on to start executing async code again, it’s no longer running on the Tokio runtime. I think what you want to be doing is running the blocking call with spawn_blocking, and when you need to call back into your async code, you spawn it on the main Tokio runtime, and only use futures::executor::block_on (from the blocking thread) to await the spawned future’s JoinHandle. I mean, ideally, you wouldn’t be doing that, either — the blocking code would just return, and then the async work would continue, rather than having the blocking code call back into asynchronous code, but if the third-party library doesn’t work that way, I guess this would be the closest you can get.

What happens if you put the blocking call (invoke_juniper) in a tokio::task::spawn_blocking or tokio::task::block_in_place call?