tokio: rfc: reducing "runtime not found" confusion
Context
Tokio’s resource types (TcpStream, time::delay, …) require a runtime to function. Users interact with the resource types by awaiting on results. The runtime receives events from the operating system related to these resources and dispatches the events to the appropriate waiting task.
Tokio does not implicitly spawn runtimes. It is the user’s responsibility to ensure a runtime is running. This is usually done using #[tokio::main] but creating a runtime manually is also possible.
Multiple runtime flavors are provided. There is a multi-threaded, work-stealing, runtime. This runtime spawns multiple threads. It is the recommended runtime for a number of cases, including network server applications. There is also a single-threaded “in-place” runtime. This runtime spawns no threads and is useful for cases like implementing a blocking interface for an async client. This is how the reqwest crate’s blocking module works.
Additionally, there are cases in which it is useful to have a single process start multiple runtimes of the same flavor. For example, linkerd currently uses a pair of single-threaded runtimes, one for data-plane forwarding work and the other for control-plane tasks (serving metrics, driving service discovery lookups, and so on). This helps keep the two workloads isolated so neither can starve the other, and allows the use of !Sync futures that avoid the overhead of synchronization, without making the entire application single-threaded. The ability to run multiple runtimes in the same process is an important feature for many use-cases.
The problem
When using a TcpStream, the resource type must reference a runtime in order to function. Given that Tokio may have any number of runtimes running in the current process, the resource type must have some strategy by which it can select the correct runtime.
Currently, this is done by using a thread-local to track the current Runtime. In many cases, a process only includes a single runtime. A problem arises when attempting to use a resource from outside of a runtime. In this case, it is unclear which runtime the resource type should use and Tokio will panic!.
use tokio::net::TcpStream;
fn main() {
// Boom, no runtime.
let future = TcpStream::connect("www.example.com");
}
The strategy for fixing this is to enter a runtime context using a runtime::Handle.
use tokio::net::TcpStream;
use tokio::runtime::Handle;
fn do_some_work(handle: &Handle) {
handle.enter(|| {
let future = TcpStream::connect("www.example.com");
});
}
This panic is the source of confusion for users who are not aware of how Tokio searches for the runtime.
History
In the very early days, tokio-core always required an explicit Handle. There was no context, thread-local or global, that stored the “current reactor”. This resulted in Handle being a field on virtually every single type or an argument to every single function. This was tedious given that, in most cases, there was only ever a single tokio-core reactor in the process. In some cases, it resulted in measurable performance degradation as the Handle field increased struct size.
Because of this, tokio-core started providing a static runtime. Resource types would default to using this static runtime. Resource types also included method variants that took an explicit &Handle, allowing the user to specify a custom runtime.
The primary problem with a static runtime is that it cannot easily be configured. However, all users who ended up configuring their runtime were forced to use the more verbose APIs with an explicit &Handle argument. Additionally, some libraries did not provide methods with an explicit &Handle argument, preventing them from being used with custom runtimes.
To solve these problems, Tokio added a thread-local tracking the “current” runtime. Now, resources would first check the thread-local and if it was not set, it would use the global runtime. This introduced a new problem. Users that intended to use a custom runtime would accidentally use their resource types from outside of their custom runtime, which would start the global runtime and shift their parts application to the global runtime. The worst part of this is everything “seemed to work” but was not doing what the user intended. Half the application ran on a static runtime with default configuration and the other half on the configured runtime. Usually, nothing was noticed until poor performance was noticed in production.
The final iteration, resulting in the Tokio of today, was to remove the concept of the global runtime in favor of the thread-local context. This prevents users from accidentally being shifted to the global runtime and things appear to work, but are in a degraded state. The consequence of this change is that attempting to use Tokio’s resource types from outside of a runtime results in a panic.
Options
There are a few ways forward from here. These options are not mutually exclusive. This issue is to discuss ways forward. Feel free to propose alternate strategies as well.
Change no behavior and improve the panic message
The thread-local context logic can remain unchanged. Instead, the panic message is improved to include more context about the problem and some options for fixing it.
Re-introduce a static runtime using a feature flag
In this case, a static runtime is re-introduced. However, it is guarded by a feature flag: rt-global. rt-global would also be included in the full meta feature.
When rt-global is enabled, Tokio resources would first check the thread-local context. If it is set, the current runtime is used. If there isn’t one set, then the global runtime is used.
The primary danger here is silently ending up with the “split application”. Since feature flags are additive, a library or component may include the rt-global feature flag and the application does not know it is accidentally using the global runtime.
Provide a separate tokio-global-runtimecrate
The specific name of the crate would need to be massaged. The idea is to have a separate crate define the static variable. In this case, users who wish to use a statically defined runtime would depend on this crate.
The main downside as I see it to this is that it makes the global runtime less discoverable.
Re-introduce &Handle method variants
In this case, all async methods include a variant that takes an explicit &runtime::Handle. Users who want to ensure a runtime exists and it is the correct runtime may opt to be explicit about the runtime used by specifying it.
This doesn’t really solve the problem that users are confused when calling TcpStream::connect panics as it requires them knowing they must call the &runtime::Handle variant. However, improvements to the panic message would include mention of this strategy.
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 14
- Comments: 26 (24 by maintainers)
From these options, I (as someone who has definitely hit this a few times and remembering being pretty confused) feel that the current situation is the most attractive if the error message was sufficiently good. Maybe we could even include a link to a documentation page that lays out (a) ways to fix it and (b) background why this is the case (as laid out here – thanks for writing it up so clearly!).
I added a new option:
Provide a separate
tokio-global-runtimecrateThe specific name of the crate would need to be massaged. The idea is to have a separate crate define the static variable. In this case, users who wish to use a statically defined runtime would depend on this crate.
The main downside as I see it to this is that it makes the global runtime less discoverable.
I feel like there are a bunch of somewhat independent issues being addressed together here, which isn’t really helping. For the issue of beginners (or experienced users who are just forgetful – this has definitely happened to me), just making the error for trying to initialize resources that require a runtime more explicit or clear will get a lot of mileage.
I’m not a fan of the feature flag approach, since additive nature of feature flags mean that it is pretty easy to get a runtime going without being aware of it.
It’s a little unclear to me whether having a single static runtime would still need to depend on thread locals, or whether there would perhaps be some level of performance benefit to having a static runtime. If that is the case, I think we should definitely offer that as an option. I don’t find the “lack of discoverability” argument against separate crate all that convincing – there are plenty of ways we could advertise that option in documentation.
I quite like the explicit of
&Handlearguments, though it’s a little unclear to me how pervasive (and thus potentially unergonomic) these would be in practice.In general, I think we can agree that feature flags should not change behavior (I am aware I failed some on this front in 0.2).
As someone who moved from
slogtotracing, I think this is a dubious point of reference; I log messages way more often than I spawn tasks.A few questions and opinions on this topic, not especially organized:
The panic message should be more detailed if it’s kept in. It can happen a lot of different ways, which may make it difficult to explain well without confusing the reader with irrelevant information:
tokioruntimeRuntime, but trying to do some setup work outside the runtime first (e.g. as seen in seanmonstar/reqwest#778)Runtime, but left it - I’m not sure exactly which ofspawn_blocking,block_in_place, etc. “leave” which parts of the runtime. Clearlythread::spawndoes.What is the consequence of polling a runtime-bound resource from a different runtime after it’s already been created?
Having both implicit and explicit constructors available, or multiple equivalent code paths that do the same thing, leads to confusion.
TcpListenershould either take an explicitHandleor use the thread-local - but not have one constructor for each option.Handleis not a method parameter, it is not clear which methods or modules require a tokio runtime and which do not unless you try one and it panics. For this reason I have a slight preference toward explicitHandle, but I also agree it’s probably too verbose for most use cases and everyone will just useHandle::current()most of the time.tokioshould not start a background runtime automatically. This seems like too big of a footgun, such as a library or application working fine in isolation but breaking or performing badly when combined with other applications/libraries when multiple runtimes compete against each other for CPU time.Hi there, as a Tokio user I don’t have a particular positive preference about which strategy to use, but I have a very strong negative preference against adding a feature flag, because of the “spooky action at a distance” disadvantage mentioned above.
Tokio’s features are already very complicated and it’s easy to end up with things only working accidentally (your code uses some part of Tokio that’s feature-gated without setting the feature, but it works anyways because some Tokio-using library in your workspace set the feature for you). Although I am sympathetic to the concerns that led to this design, as a user I find it extremely confusing and I think that adding new features will make the problem worse.
Another point regarding thread-locals vs explicitly passing handles/contexts that I’d like to briefly point out: a lot of people think that explicitly passing arguments is “faster” than using TLS, because there’s significant overhead from accessing thread-local storage relative to an argument that was passed into a function. It’s correct that accessing the TLS var has a performance impact, but the thing that this perspective overlooks is that in many cases, the argument has to be threaded through several layers of function calls where it’s not accessed before it reaches the function where it is accessed. I’ve actually heard people report that they’ve seen noticeable performance improvements switching from
slog, where logging contexts are passed as arguments, totracing, where spans are stored in TLS, because their programs are spending a lot of time copying the explicit argument on the stack. I think something like this may have an even bigger performance impact for task spawning. The typical program probably logs significantly more frequently than it spawns tasks, so in the case of passing around a handle argument for spawning tasks, I’d guess that the ratio of context-propagating function calls to context access for task-spawning handles is even more in the favor of TLS than in the case oftracingvsslog.I support option #1 to avoid changing behavior/APIs and instead improve the error message where possible. This feels more aligned with the general behavior of the Rust ecosystem – explicit and informative error messages – instead of trying to pave the road in real-time for people who don’t know they’re going the “wrong” way.
I will try to answer some questions that have been asked here.
The main issue is not that it is polled from a different runtime — rather the issue is that you now have two runtimes. A Tokio runtime spawns a thread for every cpu core, so if you spawn two runtimes, you have more threads than you have cpu cores, which can lead to inefficiencies.
There’s also the fact that it guarantees that your IO wont be handled on the same thread as where the future is polled. It is more efficient not to cross a thread boundary. Of course, this doesn’t always happen with a single runtime either unless you use the single-threaded scheduler, but it does happen for some of the tasks.
Not really.
All threads managed by Tokio are inside the runtime context, so this includes
spawn_blockingandblock_in_place. Callingtokio::spawnand friends from inside them should work.Threads spawned by other means are not inside the context unless they explicitly enter it with
enter.Yeah well this is just the sort of thing you fundamentally have to avoid in async/await, because tasks must regularly yield control back to the executor to allow other tasks to run, and
block_ondoesn’t do that. There isblock_in_place, but it’s a pretty big footgun.I think a good error message combined with good documentation can alleviate this problem. Looking at the docs for the runtime module, I feel it’s not as clear as to what is going on under the hood as reading this issue.
enteras showed above doesn’t appear on that page either.!Sendtasks and doesn’t mentionLocalSet, nor how it can work together with it.RuntimeusesAssertUnwindSafeon user’s code, but doesn’t mention Unwind safety in the docs.Handlewill silently drop tasks if theRuntimeis no longer running, but that isn’t mentioned. The general docs forHandleare one phrase, and don’t really mention why it exists and when (not) to use it.All in all I think the OP makes it clear there is no magical solution for this. The robust type system way is to pass a
Handlearound, but it’s understandable that is not necessarily convenient and comes at a perf cost. Thus I think solid docs and error messages are paramount.Alternatively a guide level documentation on tokio.rs and a link in the API docs that really walks through the reasoning behind the Runtime design might be a good solution.