sccache: when scheduler is unreachable, sccache panics with "thread 'tokio-runtime-worker' panicked at 'Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks.', .cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/enter.rs:39:9
I’m using a local release build of sccache tip-of-tree (ec1457e8cb508c12d9f177074e20ca89f790489e). I’ve got a scheduler configured in my ~/.config/sccache/config, like so:
[dist]
scheduler_url = "http://10.0.32.3:10600"
My scheduler is usually offline, which is supposed to cause sccache to fall back to a local-only configuration (and indeed that’s what happens with e.g. the version of sccache that’s bundled with mozilla-central). But in my own local build of sccache trunk, when I build Firefox while my remote sccache scheduler is offline, I quickly get build failures due to sccache panicking, with logging like this:
[2022-04-19T20:00:45Z INFO sccache::server] Enabling distributed sccache to http://10.0.32.3:10600/
thread ‘tokio-runtime-worker’ panicked at ‘Cannot start a runtime from within a runtime. This happens because a function (like
block_on
) attempted to block the current thread while the thread is being used to drive asynchronous tasks.’, .cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/enter.rs:39:9
I’ll post a full backtrace in a bit.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17 (17 by maintainers)
Commits related to this issue
- Replace usages of blocking `reqwest` to be async instead The blocking implementation of `reqwest` now uses an internal `tokio` runtime as of `0.10`. `tokio` doesn't like having nested runtimes, so it... — committed to mitchhentges/sccache by deleted user 2 years ago
- Replace usages of blocking `reqwest` to be async instead The blocking implementation of `reqwest` now uses an internal `tokio` runtime as of `0.10`. `tokio` doesn't like having nested runtimes, so it... — committed to mitchhentges/sccache by deleted user 2 years ago
- Replace usages of blocking `reqwest` to be async instead The blocking implementation of `reqwest` now uses an internal `tokio` runtime as of `0.10`. `tokio` doesn't like having nested runtimes, so it... — committed to mitchhentges/sccache by deleted user 2 years ago
- Replace usages of blocking `reqwest` to be async instead The blocking implementation of `reqwest` now uses an internal `tokio` runtime as of `0.10`. `tokio` doesn't like having nested runtimes, so it... — committed to mitchhentges/sccache by deleted user 2 years ago
- Replace `dist-client` usages of blocking `reqwest` to be async instead The blocking implementation of `reqwest` now uses an internal `tokio` runtime as of `0.10`. `tokio` doesn't like having nested r... — committed to mitchhentges/sccache by deleted user 2 years ago
- Rewrite dist client interface to uniformly use async Fixes #1161 — committed to mitchhentges/sccache by Xanewok 3 years ago
- Rewrite dist client interface to uniformly use async Fixes #1161 — committed to mitchhentges/sccache by Xanewok 3 years ago
- Replace usages of blocking `reqwest` to be async instead (#1164) * Replace `dist-client` usages of blocking `reqwest` to be async instead The blocking implementation of `reqwest` now uses an inter... — committed to emabrey/sccache by deleted user 2 years ago
- Rewrite dist client interface to uniformly use async (#1166) Fixes #1161 Co-authored-by: Igor Matuszewski <xanewok@gmail.com> — committed to emabrey/sccache by deleted user 2 years ago
- Replace usages of blocking `reqwest` to be async instead (#1164) * Replace `dist-client` usages of blocking `reqwest` to be async instead The blocking implementation of `reqwest` now uses an inter... — committed to ahollmann/sccache by deleted user 2 years ago
- Rewrite dist client interface to uniformly use async (#1166) Fixes #1161 Co-authored-by: Igor Matuszewski <xanewok@gmail.com> — committed to ahollmann/sccache by deleted user 2 years ago
I bisected this this using
git bisect
, and identified the first-bad commit within the sccache repo.e7099d6c8ad05b7a3994698ca0da92bd8452d687 is the first bad commit – its commit message is “Migrate to stable Future trait and Tokio 0.2”
This commit involved some changes to sccache’s
tokio
dependency. It seems believable that this would’ve been where this panic started happening, since the panic is happening in tokio code.I’ve got a local reproduce now 👍, a couple tweaks were needed:
./mach build
was needed to trigger enoughsccache
calls to cause the issuerelease
mode to get an identical error (otherwise I getCannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.
instead) - this matches what you called out in your report, I was just doingdebug
mistakenlyI’ll dig in further