sccache: when scheduler is unreachable, sccache panics with "thread 'tokio-runtime-worker' panicked at 'Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks.', .cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/enter.rs:39:9

I’m using a local release build of sccache tip-of-tree (ec1457e8cb508c12d9f177074e20ca89f790489e). I’ve got a scheduler configured in my ~/.config/sccache/config, like so:

[dist]
scheduler_url = "http://10.0.32.3:10600"

My scheduler is usually offline, which is supposed to cause sccache to fall back to a local-only configuration (and indeed that’s what happens with e.g. the version of sccache that’s bundled with mozilla-central). But in my own local build of sccache trunk, when I build Firefox while my remote sccache scheduler is offline, I quickly get build failures due to sccache panicking, with logging like this:

[2022-04-19T20:00:45Z INFO sccache::server] Enabling distributed sccache to http://10.0.32.3:10600/

thread ‘tokio-runtime-worker’ panicked at ‘Cannot start a runtime from within a runtime. This happens because a function (like block_on) attempted to block the current thread while the thread is being used to drive asynchronous tasks.’, .cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/enter.rs:39:9

I’ll post a full backtrace in a bit.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (17 by maintainers)

Commits related to this issue

Most upvoted comments

I bisected this this using git bisect, and identified the first-bad commit within the sccache repo.

e7099d6c8ad05b7a3994698ca0da92bd8452d687 is the first bad commit – its commit message is “Migrate to stable Future trait and Tokio 0.2”

This commit involved some changes to sccache’s tokio dependency. It seems believable that this would’ve been where this panic started happening, since the panic is happening in tokio code.

I’ve got a local reproduce now 👍, a couple tweaks were needed:

  • This issue is inconsistent, so a full ./mach build was needed to trigger enough sccache calls to cause the issue
  • I needed to do this in release mode to get an identical error (otherwise I get Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context. instead) - this matches what you called out in your report, I was just doing debug mistakenly

I’ll dig in further