bazel: Regression: Compilation Unnecessarily Restricted By Scheduler in Rolling Release

Hey awesome Team Bazel,

There seems to be a regression around parallelism. With the rolling releases, I’m seeing compilation restricted to 2 or 3 threads, with the rest (I think) held up by the scheduler.

Concretely, during build, I’m seeing output like the following, and moderately slower build times to match:

bazel build //:rule
INFO: ...<omitted>
[1 / 200] 8 actions, 2 running
    Compiling <...>; 5s darwin-sandbox
    Compiling <...>; 7s darwin-sandbox
    [Sched] Compiling <...>; 24s
    [Sched] Compiling <...>; 24s
    [Sched] Compiling <...>; 24s
    [Sched] Compiling <...>; 24s
    [Sched] Compiling <...>; 24s
    [Sched] Compiling <...>; 24s

[My interpretation is that the scheduler is having the jobs wait; please correct me if I’m wrong.]

Setting, e.g. --jobs=HOST_CPUS or --noexperimental_local_memory_estimate doesn’t have any impact. (Hence different from https://github.com/envoyproxy/envoy/issues/12455.) And linking/non-compiling jobs don’t seem to count against the lower cap.

With the latest normal release, 4.2.1 and all previous releases I’ve seen, all cores are utilized, I don’t see [Sched] and the build runs a bit faster. (But not 4x faster or anything, so the bottleneck is somewhere else–the disk perhaps?)

Version details: I’m seeing this on 5.0.0-pre.20210831.2 (and for all the rolling versions we’ve tried, which are just those after the release of 4.2). I’m using the latest macOS, 11.6, and I’m seeing it all the way back at least to 10.14.

Thanks so much, Chris (ex-Googler)