electron: child_process spawn locks up renderer for 300 - 3000ms on signed app on macOS Big Sur

This has now been fixed in libuv 🎉 hoping for a release soon!

https://github.com/libuv/libuv/pull/3257

We should consider backporting this. Node.js hasn’t yet updated their version of libuv to include the fix, but it affects Electron significantly more heavily than Node, and I think the fix is probably worth backporting to supported Electron versions (>= 15.x, 14.x is currently security-only).

There are a few patches we’d need to backport in addition to https://github.com/libuv/libuv/pull/3257. Specifically, https://github.com/libuv/libuv/pull/3537 and https://github.com/libuv/libuv/pull/3441, and maybe https://github.com/libuv/libuv/pull/3512 and https://github.com/libuv/libuv/pull/3519 as well.

We managed to fix it by writing and running a native swift client that will run command on behalf of our Electron (through XPC communication). Of course the whole things needs to be secure, so runtime check of signature and all that fun stuff.

Such a hassle but what are you going to do ? People are finally switching to BigSur & Monterey, and if your application run a lot of child_process command, their experience is horrible (if usable at all).

Right now this issue is a big thorn in Electron’s foot that will keep growing as people switch to BigSur & Monterey, and the lack of commitment from big dev companies to help fix libuv is surprising. Unfortunately it seems they’d rather fork Electron and apply the original libuv fix, rather than help find a fix which would benefit the whole community.

GitKraken users are also affected.

Electron v16.2.0, Electron v17.3.0 and Electron v18.0.0 have been released with the fix.

I will email 10 funny cat GIFs to whoever fixes this bug. It’s causing some issues for our users. For now, we are working around this slowdown by spawning a bare Node.js process and then using that process to run additional spawns, but this is pretty ugly and still adds a couple of seconds to execution time due to the 1 spawn call made from Electron.

FYI, we’re seeing the same behaviour on the beta release of macOS Monterey 12.0.1.

In 5 days Electron child_process will be slugish for the last two releases of macOS, without any fix in sight.

Thanks for looking into this, to clarify some points

The fix got reverted here due to a unit-test failure in the libuv suite.

Nope it was reverted because of unittest failures in node test suite, you can refer to https://github.com/libuv/libuv/pull/3107#issuecomment-781585353 for discussions along what specific tests failed. I got most of them passing, the only one I couldn’t fix is the test.parallel/test-child-process-spawnsync-validation-errors where it tries to test uid option. If this can be fixed then posix_spawn change can be given another attempt in libuv.

VScode kept the fix in their repo

vscode does not rely on uid, gid options of child_process module, so I decided to try the patch. But over user testing, we found two new issues in the implementation,

1. https://github.com/microsoft/vscode-remote-release/issues/4630 - the manual path resolution implementation used for posix_spawn path https://github.com/descriptinc/electron/blob/2f9ed1de9f8e533c0458c83fd2bec7b4cfb01cf0/patches/node/darwin_libuv_use_posix_spawn.patch#L413-L439 differed from execvp in that it started respecting ENOTDIR errors.

2. https://github.com/microsoft/vscode/issues/118267 - cwd option of child_process module didn’t work well in some situations, turned out to be a bug in posix_spawnp implementation https://github.com/microsoft/vscode/issues/118267#issuecomment-840122005

Attaching the final version of the patch which we use in vscode use_posix_spawn_to_spawn_subprocesses_on_macos.patch.zip

Fix the issue that causes the libuv unit-tests to fail.

Yup the ideal next step is to get the node test suite passing with the posix_spawn change. Also it would be best to test with vanilla node but if you are running it for node built alongside electron then use this script https://github.com/electron/electron/blob/main/script/node-spec-runner.js to ignore tests that are known to be broken.

I am unassigning myself from this bug due to lack of cycles, but happy to assist were possible.

The above libuv issue tracks updating fork calls on linux too, which is outside our scope for current issue. I am working on creating a minimal repro and opening an issue with libuv to get a fix just for macOS, I would be happy if we are able to have fix atleast for big sur versions and above. Will update here once I open up an issue in libuv.

Electron v18.0.0-beta.6 was just released with the fix.

Any news on this ?

This is severe. However libuv is not motivated enough to fix this issue.

Vscode has kept this patch in their code.

It’s Electron being the most affected.

How about revert the revert? #27797

It looks like a few VS Code users are now hitting this as they upgrade to Big Sur: https://github.com/microsoft/vscode/issues/105446

Did some more testing with electron 16.07. Some observations:

• the app doesn’t need to be signed, also happens with npx electron .
• the slowdown happens when child_process.spawn is called
  • from the renderer
  • from main process
  • from a worker_thread
  • from a long-running spawned child_process, every time it calls child_process
• does not happen in node.js (at least v16.13.0), has to be electron.
• It doesn’t just lock up the renderer, it also locks up the main process and the whole macos (whole macos UI get sluggish, mouse cursor starts lagging)
• more prominent effect when running multiple spawn in parallel (>=4)

I created a relatively simple reproducible example:

// worker.js

const workerpool = require('workerpool');
const childProcess = require('child_process');

workerpool.worker({
  spawn: async () => new Promise((resolve, reject) => {
    const ps = childProcess.spawn('ls', { stdio: 'ignore' });
    ps.on('exit', resolve);
    ps.on('error', reject);
  }),
});

// lockup-test.js

const workerpool = require('workerpool');

const pool = workerpool.pool(__dirname + '/worker.js', {
  workerType: 'process',
  minWorkers: 'max',
  maxWorkers: 5,
  onCreateWorker: () => console.log('onCreateWorker'),
});

const spawn = (...args) => pool.exec('spawn', args);

(async () => {
  try {
    for (;;) {
      await Promise.all(Array(4).fill(0).map(async () => {
        console.log('spawn');
        await spawn();
        await new Promise((resolve) => setTimeout(resolve, 30));
      }));
    }
  } catch (err) {
    console.error(err);
  }
})();

mkdir electron-lockup && cd electron-lockup
yarn add electron workerpool

• Now run npx electron lockup-test.js and observe a complete slowdown of the system and spawn being printed very slowly.
• If you instead run node lockup-test.js, there is no slowdown and spawn gets printed out rapidly.

Because the whole system slows down it strengthens the hypothesis that this is really a macos bug (not just electron).

Running big sur 11.6. Update: now also on Monterey 12.2.1.

I wonder if there’s an easy way to include Node.js inside electron for all platforms. If so, that could be a workaround, by using a long-running node.js child_process to spawn child_processes. Though maybe it would cause issues with require when the app is packaged, because asar modules are not available in Node.js.

Does anyone know of a solution that does not involve compiling and managing a fork of Electron, or downloading a pre-built application?

What about the possibility of using the Electron shipped with VS Code?

While performing security engagements on behalf of my customers I am discovering more and more apps that are behind the latest Electron stable release due to the fact that they have to maintain a separate fork of Electron (hence upgrading is a lot more work). While this issue is not strictly security related, this is clearly affecting their security posture.

Upstream issue : https://github.com/libuv/libuv/issues/3050

@mifi Last we tested, running from main process doesn’t help on Electron 12+ (see https://github.com/electron/electron/issues/26143#issuecomment-729393664).

It looks like Electron 8 is no longer supported/maintained: https://www.electronjs.org/releases/stable?version=8#end-of-support-for-8xy-855

Might want to wait on https://github.com/libuv/libuv/pull/3257 instead. Thanks!

Oddly, with Electron 10, we saw this only with signed builds (and only on Big Sur). With Electron 13, we see this even for development builds.

Update: Electron 11 seems to behave like Electron 10 in this regard.

@deepak1556 since vscode is already applying the patch, can we download the patched and prebuilt electron binaries from somewhere?

Edit: Forked electron and committed the patch: https://github.com/KishanBagaria/electron

~~Also built electron-v13.1.8-darwin-x64.zip with CircleCI (took 2h 42m) and released it here: https://github.com/KishanBagaria/electron/releases/tag/v13.1.8~~

darwin-arm64 build was stopped midway by CircleCI because of exceeding 5 hours in the macOS Medium executor. Support mentioned there’s no way to increase the 5 hours timeout. Access to the macOS Large executor (which should take < 5 hours) will be given only for a large annual contract. Access to the CircleCI local runner where you can use your own Mac will be given for the same large annual contract.

After battling with CircleCI config, I was able to build Electron v13.2.1 with the patch

@dennisameling has built v14.0.1 with the patch: https://github.com/dennisameling/electron/releases

I and @dennisameling built patched v15.3.0: https://github.com/KishanBagaria/electron/releases/tag/v15.3.0

FWIW the issue also exists with electron 9.4.4, but 8.5.5 does not have the issue. It’s blocking me from upgrading my app from electron 8.

dependabot keeps nagging about upgrading my electron <9 apps due to this vulnerability. It seems like electron 8 did not get a patch for that vulnerability.

Is there any chance of making an exception to the support cycle and backporting further, back to 11 and up?

The reason I say this, is this issue has been blocking upgrades in projects for so long that there are now other issues preventing upgrades to the latest releases such as context aware modules support, and the dropping of Spectron. It’s become a mammoth task to move so far forward in one jump.

Nice!Delicious!

they’re working on fixing this here: https://github.com/libuv/libuv/pull/3257

This is also present in Electron v13. Is there a workaround other than disabling hardened runtime and downgrading?

I can reproduce these results. Here are my ranges of spawn durations (in ms) on Catalina & Big Sur machines:

macOS Catalina 10.15.7 (19H2)

macOS Big Sur 11.0 Beta (20A5395g)

Both of these samples were taken on a MacBook Pro (15-inch, 2016), 2.9 GHz Quad-Core Intel Core i7, 16 GB RAM They were using Node.js 12.18.3, Chromium 87.0.4280.11, and Electron 11.0.0-beta.13.

@KishanBagaria In my test, rerender is ok, but main is not ok.

@MMcKester https://github.com/KishanBagaria/electron/releases/

@deepak1556 Where can we keep track of the patch used in VScode (and any guide to apply it). Thinking of trying it out for hyper. Thanks

The support cycle isn’t something that we can make exceptions to easily even if we wanted to (which in this case I’d still be against).

Older versions like 11 were built using CI images / VMs / compile clusters that aren’t running / available anymore and modifying / spinning up infrastructure just for a one off release would take an unreasonable amount of effort. 15-x-y is as far back as this one will go.

Awesome 🚀 Thanks a lot! What electron version will have the fix included?

any updates on this issue? It seems as everything was fine when I used version 11.4.7 (on BigSur, both signed and unsigned). But on versions 12 and 13 every spawnSync / execSync locks the renderer…

Can confirm the same behaviour. Downgraded to 11 and everything is fine but seems like a werid fix xD

perhaps it will also happen with node.js if signing the node binary

Doing a little investigation of workarounds today and have some notes. Possibly no new information here as I’ve read most of this scattered around various issue chains, but now just compiled into one comment.

I’m currently on electron 8, and have tried electron 11.5 and 13.6 today. Since I use node-pty and they have their own problems right now, 14/15 are incompatible with my code so I have no results from that, but I don’t gather they’re any better than 13 for the subject of this issue.

The choice of electron versions may feel a little un-satisfying. These are just the two I’ve seen referred to most and where the problems supposedly differ.

Possible ways & workarounds to spawn a process

There are a few suggestions I’ve seen floating around to work around the issue:

1. (baseline) Spawn within a renderer/BrowserWindow
2. Spawn within the Main process and use Electron IPC to communicate with the renderer
3. Spawn within a process forked off from Main (via child_process.fork) and use child_process IPC to communicate to Main and then Electron IPC to the renderer (have seen it suggested as a possible workaround in later electron versions)

Which workarounds… work-around?

I have written spawns which do git status for each of the above workarounds:

• Electron 8: both main and renderer spawns run around 10-15ms, in both development and production builds.
• Electron 11: both spawns perform around 10-15ms in dev build, renderer spawn goes to 300ms in production build.
• Electron 13: both spawns perform around 300ms in development
• Electron 13 with ‘fork’ from main: 300ms in development

So for E11 it seems that using the main process and IPC, that’s a fine workaround. For E13+ the workaround breaks and doesn’t have a clear replacement.

As a developer I find it hard to justify investing in a workaround which adds complexity and I know is just going to break when I eventually need to get off E11, especially since electron 11 is already out of support at this point

Guess electron 8 is the place to stay until there’s some movement with a real fix 😪

Two other workaround solutions:

• Running spawn instead from the main process (not renderer)
• Downgrading to electron 8

Today I had some time to look into the issue. Because the issue touches Electron, node, libuv and the discussion is scattered over various bugreports and PRs I would like to use this post here to quickly recap the current situation. Please correct me if anything is wrong.

Chronology

1. The spawn performance bug got reported
2. The fix #27026 got commited to libuv that uses posix_spawn.
3. The fix got reverted here due to a unit-test failure in node, see here
4. VScode kept the fix in their repo
5. Two new issues occurred in VScode, check out for more info two posts further down here.

Recap

Last working version for me: Electron v8.5.5 First broken version for me: Electron v9.0.0-beta.1

According to this release table here are some version details:

The error seems to be in libuv. Since libuv is located in deps/uv/ of node we can diff the two node versions v12.13.0 and v12.14.1: https://github.com/nodejs/node/compare/v12.13.0...v12.14.1

@seb-mtl

Node version: https://github.com/electron/electron/blob/main/DEPS#L19-L20 Patches we apply on top of Node: https://github.com/electron/electron/tree/main/patches/node

I found the posix_spawn use quite promising despite the failure in the test suite of libuv. I would like to dig more into this to see what’s happening there. Since Electron brings its own(?) node implementation I was expecting a specific node/libuv version somewhere in the Electron project. Could someone share some brief insights on this?

@cupcakearmy I hope that it will be fixed before Electron v11 will reach end-of-support 🥲

As a workaround, I downgraded Electron to version 11.4.6, and the slowness is gone. When running version 12, CPU usage would often go to 100% for a second or so when spawn was called.

The weird thing was that for me even when calling spawn from the main process the UI would freeze. I also tried tried calling from a web worker, same problem.

My colleague has been working on a patch for libuv. It currently has a breaking change, but depending on how you use child processes it may not affect you (see PR below for more details).

We’ve done a custom Electron build with this libuv patch, which has solved the Big Sur issue for us. If anyone else is experiencing the issue, you can try patch https://github.com/electron/electron/pull/27026

P.S. Many thanks for https://github.com/electron/build-tools, that made the build process simple!

Per this comment https://github.com/microsoft/vscode/issues/105446#issuecomment-729390368 - I just confirmed that the test project has the slowdown on the browser (main) process when using the latest nightly electron build.

0: spawning "ls" process
0: spawn launched in 369 ms
1: spawning "ls" process
1: spawn launched in 390 ms
2: spawning "ls" process
2: spawn launched in 415 ms
3: spawning "ls" process
3: spawn launched in 389 ms
4: spawning "ls" process
4: spawn launched in 421 ms
5: spawning "ls" process
5: spawn launched in 402 ms
6: spawning "ls" process
6: spawn launched in 397 ms
7: spawning "ls" process
7: spawn launched in 420 ms
8: spawning "ls" process
8: spawn launched in 419 ms
9: spawning "ls" process
9: spawn launched in 420 ms

That’s without even signing the app - just running yarn start.