electron: [Bug]: memory limitations introduced with Electron 14+

Aaand it’s now live in latest Electron 21 / Chrome 106 as of today. For all 2 of you still following this issue 🥲

For me, I’ve never been happier to npm update electron.

Ok @ckerr thanks for confirming the cause. This is obviously a disappointing non-resolution though, and something of a shock to those of us who have built up large projects relying on Electron not expecting this undocumented major change. I do wonder how much of a “rare” use case this really is. Something like a Figma, I would imagine, could hit up against these limits in more complex hi res projects.

I will eventually have to look into manually building Electron, but until then it’s #Electron134Lyfe.

@jssdgit or anyone else interested, I’ve made this experimental build available here https://github.com/marcelblum/electron-windows-memory-unlimit

This is becoming a major issue.

The latest versions of typescript editor services do not work with the tsserver-bridge workaround. Our mono repo will not be able to upgrade to the latest version of typescript. There is going to unfortunately be a lot of finger pointing between electron, vscode, and typescript as to who should solve this problem.

I think supporting a higher heap size is a reasonable request. Please, please, please consider re-opening this issue and fixing the problem.

Thanks for your input @nornagon. I’ve read that v8.dev blog post several times over the past couple months by now, and while some of the lower level concepts are beyond my knowledge area, my takeaway is that this feature goes a long way towards memory conservation for the typical browser user - ie 20 tabs open of Facebook & NYT etc. each with iframes, ads, videos, bloat - while imposing some limits that are mostly irrelevant for same user. Whereas with Electron, where the dev is here for the desktop app development framework that just happens to be a browser engine with all the unfun limits removed, the changes introduced by pointer compression seem less beneficial, even regressive. That said, I understand the far reaching ramifications of disabling pointer compression in Electron (though it could be argued Electron was doing just fine without these “benefits” through v13!).

As far as use case, sure your typical note taking or chat app shouldn’t need that much memory, but anything juggling lots of high res media can reasonably hit the new limit, especially those using some of Chromium’s greatest newer features that enable video & audio editing/generating, complex games, content creation tools, etc. To give a more specific example, in my case I’m using the Web Audio API which has grown quite powerful, but doing realtime processing at low latency requires loading entire audio files to RAM. Chromium supports up to 384kHz multichannel 32-bit audio which takes up gobs of memory. In practice obviously no one would use 384kHz audio on a website and even most pros don’t record beyond 192kHz. But Electron offers the promise of tapping these capabilities. Lets take a more realistic example and say I wanted to create/edit a song mixing 15 stereo tracks @32-bit/96kHz, each 7 minutes long. The cool thing is, Electron/Chromium can do this and do it well! But not even including a UI, that’s 15 * 7 * 60 * 2 * 96000 * 4 = 4.8GB needed, and if we want to go multichannel or higher quality sampling rate or longer/more tracks, that will inflate tremendously. Note: Part of this stems from the inflexibility of the Web Audio API as it doesn’t offer an efficient way to only partially load sound files (have to decode the entire file to RAM first) or stream from disk (can only be done with a MediaElementAudioSourceNode which brings limitations), but I also think those types of features are understandably out of the scope of Web Audio, and shouldn’t be as much of a concern for a desktop app anyway. Bottom line, if you want to be able to quickly bounce around a multitrack audio project in realtime while keeping things in sync, those tracks pretty much all need to be fully in memory for things to work smoothly.

And if you wanted to add visual media into the mix, the memory needs increase even more. On the one hand I get that it’s asking quite a bit of Electron to do these things and I’m not trying to create a full DAW or video editor with Electron - but on the other, the Chromium devs have brought these powerful features to their engine, and they are just sitting there begging to be used and pushed to their limits… 😉

If you don’t mind my picking your brain a bit further, your comment has raised several questions for me to help better understand the issue and limits:

4GB heap limit

Could you elaborate on what this means in practice and how that jives with my test results above, which show an 8GB limit (not 4GB) per process in v14+?

Node.js disables pointer compression at build-time by default

But are you saying this is not the case in Electron’s implementation of Node? My test results were that even Electron’s main process has its own 8GB limit in v14+. I understood that the main process is an instance of Node.js, is that incorrect? Or does Electron’s bundled version of Node share the same build of V8 with Electron’s Chromium, with pointer compression enabled?

ArrayBuffer data is not stored on-heap, so it’s possible to hold more than 4GB of data if it’s in ArrayBuffers.

But my tests above do just that - use Buffer.alloc() to fill memory with dummy data - and still hit this 8GB limit. In what context would ArrayBuffers not be constrained by these limits?

If anyone’s curious, this upstream change (Chromium increase in hardcoded memory limit on Windows from 8GB to 16GB) is now live in Chrome beta channel as of circa v106. You can test for yourself running latest Chrome Canary with --no-sandbox. I see that latest Electron@nightly v21 is using Chromium v105.0.5187.0 so it does not include this change. Hopefully by the time Electron 21 graduates to release status it can upgrade to Chromium >106.

@nornagon I got the impression @MarshallOfSound is strongly against this since it would require patching Chromium, but I’d be happy to if there is agreement on that being the best way forward. But in the mean time it’s looking like there’s a decent chance the Chromium team will be bumping the limit to 16 on their end. If that were the case the only question that remains is if Electron would patch it to increase that further.

I was able to bump the Windows memory limit in a custom build, more details here. TLDR: The culprit wasn’t pointer compression, rather a limit in Chromium which only applies to Windows sandboxing but was (seemingly clumsily) hardcoded to be in effect regardless of sandbox status. And I guess Chromium devs also were trying to protect js devs from themselves and put in a hard limit on memory leaks from bad js code.

Some thoughts:

Pointer compression is not the limiting factor in most high memory use cases because most activities that that use a lot of memory - media loading, buffers for binary data, etc. - are off heap. I guess if you wanted to have a >4GB string variable, or a giant json object with millions of entries, that’s where the pointer compression limits would come in (?). But I think for most of use cases for the concerned parties on this thread the problem can be traced to those hardcoded Chromium limits introduced around Chromium 93 which made it into Electron 14, and then were modified for macOS around Chromium 98.0.4758.141 / Electron 17.3.

Proposed solution: Either request the change upstream so that the limit in Chromium is only enforced if sandbox is on, which would trickle down nicely into Electron, or Electron should patch its version of Chromium with this change (and I’d vote for removing the limit entirely rather than 16gb if there are no bad side effects).

Note with sandbox on in Windows it’s impossible to exceed 8gb due to platform limits. But with these changes Electron devs would be able to sidestep these limits by using nodeIntegration on / sandbox off.

Can someone provide a pointer (pun intended) to instructions on HOW to build your own custom electron with v8_enable_pointer_compression = false set? I have been struggling with this for days, especially given that it takes 8 hours to compile and link electron on my machine. Here’s what I did, and it didn’t work:

(1) gclient sync -f (2) gn gen out/Release --args="import(\"//electron/build/args/release.gn\")" (3) Manually edit //v8/BUILD.gn to set v8_enable_pointer_compression = false and v8_enable_pointer_compression_shared_cage = false (4) ninja -C out/Release electron

I know it didn’t work because I then ran a canned script to allocate fixed length arrays. That canned script can get to 16 GB on older versions of electron (e.g. 12.2.3) but only gets to 8.3 GB with the custom build (off of nightly) that I do myself.

Thanks!

I don’t suppose it would make sense to work with typescript to have it use less memory? 😅 8GB is quite a lot.

But: that’s also a pretty compelling real-world use case. Something that occurred to me would be to see if it would be possible to choose between pointer-compression and non-pointer-compression at isolate construction time, rather than compile time. I’m not sure if that’s technically feasible; I’ll try to check with the v8 team.

We are running into this limit because VSCode uses electron to run typescript, and we have a large mono-repo. We are working around this with a package called tsserver-bridge, which replaces tsserver.js with a thin wrapper that runs node (which does not have pointer compression on by default). Unfortunately, that package relies on the presence of a node_modules directory, which will block our migration to yarn plug’n’play. Without tsserver-bridge, many of our developers complain of tsserver being extremely unreliable, crashing due to hitting this memory limit.

Related: https://github.com/microsoft/vscode/issues/127105

@marcelblum Silly of me to expect that their releases page should be updated. Thanks so much for the tip! Just updated my electron and confirmed first hand that we can (once again) allocate 16 GiB worth of typed arrays. Onward and upward!

@marcelblum nice debugging. Perhaps consider submitting a PR?

An interesting recent finding (at least interesting to me 😄): On macOS (both arm & intel), since Electron 17 through to nightly, the limit has increased to 16GB. This is much less of a hindrance than 8GB, needless to say, as it’s a lot harder to hit 16GB even with hi res media use. I wonder what led to that change, which presumably has nothing to do with pointer compression. Meanwhile on Windows Electron remains very hard limited at 8GB on all versions since 14. I would go as far as to say that this is now primarily a Windows-only problem from where I sit.

Revisiting this after many months and it’s dismaying to see that many fundamental questions on this front remain unanswered/undocumented. Chief among them: What are the exact limits and in what usage contexts and what platforms? There is a lot of FUD surrounding this subject to be pieced together from various threads and incomplete info sources (4GB? 8GB? 16GB?). Why do different platforms seem to have different limits? Has anyone been able to successfully disable pointer compression in practice and get an observable change in memory handling? I’ve tried a bit and I haven’t, posting WIP results at #35032. It’s a difficult problem to iterate over given the 8+ hour compile time.

Unfortunately we don’t really have any great options here. Pointer compression is a compile-time option enabled by default in Chromium, with a wide variety of implications, one of which is the 4GB heap limit. You can read more about the technical details here: https://v8.dev/blog/pointer-compression. Further, there are new & upcoming security features in V8 which depend on pointer compression being enabled, such as the V8 sandbox.

Node.js disables pointer compression at build-time by default: https://github.com/nodejs/node/blob/3f403e5e285f350620f7d66329fa890806f7fc7a/configure.py#L445-L449

Options are:

1. Do nothing. Leave pointer compression enabled, dropping support for >4GB heaps.
2. Disable pointer compression. This would significantly increase memory usage, reduce performance, and reduce security, as well as diverging us from Chromium.
3. Ship a secondary set of builds with pointer compression disabled. This would double our already sky-high release build time and asset count.

In my opinion, (2) and (3) are unacceptable tradeoffs for the benefit provided.

Would you be able to tell us a little more about your use cases that require >4GB heap space? There might be ways to rearchitect things to fit in a smaller heap; for instance, ArrayBuffer data is not stored on-heap, so it’s possible to hold more than 4GB of data if it’s in ArrayBuffers.

Exact Chromium version where this landed seems to be 106.0.5221.0.

Still haven’t grokked why the same can’t be done for Windows as this reads like “a Googler was getting crashes running Figma in Chrome on Mac so we increased the limit for them” 🤦‍♂️

Welcome to crbug 😅 I’m reading that as the 8GB limit on Windows is enforced by the sandbox not by the heap limitations in V8. E.g. try allocating the large buffers in the main process, if you can then the V8 flags aren’t being applied anymore. The limit from the sandbox is a separate problem that you could probably validate by running with --no-sandbox

For reference, here’s a thread from the Node TSC in which they ended up not really making a decision about enabling pointer compression: https://github.com/nodejs/TSC/issues/790

Unfortunately it looks like making pointer compression a runtime-toggleable flag is technically infeasible; a v8 dev told me it would mean gating every memory access behind a load and branch. So it looks like we are back to square one: either we have it enabled or we don’t.

One possible option for tools that really need a jumbo-sized V8 heap would be to bundle Node with the app. I assume that’s more or less what @MarkCelani-at is talking about with tsserver-bridge. It’s unclear to me what the node_modules directory has to do with that, but it seems like a solvable problem. Node’s v8 is compiled with pointer compression disabled, so it would be fine to expand the heap size there.

I’m quite reluctant to disable pointer compression project-wide. As such, I’m not sure I want to reopen this issue just yet. It seems like for many use cases there exist feasible workarounds, such as running a separate Node process, or building a custom version of Electron (which I know VS Code does), so I’m not yet convinced that the right option is to disable pointer compression by default.

This doesn’t affect our project, but it seems like this issue should not be closed.

I’m not @nornagon but I believe this is due to the V8 9.2 pointer compression changes. I tried out @marcelblum’s test repo on Electron 14.0.0-nightly.20210413 (which was Electron’s last pre-V8 9.2 release, using 9.1.127-electron.0 and then the next nightly from two weeks later, 14.0.0-nightly.20210426 (V8 9.2.2-electron.0). 20210413 can grow past 8 GB, while 20210426 hits the limits reported here.

Since pointer compression is a compile-time option, I don’t believe there’s any way to work around this limitation without rebuilding Electron with pointer compression disabled. I can’t speak for the entire project, but IMO it’s unlikely that alternate builds would be made available for this: doubling the number of binaries that Electron puts out doesn’t seem tenable for a use case that few apps are asking for.

Given all that… with enough CPU time, it’s not that difficult to roll your own Electron. If crossing the V8 memory ceiling is a hard requirement for your project, my suggestion is to DIY an electron build with v8_enable_pointer_compression disabled in BUILD.gn.

I’m closing this issue because I don’t think there’s any further action that the project can take on this, but @nornagon (or any other maintainer) feel free to re-open if you feel differently!

FWIW, we gave up on the custom build approach because the performance degradation of the build without pointer compression was too great (speed reduced by more than half for some app-specific benchmarks). Although, some day soon I would like to publish the build scripts we set up because I think they could be helpful to others.

Hi folks — if there is anyone here who has experience making custom Electron builds to remove these limits I would appreciate some advice. I have tried making a custom build with kPoolMaxSize = 1024 * kGiB and v8_enable_pointer_compression = false, and bumping --max-old-space-size and --max-semi-space-size at runtime. This does seem to allow the heap size to grow larger than 16GB on macOS, but on Ubuntu (x64) the process still crashes around 16GB. On both platforms ArrayBuffer allocations start failing around 16GB as well. Is there some other limit in play that I can tweak elsewhere? (FWIW I’ve posted the same question on https://bugs.chromium.org/p/chromium/issues/detail?id=1243314)

I have successfully made custom build of electron 22 and it works well for now, only v8_enable_sandbox disabled. if i disable both v8_enable_sandbox and v8_enable_pointer_compression, electron crashes.

well there’s this and this, and I recall finding more info scattered across posts in the v8 blog. Just be aware that the v8 sandbox and Chromium sandboxing are separate things not to be confused.

@jssdgit Not a custom build or nightly, it’s in latest electron stable. Looks like the https://www.electronjs.org/releases page isn’t getting updated for some reason but check https://releases.electronjs.org/ and https://www.electronjs.org/blog/electron-21-0

As I understand the latest docs, sandboxing in Electron is off by default. Certainly if you are instantiating a BrowserWindow with webPreferences: {nodeIntegration: true, contextIsolation: false} sandboxing is off.

edit: actually looks like this was just changed with the new release of electron today, sandboxing is now on by default in electron 20 unless nodeIntegration is on. But note Chromium sandboxing and v8 sandboxing are 2 totally different things.

In agreement with the maintainers above, flipping the default has costs that do not outweigh the benefits. For now workarounds similar to vscodes are your path forward if these memory limitations actually impact you.

VSCode has workarounds for this issue https://github.com/microsoft/vscode/issues/127105#issuecomment-1124504462 using vanilla node environments, and we won’t be flipping the pointer compression default for the desktop application to align with Chromium, Electron.

@Slapbox I wouldn’t call it bare bones, the numbers I noted were what I saw after all assets of my app were loaded, that includes several js libraries & native node modules, and a ui with dozens of svgs and several canvas contexts. The additional BrowserWindows were essentially duplicates of the first with the same full ui & new instances of all the same libraries & modules. Obviously every app is different but I wonder how many Electron apps even use more than 1 BrowserWindow. But granted, there is more of an argument for a benefit for the multiple window use case, and I know @nornagon mentioned V8 security features that depend on pointer compression and I don’t want to discount that.

Also see https://github.com/electron/electron/issues/22705#issuecomment-887744665 CC @nornagon

V8 pointer compression puts a 4GB maximum on the heap size. So on machines with >16GB, there is no way to increase the heap size from the default.

So it looks like after the pointer compression joined the game the only way to increase the limit is to compile own @electron bundle from sources with disabled “pointer compression” stuff and then set the max-heap-size / max-old-space-size arg?