xo: Huge memory usage regression: 580 MB in 0.43 → 6300 MB in 0.44
Using Node.js 14.17.5 with NPM 6.14.14 for this reproduction recipe (npx works slightly differently in NPM 7).
$ git clone -b v5.8.1 https://github.com/zulip/zulip-desktop.git
$ cd zulip-desktop
$ npm i
$ rm -rf node_modules/.cache
$ npx -p xo@0.43.0 time xo
…
110 errors
Command exited with non-zero status 1
45.72user 0.89system 0:28.89elapsed 161%CPU (0avgtext+0avgdata 581408maxresident)k
0inputs+0outputs (0major+155120minor)pagefaults 0swaps
$ rm -rf node_modules/.cache
$ npx -p xo@0.44.0 time xo
…
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
…
Command terminated by signal 6
200.27user 7.35system 1:29.69elapsed 231%CPU (0avgtext+0avgdata 2274772maxresident)k
0inputs+0outputs (0major+717566minor)pagefaults 0swaps
$ rm -rf node_modules/.cache
$ NODE_OPTIONS=--max-old-space-size=8192 npx -p xo@0.44.0 time xo
…
148 errors
Command exited with non-zero status 1
182.59user 5.34system 1:54.32elapsed 164%CPU (0avgtext+0avgdata 6284536maxresident)k
2808inputs+0outputs (2major+1631648minor)pagefaults 0swaps
I bisected this memory usage regression to e2e715dbff660187d432a6bdad6c3da435f80194 “Simplify lintFiles (#583)”. Cc @fisker.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 4
- Comments: 28 (26 by maintainers)
Commits related to this issue
- Simplify `lintFiles` (#583) — committed to xojs/xo by fisker 3 years ago
- Limit concurrency of linting Trying to work around #599 — committed to xojs/xo by sindresorhus 3 years ago
- Limit concurrency of linting Trying to work around #599 — committed to xojs/xo by sindresorhus 3 years ago
- fix: group files by config (closes #599) — committed to spence-s/xo by spence-s 3 years ago
- Downgraded xo to avoid memory problems introduces in 0.44, see https://github.com/xojs/xo/issues/599 — committed to Mirroar/hivemind by Mirroar 3 years ago
@devinrhode2 I’m glad you’re enthusiastic about finding generic ways to improve XO’s memory usage. However, my concern is that you’re making this report harder to follow by presenting generic ideas that aren’t related to the specific regression introduced by a specific XO commit that I reported here. Would you be willing to move your ideas to a new issue? Thanks.
@devinrhode2 If you take a look at my original test case, you’ll see that it’s using TypeScript 4.3.5. You’ll also see that I already did the work of bisecting the problem to a specific xo commit. It was not introduced by a TypeScript upgrade or your other suggestions. If you are seeing a problem related to those things, I suggest filing a separate issue with a detailed test case.
I took a quick snapshot in Chrome and it seems like most of the memory is spent on TypeScript. My guess is that since we’re now creating a new instance of
new ESLint()for each file, something regarding ESLint plugins are not cleaned up and we keep initiating new TypeScript parser instances (which are huge) without releasing them.@fisker Using the dev tools memory inspector might reveal where the memory leak is.
I also tried upgrading to ESLint 8, which also didn’t help.
Latest
xoconsistently crashes with out-of-memory error (I have 8GB of RAM) for me too. Would it be possible to revert https://github.com/xojs/xo/commit/e2e715dbff660187d432a6bdad6c3da435f80194 until this issue is resolved?Note that, like I alluded to in the report,
npxin the NPM 7 distributed with Node.js 16 has a bug where it incorrectly prefers the installed version over the version explicitly provided on the command line (npm/cli#3210).But, if you account for that (and don’t forget to delete
node_modules/.cache), this reproduces equally with Node.js 16.