knip: [FATAL ERROR]: JavaScript heap out of memory

Description

I tried Knip in different company projects (Nx integrated-monorepo based). In the small one it works correctly and we were able to leverage the potential of the tool 🚀 but where it would be most useful like in large monorepos (with 34_875 files or more) It fails for JavaScript heap out of memory

Command

Tried with both, knip latest 2.43.0

$ pnpm exec knip

and the canary version 3.0.0-canary.4

$ bunx knip@canary

the result is the same:

Output

Analyzing source files...

<--- Last few GCs --->

[25564:0x130008000]   326662 ms: Mark-Compact 4038.8 (4139.6) -> 4024.9 (4141.4) MB, 1334.46 / 0.00 ms  (average mu = 0.156, current mu = 0.025) allocation failure; scavenge might not succeed
[25564:0x130008000]   327980 ms: Mark-Compact 4040.9 (4141.4) -> 4027.4 (4143.9) MB, 1262.71 / 0.00 ms  (average mu = 0.104, current mu = 0.042) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Software Overview

knip: 2.43.0 or 3.0.0-canary.4
nx: 17.0.3
nodejs: 20.9.0
OS: Sonoma 14.1.1

Hardware Overview:

Model Name: MacBook Pro
Model Identifier: Mac14,7
Chip: Apple M2
Total Number of Cores: 8 (4 performance and 4 efficiency)
Memory: 16 GB

About this issue

Original URL
State: closed
Created 7 months ago
Comments: 15 (8 by maintainers)

Most upvoted comments

Thanks for all the help, @ild0tt0re! It was really helpful.

This issue was one of the reasons I was motivated even more to work on lowering memory usage and improving performance. I wrote about it (at https://knip.dev/blog/slim-down-to-speed-up/) and you can install knip@canary to try it out if you like.

Hey @webpro you did it! 🏆 👏 👏 👏 👏 👏
I was able to run the tool only with npx knip@canary without NODE_OPTIONS or other hackery.

There’s a lot more plans in this area, so even if it doesn’t work out today it might be soon. E.g. caching and recovery should be helpful features as well. Also, the size of projects like yours might give false positives, but at least, hopefully, there will be some useful output.

I cant’t wait! A reliable and high-performance tool like this is exactly the missing piece in the JS/TS community

(By the way, I couldn’t find anything related to Knip in the memory heap profile, there’s no matches when I grep for e.g. “knip” or even “typescript”. It all seems related to npm cli. I might be misunderstanding the purpose, though.)

Thanks for the check!

ild0tt0re on Dec 15, 2023

Thanks for all the help, @ild0tt0re! It was really helpful.

This issue was one of the reasons I was motivated even more to work on lowering memory usage and improving performance. I wrote about it (at https://knip.dev/blog/slim-down-to-speed-up/) and you can install knip@canary to try it out if you like.

There’s a lot more plans in this area, so even if it doesn’t work out today it might be soon. E.g. caching and recovery should be helpful features as well. Also, the size of projects like yours might give false positives, but at least, hopefully, there will be some useful output.

(By the way, I couldn’t find anything related to Knip in the memory heap profile, there’s no matches when I grep for e.g. “knip” or even “typescript”. It all seems related to npm cli. I might be misunderstanding the purpose, though.)

webpro on Dec 15, 2023

I’ve added an undocumented --isolate-workspaces flag, but I think it only uses more memory, not less. Any chance you could give it a shot? Use Knip v3 and knip --isolate-workspaces.

webpro on Nov 28, 2023

Thanks for putting this together. Unfortunately there’s not much I can do from this information alone. I need a reproduction.

Things you could try:

Give Node.js more memory with e.g. NODE_OPTIONS=--max-old-space-size=5120 npx knip - tweak the number for your machine.
Use --workspace [dir] to focus on specific a workspace.

Something I want to try:

One thing Knip does is some sort of “workspace merging”, based on tsconfig.json settings: different basePath and paths may result in more “workspace workers” internally. This results in less programs for the TypeScript backend, speeding up the process overall. I want to see whether this optimization could optionally be turned off to prevent errors you’re seeing, at the expense of speed. But again, this requires rep®os that fail to even start looking into it and try it out.

webpro on Nov 26, 2023