typescript-eslint: OOM when many (>10) project configs passed to the parser
@bradzacher and @JoshuaKGoldberg - hijacking OP to say:
We are currently investigating solutions to this with the TypeScript team: https://github.com/microsoft/TypeScript/issues/35676, https://github.com/typescript-eslint/typescript-eslint/issues/1718, https://github.com/typescript-eslint/typescript-eslint/issues/2094, #6754
If you run into an OOM, please provide as much information as you can, including answering the following questions:
- Is your project open-source? (if yes - link please)
- Is your package a monorepo? If yes:
- Are you using lerna, or some other method?
- How many packages are in your monorepo?
- Do you have one
tsconfigper package, one at the root, or a mish-mash of both? - How interdependent is your codebase (i.e. are there lots of utility packages that are used by every package, or is it mainly packages with clear separations?)
- How do you build your project(s)?
- Is it via
tsc,webpack/rollup/etc, or a mish-mash of both?
- Is it via
- How many TS files are in your codebase?
- Approx how many lines of code are in your codebase?
- How many NPM dependencies do you have?
Also, please provide the following (you can leave the last two out if you’re open source):
- A complete lint run output with the
DEBUG=*env variable set?DEBUG=* yarn lint- Feel free to censor the output filenames if you’d prefer, but try not to delete any lines.
- The tsconfig(s) that you pass via the
parserOptions.projectoption - Your
.eslintrc
The more information you provide, the easier it is for us to look into it and figure out root causes.
What code were you trying to parse?
A relatively huge code base with a lot of TypeScript (using project references), JavaScript and React. Unfortunately I can’t give you a sample project because in my small demo projects it works. It only happens in that larger project that isn’t open source. I also tried to disable @typescript-eslint/recommended-requiring-type-checking but that didn’t work.
What did you expect to happen?
Not to run out of memory
What actually happened?
It runs out of memory.
<--- Last few GCs --->
start of marking 1708 ms) (average mu = 0.196, current mu = 0.093) all[49915:0x102883000] 112800 ms: Mark-sweep 2040.1 (2058.3) -> 2038.9 (2056.5) MB, 1388.1 / 0.0 ms (+ 110.5 ms in 27 steps since start of marking, biggest step 7.2 ms, walltime since start of marking 1508 ms) (average mu = 0.104, current mu = 0.007) allo[49915:0x102883000] 114242 ms: Mark-sweep 2039.9 (2056.5) -> 2039.0 (2056.0) MB, 1436.4 / 0.0 ms (average mu = 0.056, current mu = 0.004) allocation failure GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
0: ExitFrame [pc: 0x10092fbd9]
Security context: 0x295dc96c08a1 <JSObject>
1: getDeclarationName(aka getDeclarationName) [0x295d351b21f9] [/Users/me/project/node_modules/typescript/lib/typescript.js:~29254] [pc=0x1c6ba57836fc](this=0x295db0c004a9 <undefined>,0x295dbd513309 <NodeObject map = 0x295d58d4a4d1>)
2: declareSymbol(aka declareSymbol) [0x295d351b2279] [/Users/me/project...
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Writing Node.js report to file: report.20191111.151654.49915.0.001.json
Node.js report completed
1: 0x10007e743 node::Abort() [/Users/me/.nvm/versions/node/v12.13.0/bin/node]
2: 0x10007e8c7 node::OnFatalError(char const*, char const*) [/Users/me/.nvm/versions/node/v12.13.0/bin/node]
3: 0x100176267 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/me/.nvm/versions/node/v12.13.0/bin/node]
4: 0x100176203 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/me/.nvm/versions/node/v12.13.0/bin/node]
5: 0x1002fa2b5 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/me/.nvm/versions/node/v12.13.0/bin/node]
6: 0x1002fb984 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/Users/me/.nvm/versions/node/v12.13.0/bin/node]
7: 0x1002f8857 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/me/.nvm/versions/node/v12.13.0/bin/node]
8: 0x1002f683d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/me/.nvm/versions/node/v12.13.0/bin/node]
9: 0x100301f54 v8::internal::Heap::AllocateRawWithLightRetry(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/Users/me/.nvm/versions/node/v12.13.0/bin/node]
10: 0x100301fcf v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/Users/me/.nvm/versions/node/v12.13.0/bin/node]
11: 0x1002cebc7 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType) [/Users/me/.nvm/versions/node/v12.13.0/bin/node]
12: 0x1005f7725 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/me/.nvm/versions/node/v12.13.0/bin/node]
13: 0x10092fbd9 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/me/.nvm/versions/node/v12.13.0/bin/node]
14: 0x1c6ba57836fc
Versions
| package | version |
|---|---|
@typescript-eslint/typescript-estree |
2.6.1 |
TypeScript |
3.7.2 |
node |
v12.13.0 |
npm |
6.12.0 |
Config
{
"parser": "@typescript-eslint/parser",
"parserOptions": {
"sourceType": "module",
"ecmaVersion": 2015,
"ecmaFeatures": {
"jsx": true
},
"project": "./packages/**/tsconfig.json"
},
"plugins": ["@typescript-eslint", "react", "react-hooks"],
"extends": [
"eslint:recommended",
"plugin:@typescript-eslint/eslint-recommended",
"plugin:@typescript-eslint/recommended",
"plugin:@typescript-eslint/recommended-requiring-type-checking",
"plugin:react/recommended",
"plugin:prettier/recommended",
"prettier/@typescript-eslint"
],
"rules": {
"@typescript-eslint/explicit-function-return-type": "off",
"@typescript-eslint/no-explicit-any": "off",
"@typescript-eslint/camelcase": "off",
"@typescript-eslint/no-use-before-define": "off",
"@typescript-eslint/unbound-method": "off",
"@typescript-eslint/no-var-requires": "off",
"@typescript-eslint/prefer-regexp-exec": "off",
"require-atomic-updates": "off",
"react/prop-types": "off",
"react/display-name": "off",
"react-hooks/rules-of-hooks": "error",
"react-hooks/exhaustive-deps": "warn",
"prefer-promise-reject-errors": "off",
"react/no-unescaped-entities": "off",
"react/no-find-dom-node": "off"
},
"settings": {
"react": {
"version": "detect"
}
},
"env": {
"es6": true,
"node": true,
"browser": true,
"jest": true
}
}
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 77
- Comments: 139 (60 by maintainers)
Commits related to this issue
- replace tslint with eslint Convert most tslint rules to eslint, changed most inline comments in the sources as well. Some tslint rules still work, hence why you will be able to see `// tslint` commen... — committed to eclipse-theia/theia by paul-marechal 7 years ago
- replace tslint with eslint Convert most tslint rules to eslint, changed most inline comments in the sources as well. Some tslint rules still work, hence why you will be able to see `// tslint` commen... — committed to eclipse-theia/theia by paul-marechal 7 years ago
- replace tslint with eslint Convert most tslint rules to eslint, changed most inline comments in the sources as well. Some tslint rules still work, hence why you will be able to see `// tslint` commen... — committed to eclipse-theia/theia by paul-marechal 7 years ago
- Modified the way eslint and typecheck is done so that it is done at the project level instead of individual project as suggested here : https://typescript-eslint.io/docs/getting-started/linting/monore... — committed to fredbonin/aws-secure-environment-accelerator by fredbonin 5 years ago
- Cd lint prettier optimization (#852) * Modified the way eslint and typecheck is done so that it is done at the project level instead of individual project as suggested here : https://typescript-eslin... — committed to aws-samples/aws-secure-environment-accelerator by fredbonin 5 years ago
- fix linting logic the eslint plugin for typescript apparently behaves catastrophically poorly when there's multiple tsconfig.json files: https://github.com/typescript-eslint/typescript-eslint/issues/... — committed to tc39/ecmarkup by bakkot 5 years ago
- feat(typescript-estree): add experimental mode for type-aware linting that uses a language service instead of a builder program ## PR Checklist - [x] Steps in [CONTRIBUTING.md](https://github.com/typ... — committed to typescript-eslint/typescript-eslint by deleted user 4 years ago
- feat(typescript-estree): add experimental mode for type-aware linting that uses a language service instead of a builder program ## PR Checklist - [x] Steps in [CONTRIBUTING.md](https://github.com/typ... — committed to typescript-eslint/typescript-eslint by deleted user 4 years ago
- Switch to parserOptions.project = true https://github.com/typescript-eslint/typescript-eslint/issues/1192#issuecomment-1483882009 — committed to graphile/crystal-pre-merge by benjie 3 years ago
- fix: eslint config file parseOptions.project path is updated (#1008) I found out that the problem of slow down is the use of parseOption.project with multiple tsconfig files which increases usage o... — committed to documenso/documenso by Mythie 3 years ago
Thank you for the detailed answer.
That’s good to hear 😅 I create a
tsconfig.eslint.jsonthat looks like thisAnd set
parserOptions.projectto./tsconfig.eslint.jsonand now it seems to work. Great 👍But does that mean you have to optimise something for TypeScript project references? It smells a little bit like a workaround. Because I can imagine that we will have repositories with 100 packages or more in it in the future with different compiler settings. With one single
tsconfig.eslint.jsonevery project that gets linted has now the same configuration.Just adding some information for people as to why we OOM so easily and why this hasn’t been fixed and will likely not be fixed any time soon.
The crux of the issue is this: TS is designed to load exactly ONE project at a time into memory.
Your repo is organised by projects (tsconfigs). When you do a build, you ask TS to build each tsconfig. Even with project references and dependent builds - TS will build tsconfigs one at a time so that only one exists in memory at a time.
An ESLint CLI run is fundamentally different. First of all ESLint’s file ordering doesn’t necessarily match how your projects are setup. For some it will, for others it won’t. This makes it a difficult detail for us to rely on. This is an issue because it means that essentially we treat ESLint’s file access as random - we don’t know when ESLint will want to parse+lint any given file. Maybe it’ll be in an order such that ESLint lints one single project to completion before linting the next… but more than likely it won’t. This means that we end up loading multiple projects into memory at any given time. Additionally we don’t know exactly when we can drop a project from memory [1].
For the IDE usecase it’s even worse because file access is truly random and each file will be linted
0..ntimes!For sufficiently large repos loading multiple projects into memory by itself is a problem that can cause OOMs!
For other repos, a related issue can cause OOMs -
.d.tsfiles. Unfortunately TS does not provide any functionality to share heavy data between projects. If two projects load the same@typespackage - there’s now two copies in memory. This isn’t terrible as@typesare usually pretty small, however for projects with lots of@types, this really adds up!But a worse problem still is inter-repo dependencies. Imagine a repo where You have three projects that are related like this:
In this project both A and C need the types for B. TS satisfies this by loading the
.d.tsfiles for B. So if you lint all 3 projects at once - then you have.tsfiles for A, B, and C, and you have two copies of the.d.tsfiles for B [2]! So we really have 3 copies of B in memory!!! This obviously compounds with the interdependency of your repo and can be made worse if the public signature of a package depends on another package as you can load more types than you’d like.SO with all that information - what’s the solution? Well there are a few:
.d.tsfile or.tsfile. This deduplicates memory usage significantly and hence can fix OOMs. HOWEVER this has a drawback in that you need to be able to fit your entire repo into memory - which isn’t always the case.[1] There’s a potential thought that we could track the list of files that have been linted, and compare it against the list of files TS reports for a project, and dump a project when those lists are the same - but this assumes two things:
--fixcan lint the same file up to 11 times as it fixes issues and (b) symlinks can cause a file to be linted multiple times.If we dump the project out of memory before we’re actually done with it then the lint run will be hit with the cost of rebuilding the project again wastefully (which will usually be 10s of seconds).
[2] One attempt to solve this was by leveraging project references (#2094) to teach TS about the mapping from
.d.tsto.ts. However because there’s no way to share data between projects - this doesn’t really help us at all. In fact this can make things worse because we end up loading N copies of the.tsfiles - which are always going to be heavier than a.d.tsfile!In my case, the issue was caused by having
forceConsistentCasingInFileNamesset totruein my tsconfig.json. Once I turned it off, the problem was resolved. I have a 20+ package monorepo. typescript@4.9.5 eslint@8.42.0 @typescript-eslint/parser@5.59.9In DeepCrawl we solved it by not having
projectat all in the root-level ESLint config fileparserOptions. Instead we do the following for the every package separately:It made our linting very fast and solved OOM errors.
Turning on / off the rules will do nothing for your memory usage - the rules are loaded and executed on demand, and store little-to-no state. Turning on / off rules does not effect any state anywhere else either.
Adding or removing
parserOptions.project, however, tells our parser whether or not to coordinate typescript to gather type information for each project. When you run without it, we simply parse each file in “isloated modules” mode (i.e. typescript parses the file as if nothing else exists, and purposely doesn’t error on imports etc). When you run with it, for each tsconfig you provide, we will construct a ts program for it. This essentially causes typescript to do a complete parse and typecheck cycle on every single file matched by that tsconfig.Your codebase is actually pretty small by the standards of some of our users.
I think the problem is the number of separate projects you have.
When you do a lint run, we construct 13 separate programs. From my understanding, this means that there is no memory reuse between them - i.e. if package
fooandbarboth use types forreact, then both will have a copy of the types in memory. (cc @uniqueiniquity - is my understanding correct here?)If I’m correct, then this is especially bad you have a lot of interdependent relationships within your project itself, then this would cause typescript to store multiple copies of each workspace project. I.e. if
foodepends onbar, andbazdepends onbar, then you’ll have 3 copies ofbar- one forfoo, one forbaz, and one standalone used for linting itself.Again, if I’m correct with everything above, I would say that this is duplication is the source of the memory usage.
I think you should be able to fix this by creating a
tsconfig.eslint.jsonin the root of your repo, which includes all files in your repo. Essentially, take all of your tsconfigs and merge them together.This will mean that we create just a single program which encompasses the entire workspace, which means there should be a lot less (if any) duplication.
LMK if that works.
After spending a few hours debugging why my
lintcommand was suddenly experiencing crashes (including when going to previously working git commits) I discovered thattypescriptwas processing one of my node_modules folder (the folder had a non-standard name - and could not be ignored bytypescriptdefault ignore settings) that I forgot to add to theexcludeproperty in mytsconfig.eslint.jsonfile.Discovery
I discovered the cause of the crash by logging the following lines to the console (e.g. replacing
writeLogwithconsole.log):https://github.com/microsoft/TypeScript/blob/236012e47b26fee210caa9cbd2e072ef9e99f9ae/src/compiler/watchPublic.ts#L390-L394
The # of
roots(from above) had ballooned up to several thousand, where before I had <100.All the processing of these roots are what led to the program running out of memory.
Solution
I think if a developer is suddenly experiencing an “out of memory” problem, their TSConfig project (e.g.
tsconfig.jsonortsconfig.lint.json, etc.) is probably loading a lot more files than the developer expects.The solution for this situation is simple: verify that TypeScript is not loading any files it shouldn’t be by running
tscwith the--showConfigflag.Example: If a user is using the
tsconfig.eslint.jsonproject foreslint, the user can runtsc --project ./tsconfig.eslint.json --showConfig. Then, look through thefilesproperty and verify that those are the files TypeScript should be processing.Afterwords
I’m not sure if this is in the scope of the project - or even if it’s possible - but if
typescript-eslintcould detect the program running out of memory and then warn in the console that it may be caused by TypeScript loading too many files in the project and log the potential solution, that could resolve some people’s problem.@samwho please can you email me next week (james@nrwl.io) - I’d love to collaborate with you personally to work through this, I can bring in the latest project references work in Nx that Jake mentioned above and we can figure out the optimal typescript-eslint setup
https://github.com/microsoft/TypeScript/pull/58398 is merged and in
typescript@5.5.0-dev.20240507. For those with theforceConsistentCasingInFileNamesrelated issues, it’d be awesome if you could try this version of TypeScript out. Additionally, #9042 should fix it for everyone in a future typescript-eslint issue.I was having this issue and
forceConsistentCasingInFileNames: falsefixed it for me@mctrafik hello, we also have a monorepo, with 26 packages, and :
According to the latest doc, projet should be set to true for such configuration. https://typescript-eslint.io/packages/parser/#project
That solved the problem for us :
This triggered OOM after 15 seconds or so
This does not, linting takes 288ms, VSCode eslint extension works fine again.
In case this is not enough, here is our current .eslintrc.js, that makes no OOM (I removed all our custom rules, I guess they have no impact since they only override recommended ones)
and the versions of packages we currently use :
@nwalters512 your issue there is more than likely due to the fact that TS cannot share data between programs, so we end up creating duplicates of shared items. For example you will have 26 copies of your
node_modulestypes. This is why using a singletsconfig.eslint.jsondoesn’t OOM - because having just one tsconfig means there’s only one copy of everything.There is OFC going to be the problems with using one tsconfig because your packages need to be homogeneous so that they can be represented by one tsconfig - which yours obviously cannot.
Options for now:
--max-old-space-sizeWe’re playing with some ideas to try to improve things here - but the reality is that TS wasn’t designed to be used like this, so it takes some effort to get it right.
Some of the typescript folks are looking into the performance and memory side of things right now. A few bugs and optimisations have been identified, and are being worked on right now. Some of it will be released with TS3.9, some of it beyond that.
I believe most of the OOM issues stem from typescript itself - we’ve got a unique use case here, so the paths we’re hitting aren’t always well optimised yet. From the performance side of things, there are issues on both sides for the same reasons.
I’ll keep this thread updated as things come in.
Workarounds, for now:
If you’ve got many packages in your project, then consider:
tsconfig.eslint.jsonwithin which youincludeall of the files to be linted.If you’ve just got a lot of files, then consider:
In both cases, consider splitting your lint into two configs:
This setup will help keep your IDE snappy, and the memory footprint low, whilst providing a decent amount of real-time lint information. Then you can rely upon CLI runs locally and in your CI to run the extra type information rules.
Hmm. I didn’t think about this. Great work finding it.
Looking again at the tsconfig you’re providing, you don’t have an
includeorexclude. To typescript, is the same as passinginclude: ["**/*"], which means it’ll ofc match your build output folder (IIRC TS will automatically not match node_modules, so it’ll protect you from that).This is a tricky thing that we don’t exactly sync on with ESLint.
.eslintignorewill tell ESLint what files shouldn’t be linted. However because we gather type info, we also tell typescript to parse and check every file matched by the tsconfig. It means sometimes we parse more than gets linted, but it ensures we get complete and correct type information.I’ll make sure to add this to the new docs I’m writing.
If you explicitly add
includeto your tsconfig, it should fix it.+1 to what @jakebailey said: this has been a catch-all for so long, we don’t have a good way to know which issues referenced in it are still relevant. Some of what’s previously been posted was resolved by
parserOptions.EXPERIMENTAL_useProjectService. A lot more was resolved by @jakebailey and @sheetalkamat’s & the TypeScript team in general’s work in TypeScript, especially https://github.com/microsoft/TypeScript/pull/58398.Closing this out as many of the original reasons for this issue has since been resolved. ✅
If you’ve tried out everything in this thread -including the experimental project service and new TypeScript fixes- then please file a new issue that includes the information requested from this issue:
Thanks all! 💜
We were experiencing an OOM error on a relatively small monorepo (<200 files and 10 projects).
Enabling
EXPERIMENTAL_useProjectService: trueinparserOptionsas per https://github.com/typescript-eslint/typescript-eslint/pull/6754 not only eliminated the OOM error, but sped up linting dramatically.Granted, it’s experimental, but if you have the leway, this was a valid solution for us!
for balance, setting
forceConsistentCasingInFileNames: falsemade no difference to our memory usage.The difficult thing is that
So if we were to drop a tsconfig from memory only to need it later it would greatly slow down the CLI lint run. Imagine if we had to do that a few times - it could add up to 10s of seconds or even minutes extra.
And for the IDE usecase - imagine reopening a file you’ve already opened and having it take an extra 10+s to lint again - it would be a super frustrating experience! For large repos type-aware linting can already feel sluggish in the IDE so slowing it down more would suck!
We’re hoping that Josh’s experiments using the language service API will provide a better experience, or there’s the experiments in a bespoke CLI too so we can give ourselves the information and predictability we need.
Something’s very odd in your case; I tried upgrading you to 7.8.0 and setting
EXPERIMENTAL_useProjectService, and far too many files were being being loaded on the “default” inferred project when they should have been covered by atsconfig. e.g.:I’m not super familiar with how things work without the ProjectService to know if there’s a relation here.
You’re also not using project references, so that may negatively impact your performance due to reduced sharing of resources (e.g. file contents, ASTs) between projects. But, I do see that you’re using nx, so project references may start being a thing there soon.
Unfortunately this thread started in 2019 and has grown to encompass basically any OOM out there; given some of the OOMs are going to be fixed soon, but some not, I do wonder if we should find a way to split this up into more similar repros.
To those having an OOM that is “fixed” by setting
forceConsistentCasingInFileNames=false, can you try using"typescript": "npm:@typescript-deploys/pr-build@5.5.0-pr-58299-2"(from https://github.com/microsoft/TypeScript/pull/58299#issuecomment-2074000052) and see what happens?This was the solution for me too. We weren’t seeing high memory usage but we were seeing ~2 seconds per keystroke in VSCode spent on creating diagnostic files.
I added a
console.trace(${file.path} length ${fileReasons.get(file.path)?.length});call here: https://github.com/microsoft/TypeScript/blob/2e3b8af85e1f782634349139819151881369e3ea/src/compiler/program.ts#L4639 and got a lot of messages like this.@benlesh one of the unfortunate things about ESLint is that it doesn’t give us any of the information that we need to properly manage memory in a monorepo.
The only information we’re given is the path to the current file, that file’s code, and its parser options. We don’t know what files are being linted in the run or what files are left to lint. Heck, we don’t even know the difference between an IDE run (where files are randomly linted 0…n times) and a CLI run (where files are generally linted once - but actually up to 10 times with fixers) because ESLint doesn’t actually have this concept.
What this means we can’t do a simple thing like “dump a tsconfig from memory when all its files are parsed” because we have no way to know when it’s “done done”. So we just keep accumulating memory forever until node OOMs.
Sadly this is just a limitation of ESLint - in particular the CLI and not something that can be changed for various reasons outside our control.
For large monorepos generally the suggestion is to stop using the ESLint CLI directly and either use other tools or build your own CLI for your usecase. An example of the former is using tools like nx which will do “one lint run per package”. For the latter you can use the ESLint APIs and our APIs to clear or storage manually when you know a tsconfig is done.
We have had some ideas for a CLI wrapper to help here but it’s just hard to do something generically to fit every company/project’s workflows. But it might be something we reprioritise in the near future.
As an example - at Canva we have a lot of files to lint and the ESLint CLI itself is super slow at scale because it does everything serially - so we built our own CLI using ESLint’s APIs that parallelises things and leverages various assumptions about our codebase’s setup for even better perf. The result is being able to lint tens of thousands of files in a few minutes, rather than hours.
I tried the approach suggested by one of the multi-repo docs to run a shell script to run eslint once per typescript project, and found that entering debug mode
and looking for the following signature:
was very helpful. My issue was how the includes and excludes in tsconfig were being relayed into eslint (I’m generating an eslint command line from the tsconfigs), and, when there was a subtle mismatch (or relative path issue), things slowed down to an obscene crawl. Whenever “triggering folder update” happens, all was lost for me.
Hope this helps others in this thread!
A few quick reasons:
TypeScript generally only deals with one tsconfig at a time.
tscyou give it a singletsconfigfile which it does its compilation run over.For project references compiled via CLI, TypeScript only needs to know the files in project B that project A depends on.
TypeScript controls its entire process end-to-end.
TypeScript has different codepaths for different usecases.
tscuses one set of code to setup everything in a way that is optimised for the “one and done” nature of the CLI run.OTOH, for us:
TypeScript-ESLint has to deal with many tsconfigs at the same time.
eslint ., we have to load each and everytsconfigyou’ve configured.ESLint does not tell us what files are being linted in a given run.
package/foo/foo.tswill be linted. So we don’t know when or if we can dump its project from memory.ESLint does not differentiate between “one and done” runs (eg CLI) and “continuous” runs (eg IDE).
ESLint(formerlyCLIEngine) which just does a lint run - so it doesn’t tell us anything about what’s going to happen.No doubt there are some bugs in our code which could waste some memory. They would require time to find and then fix. This would be an expensive thing to action.
We haven’t fully explored project references just yet (#2094), but there is some potential here for reducing some memory usage. It’s a very complicated thing and I’m not entirely certain if it can truly reduce the memory usage for all cases. This would be an expensive thing to action as it’s exploratory work.
We haven’t really leveraged the project’s relationship with TypeScript. I’m pretty confident we could work with the TS team to design a specialised API that supports our usecase more closely than the generic APIs we’re currently using. This would be an expensive thing to action as it would require extensive collaboration between two separate projects, as well as OFC building the functionality into TypeScript.
There’s potential to improve the ESLint APIs such that they can help us differentiate between IDE/CLI runs, and also so they can tell us what files are being linted in a run. This would be an expensive thing to action as it would require writing, raising and collaborating on RFCs to ESLint.
There’s definitely ways that we can improve this problem; it’s not just something we’re forever stuck with. But it requires time, and time is something we have in very short supply.
Correct - we’re all volunteer maintainers - myself, @JamesHenry, @JoshuaKGoldberg and @armano2. Most of the volunteer time just covers basic maintenance of the project (triaging issues, PR reviews, and keeping up with TS releases), or on related projects [1] [2].
And as it’s our personal time we’re giving for free - we also prioritise features that we are passionate about and those that we think we can accomplish on a reasonable timescale.
But this would likely require weeks man-hours effort to fix - which would translate into many months of time given our time budgets right now.
Omg, I just found my problem: The generated declaration files and/or .js files in the --outdir directories.
After deleting all “build” dirs the ESLint process maxes at ~ 550 MB.
Note that I have “build/” in my
.eslintignore, so … is it a bug that the input of the type-checking doesn’t exclude the.eslintignoreentries?I think we might be able to set this as an override when we load a config so that everyone gets this? It does seem weird that this would cause an OOM given that this is the only reference to the flag in the TS codebase: https://github.com/microsoft/TypeScript/blob/f0c3c3f6eabd9bd771f6196f4cbc75eaf95e78f9/src/compiler/program.ts#L3564-L3576
(also side note
!(options.forceConsistentCasingInFileNames === false)🤮)As requested in the docs + the original comment on this issue, I’m sharing information about an OOM in my project.
Is your project open-source?
Yes. Here’s a link to a branch where the issue can be reproduced: https://github.com/PrairieLearn/PrairieLearn/tree/typescript-eslint-oom-repro. Run
yarn && make build && make lintto trigger an OOM.Is your package a monorepo?
Yes.
Are you using lerna, or some other method?
Yarn workspaces + Turborepo.
How many packages are in your monorepo?
Do you have one tsconfig per package, one at the root, or a mish-mash of both?
One per package.
How interdependent is your codebase (i.e. are there lots of utility packages that are used by every package, or is it mainly packages with clear separations?)
They are relatively separated. There are a few (e.g.
@prairielearn/postgres) that are used by several other packages.How do you build your project(s)?
Most code is processed with
tsc; client-side code is bundled withesbuild.How many TS files are in your codebase?
146, though we’re just starting our conversion, so it still has a healthy amount of JavaScript mixed in (772 files). Both TS and JS files are processed by ESLint.
Approx how many lines of code are in your codebase?
Per
tokei, 137727 lines of JavaScript and 9308 lines of TypeScript.How many NPM dependencies do you have?
Per
yarn info -R --name-only | wc -lin the root directory, 504 (this includes both direct and transitive dependencies).Additional info
The output from running
DEBUG=* make lintis 56MB and ~750k lines, so I can’t share it inline here. The project is open-source, so it should be feasible for a maintainer to reproduce the failure and get the log output.I was able to get ESLint to not OOM by adding a single
tsconfig.eslint.jsonat the root and using that as theproject. However, this appears to lead to correctness issues: the types seen by ESLint seemingly don’t reflect the existence of some types. For instance, usage of Node APIs triggers a bunch of theanyrules that shouldn’t actually apply if ESLint (or I guess TypeScript itself) was correctly using the types from@types/node. I could add those types to atsconfig.eslint.jsonfile, but that would then lead to other correctness issues in code that I specifically don’t want to use Node APIs (e.g. client-side JavaScript).Ah, so this is a new feature! Amazing. I just edited my prior post to describe it as well. It works perfectly with Nx!
Taking a naive approach may work for some setups where one tsconfig matches an entire subfolder, but will not work for many other cases because there’s no constraints in TS to enforce a specific structure.
Simply put - it is impossible to pick which tsconfig corresponds to which file without parsing the tsconfig and enumerating the set of files that it includes and excludes.
The array form of
parserOptions.projectis really optimised for whole-codebase lint runs and for advanced users that want strict control over exactly what configs are loaded.If you want a lighter experience, consider setting
parserOptions.project = true- which was released in v5.52.0, which will automatically resolve the nearest single tsconfig.https://typescript-eslint.io/architecture/parser#project
We also have a mode to further speed up cli runs that were testing out and will release soon™️
No. This is one strategy that can work for some projects, because it means that files are loaded once instead of
ntimes. But if you have a requirement to have different tsconfigs for different parts of your project - you will not be able to use this strategy. So you can either try to merge them to as few configs as possible, or you can run eslint in isolation on each project using a tool likelernaornx(or script it manually)Yeah, I’m using Yarn workspaces for that.
So ok, we will close this issue right now and you put that stuff into your own backlog and investigate that issue in the future, right? It seem to me that this will be a bigger construction site 😉
We’re experiencing this same problem in our monorepo as well. I’ve created a PR that displays the problem here: https://github.com/Budibase/budibase/pull/13642. Given that it’s open source, I hope you’re able to get all of the information you need to help debug the problem. If not, I’m more than happy to provide any extra info you might need. Any guidance on how we can mitigate this would be greatly appreciated. I’ve tried the
forceConsistentCasingInFileNameschange and saw no difference.@JoshuaKGoldberg we’re hitting this where I work as well… Basically a large monorepo that’s trying to have “one lint to rule them all” and it straight up can’t handle it.
cc @ssilve1989
We were having the same issue, and
forceConsistentCasingInFileNames: falsefixed this for us as wellThis is a good read for you to help improve performance. https://typescript-eslint.io/linting/troubleshooting/performance-troubleshooting
In a nutshell lint runs can very easily do more work than typechecking - especially if you’re using type-aware rules.
Just here to report that we also ran into OOM with a large monorepo, after adding
project: true(to allow rules that need it) and having many projects that couldn’t find tsconfig. So perhaps something related to finding the tsconfig is leaking memory.The logs only contain
node_modulespaths, I tested via:…and got no output.
@jakebailey that line was added 7 months ago…!
@bradleyayers do the logs contain non
node_modulespaths or onlynode_modulespaths?We recase paths that we see so that it matches TS’s internal casings so that we can correctly query TS - but we don’t touch source code / import paths that TS has internally - so I wouldn’t have thought it’d cause errors in
node_modules…@jakebailey good thought, you appear to be correct in that there’s a large set of errors building up, I added the log line as you suggested:
Here’s a snippet of the output:
A couple of things jump out:
Users/is swapped tousers/As an alternative, would it make sense to at least optimize watch program creation?
Edit: RTFM! Setting
parserOptions.projecttotruedoes almost exactly this, looking for the nearesttsconfig.jsonand then starting only its watch program. For the use case I described, this leads to much faster results than passing a set of projects. Who knew!In search of better DX, we’ve adopted Nx and isolated packages so they consume each other’s builds, but we’re still linting the whole codebase in one shot. This is very slow, so I’m hoping to take advantage of Nx there too:
.eslintrc.cjsin the workspace root.parserOptions.projectfrom"./tsconfig.eslint.json"to"./*/*/tsconfig.build.json".My assumption was that this would generally create one TypeScript watch program per run, e.g.
eslint packages/package-04/src/index.tswould create the watch program forpackages/package-04/tsconfig.build.json.Instead we see this:
Even a naive path-matching optimization here should improve best-case performance when linting only part of a codebase.
it’s possible to split things up and use one tsconfig per package. The important bit is that your parallelisation tooling needs to make sure it only lints the files from one tsconfig per thread. As soon as your tooling attempts to lint the files from two or more tsconfigs in one thread, then you risk OOMs on large projects.
As an example - I have a proof-of-concept for a CLI to do this (#4359) which parallelises based on the tsconfigs to prevent OOMs and improve lint times - I just haven’t had the bandwidth to push it forwards.
In terms of configuring it there is a difficulty because ESLint will merge
parserOptions.projectvalues under certain circumstances, instead of overriding. I.e. if you setproject: ['./tsconfig.json']in the root, andproject: ['./package/tsconfig.json']in the package, eslint may actually merge those intoproject: ['./tsconfig.json', './package/tsconfig.json']- which obviously completely defeats the purpose of trying to isolate things.So you need to be really careful in how you configure things.
I haven’t tested this, but I think that one way to do it might be to use an
overridesconfig to set specific files to have specific configs so that ESLint physically cannot merge anything, eg something like this (not tested, just an illustrative example):That way you can be sure that if your parallelisation tooling lints
./packages/foo/src/file.ts- ESLint is guaranteed to only collect type information based on./packages/foo/tsconfig.json, and never./packages/bar/tsconfig.jsonor./tsconfig.json- which helps isolate threads and prevent OOMs.In case we are still gathering information, I’m trying to turn on type-aware rules for Docusaurus. We are using Lerna, but for lint, we do a single run at the root.
We have 32 packages, plus a tsconfig at the root. The root tsconfig includes the test files, while the package ones don’t. The dependency graph is pretty deep—packages are interdependent. They are built by running
tsctopologically across workspaces. We have ~8000 TS files and ~700000 LOC? (Very inaccurate stats…)I started with the “one tsconfig.json per package” config guide:
Debug log
It doesn’t work with any reasonable memory limit, so I ended up removing
'./packages/*/tsconfig.json'. Now there are a few false-positives (due to using the wrong tsconfig), but at least I can move forward and fix most of the lint errors.We’re experiencing a similar issue with a relatively complex
.eslintrc.jsconfig, after updating to v5.13.0.Here’s our configuration:
We get this output in GitHub Actions CI, and only in CI:
Turning it on should be a simple one line change to your config. I’ve documented it fully in https://github.com/typescript-eslint/typescript-eslint/issues/3528
But the TL;DR is do this:
(Eventually we’ll make this the default for everyone once we’ve had some time for real world tests)
@bradzacher see my comment above. My repo is public and I’m looking into migrating to eslint using the angular upgrade command, but this is a blocker. Would you like to test your code on my repo or instruct me how to use the fix?
@bradzacher surely this is a temporary workaround that you recommend?
You mentioned how maintainers have limited time to work on such issues and I understand, but:
I am skeptical about that statement. How does
tscnot OOM when dealing with many configuration files? And given that TypeScript perfectly handles this, I see little reason to think it should be different with eslint? Besides a quirky implementation waiting to be reworked?Just trying to wrap my head around what’s discussed in this thread, as we have the same issue in https://github.com/eclipse-theia/theia. Our workaround is to invoke eslint once per package. Our monorepo contains 46 compilation configs.
@jaysoo How did you figure out that was your problem, and how did you go about determining which tsconfigs/files were at fault?
We’re running into this as well in foxglove/studio. We’ve had to continue bumping up
NODE_OPTIONS='--max-old-space-size ...'when running lint in CI.Log output: lint-oom.txt.zip
ls node_modules | wc -l=> 1707,yarn info --name-only --recursive | wc -l=> 1528No. We run in one thread and one thread only. ESLint itself is currently not designed for concurrency or even asynchronicity.
They’re working on new internals and APIs to enable concurrency, but based on the early direction we’ve seen them go in we won’t be able to leverage it because of issues like this one.
I have a monorepo that currently contains around 30 packages spread around multiple directory structures. my “project” config was: project: [ “./tsconfig.json”, “./services//tsconfig.json", "./research//tsconfig.json”, “./client//tsconfig.json", "./example//tsconfig.json”, “./lib/js/*/tsconfig.json”, “./script/tsconfig.json” ] This caused an OOM error even when using 6gb of old heap size.
When I use a single tsconfig.eslint.json in the root of repository with my .eslintrc.js:
tsconfig.eslint.json
I can lint the entire repository in under a minute. The problem is, every time I want to lint an individual package (or file), eslint has to load every TS file in my entire monorepo making it very slow.
I also attempted to use a properly configured tsconfig.json in each of my packages with the proper “references” to all dependencies. I then added an “eslintConfig” to each of my projects that extended the root .eslintrc.js and defined the ‘project’ as “./tsconfig.js”.
This worked great for linting an individual package in the monorepo but when I tried to run it from the root, it failed with a whole bunch of errors (one for every ts file in the repo);
I even tried changing the root tsconfig.eslint.json to only have project refeferences (to all projects in the monorepo) and removed the “include”. This again gave the OOM error.
Thanks for the update! Just to confirm: I have applied every workaround you listed. Therefore, I will eagerly await TypeScript 3.9 😄
Sorry for polluting this issue history with my commits, I don’t like GH doing that…
But yea, will subscribe to this issue so that I get notified once you guys find a way to fix it!