typescript-eslint: [2.4.0][Watch][Linux] Out of file watchers errors

What code were you trying to parse? A sample monorepo, with an extremely low number of source files.

git clone https://github.com/wixplosives/sample-monorepo`
cd sample-monorepo
yarn
yarn lint

What did you expect to happen? Lint to pass. No watching done what-so-ever, as I’m not in any watch mode of eslint.

What actually happened?

Versions

package version
@typescript-eslint/eslint-plugin 2.4.0
@typescript-eslint/parser 2.4.0
TypeScript 3.6.4
node 12.12.0
npm 6.11.3
yarn 1.19.1

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 2
  • Comments: 34 (19 by maintainers)

Commits related to this issue

Most upvoted comments

We ran into the same issue. The workaround is to set noWatch in eslintrc.js as follows:

{
  parserOptions: {
    noWatch: true,
  },
}

I’ll clarify myself a bit:

  • first request comes in
  • createProgram (not watching) & register as invalid upon next tick
  • when running the eslint cli, next request comes in synchronously and reuses Program.
  • next tick comes -> mark as invalid
  • another lint request comes (in case of an IDE), revalidate program- parse config, stat every file, verify everything matches. If mismatch, create a new program reusing the old one.

The language service does something similar: https://github.com/microsoft/TypeScript/blob/v3.6.4/src/services/services.ts#L1172

EDIT: this is less efficient than a watching program, but is sensible enough for IDEs to provide completions, so I can see how it might be speedy enough for live linting.

I removed the noWatch flag, went to node_modules, found tsconfig-parser.js, and added: console.log(path, options) right before the call to chokidar.watch.

Output It gave me:

[avi@localhost sample-monorepo]$ yarn lint
yarn run v1.19.1
$ eslint . --ext .js,.ts,.tsx -f codeframe
/home/avi/projects/sample-monorepo/tsconfig.json { interval: 2000 }
/home/avi/projects/sample-monorepo/node_modules { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/packages { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/tsconfig-paths-webpack-plugin { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo { depth: undefined, interval: 250 }
/home/avi/projects/sample-monorepo/html-webpack-plugin { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/mini-css-extract-plugin { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/tsconfig-paths { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/chalk { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/tapable { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/webpack { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/html-minifier { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/react { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/csstype { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/prop-types { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/react-dom { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/sanitize.css { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/express { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/compression { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/express-serve-static-core { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/serve-static { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/range-parser { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/mime { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/body-parser { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/connect { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/minimatch { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo/node_modules/@types { depth: 0, interval: 250 }
/home/avi/projects/sample-monorepo { depth: 0, interval: 250 }
Done in 3.57s.
[avi@localhost sample-monorepo]$ 

I then added the following to node_modules/eslint/bin/eslint.js:

// right after the require('fs')
const originalWatch = fs.watch
fs.watch = (...args) => {
    const [p] = args;
    console.log('fs.watch', p)
    return originalWatch(...args)
}

The problematic call is: /home/avi/projects/sample-monorepo { depth: undefined, interval: 250 }

It doesn’t specify depth (due to typescript telling us to be recursive), and causes chokidar to watch all files in repo. Yes, not just node_modules, but even ones inside .git. 😃

I can also see chokidar calls watch several times for the same paths, which is odd. No other abstraction is watching. I verified by reenabling the noWatch flag.

Since the current implementation uses chokidar with fsevents disabled, you might as well use require('fs').watch directly, managing which FSWatchers are open. Supporting the recursive flag only for directories shouldn’t be too hard (readdir -> stat -> isDirectory -> do it for subdirs).

I should mention I use Node 12.12 on Fedora 30 locally.

Unfortunately, the ESLint APIs have no concept of “watch”, as it’s designed to be stateless (we’re a very unique case in our stateful parsing). I had considered PRing an option into ESLint to help us here, but that ties us in to a minimum version of eslint (a lot of our users are stuck on v5).

I also considered PRing the eslint IDE plugin, but that ultimately sort of fixes one use case. Then I have do the same for the plugins for intelliJ, webstorm, visual studio, sublime, vim, etc… It’s not really a maintainable solution to try and get every single eslint IDE plugin to special case our parser. Additionally it’s also a pretty gnarly experience if we force users to pressure the plugin maintainers to add the special case.

Finally I considered PRing typescript to expose more internals so we can work around this problem without asynchronicity (or maybe without watches at all), but that would be worse, because it’d tie us to a minimum version of typescript, when a lot of our users are barely past v3.

So the million dollar question is: how can we make this work without any external changes?

@JounQin This error is not exclusive to CI environments.

I can’t help but wonder why would the abstraction that remaps typescript ast to estree has to do any sort of watching

See this for a fullish overview: https://github.com/typescript-eslint/typescript-eslint/issues/1079#issuecomment-541867845