berry: [Bug] webpack-dev-server can't recompile: Can't open file: No file descriptors available

  • I’d be willing to implement a fix

Describe the bug

We are using webpack-dev-server middleware to develop locally. The initial compilation is fine as well as the production builds (via webpack-cli -p) but as soon as I change a file the dev-server’s recompilation fails with:

ERROR in ./app/learning-center/TrainingsPage/TrackTable.js
Module build failed (from /Users/nkalinov/dev/simonapp/.yarn/$$virtual/babel-loader-virtual-04f6d5f491/0/cache/babel-loader-npm-8.0.6-ccc68d8d38-1.zip/node_modules/babel-loader/lib/index.js):
Error: Can't open file: No file descriptors available
Require stack:
- /Users/nkalinov/dev/simonapp/.yarn/cache/loader-runner-npm-2.4.0-c414104c2f-1.zip/node_modules/loader-runner/lib/loadLoader.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/loader-runner-npm-2.4.0-c414104c2f-1.zip/node_modules/loader-runner/lib/LoaderRunner.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/NormalModule.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/NormalModuleFactory.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/Compiler.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/webpack.js
- /Users/nkalinov/dev/simonapp/.yarn/$$virtual/webpack-dev-server-virtual-d46dc61805/0/cache/webpack-dev-server-npm-3.10.1-34b49b6a2b-1.zip/node_modules/webpack-dev-server/bin/webpack-dev-server.js
    at new ZipFS (/Users/nkalinov/dev/simonapp/.pnp.js:27790:15)
    at ZipOpenFS.getZipSync (/Users/nkalinov/dev/simonapp/.pnp.js:32288:52)
    at ZipOpenFS.makeCallSync (/Users/nkalinov/dev/simonapp/.pnp.js:32187:17)
    at ZipOpenFS.existsSync (/Users/nkalinov/dev/simonapp/.pnp.js:31685:17)
    at VirtualFS.existsSync (/Users/nkalinov/dev/simonapp/.pnp.js:26209:24)
    at /Users/nkalinov/dev/simonapp/.pnp.js:39078:28
    at Array.find (<anonymous>)
    at applyNodeExtensionResolution (/Users/nkalinov/dev/simonapp/.pnp.js:39076:10)
    at resolveUnqualified (/Users/nkalinov/dev/simonapp/.pnp.js:39430:27)
    at resolveRequest (/Users/nkalinov/dev/simonapp/.pnp.js:39462:14)
    at Object.resolveRequest (/Users/nkalinov/dev/simonapp/.pnp.js:39524:26)
    at Function.module_1.Module._resolveFilename (/Users/nkalinov/dev/simonapp/.pnp.js:38753:34)
    at Function.module_1.Module._load (/Users/nkalinov/dev/simonapp/.pnp.js:38638:40)
    at Module.require (internal/modules/cjs/loader.js:848:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at loadLoader (/Users/nkalinov/dev/simonapp/.yarn/cache/loader-runner-npm-2.4.0-c414104c2f-1.zip/node_modules/loader-runner/lib/loadLoader.js:18:17)
 @ ./app/learning-center/TrainingsPage/index.js 47:41-64
 @ ./app/learning-center/index.js
 @ ./app/app/App.js
 @ ./app/main.js
 @ ./app.js
 @ multi /Users/nkalinov/dev/simonapp/.yarn/$$virtual/webpack-dev-server-virtual-d46dc61805/0/cache/webpack-dev-server-npm-3.10.1-34b49b6a2b-1.zip/node_modules/webpack-dev-server/client?https://localhost:8081 (webpack)/hot/dev-server.js ./app.js

ERROR in chunk main [entry]
Can't open file: No file descriptors available
Require stack:
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/web/JsonpMainTemplatePlugin.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/web/JsonpTemplatePlugin.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/WebpackOptionsApply.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/webpack.js
- /Users/nkalinov/dev/simonapp/.yarn/$$virtual/webpack-dev-server-virtual-d46dc61805/0/cache/webpack-dev-server-npm-3.10.1-34b49b6a2b-1.zip/node_modules/webpack-dev-server/bin/webpack-dev-server.js

ERROR in chunk main [entry]
simon-main.edd606415a1133a91f8b.js
Can't open file: No file descriptors available
Require stack:
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/web/JsonpMainTemplatePlugin.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/web/JsonpTemplatePlugin.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/WebpackOptionsApply.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/webpack.js
- /Users/nkalinov/dev/simonapp/.yarn/$$virtual/webpack-dev-server-virtual-d46dc61805/0/cache/webpack-dev-server-npm-3.10.1-34b49b6a2b-1.zip/node_modules/webpack-dev-server/bin/webpack-dev-server.js
ℹ ｢wdm｣: Failed to compile.

To Reproduce

The minimal information needed to reproduce your issue (ideally a package.json with a single dep). Note that bugs without minimal reproductions might be closed.

IMPORTANT: We strongly prefer reproductions that use Sherlock. Please check our documentation for more information: https://next.yarnpkg.com/advanced/sherlock

Screenshots

If applicable, add screenshots to help explain your problem.

Environment if relevant (please complete the following information):

  • OS: OSX
  • Node version: v12.14.1
  • Yarn version 2.0.0-rc.28

Additional context

Let me know if webpack conf would be helpful… Any additional tweaks needed to support recompilation/hmr?

Thanks!

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 1
  • Comments: 18 (6 by maintainers)

Most upvoted comments

I’m running into this issue with a larger app with roughly 1000 package dependencies.

Merely building the app works fine, but with watch I get the same Error: Can't open file: No file descriptors available.

After some digging, I think I’ve narrowed down the problem to watchpack, and specifically this method:

https://github.com/webpack/watchpack/blob/338675e975e798eabbfd7f99f8991ae0a0c42ad2/lib/watcherManager.js#L16-L32

One hacky workaround I’ve found is:

	getDirectoryWatcher(directory, options) {
		if (DirectoryWatcher === undefined) {
			DirectoryWatcher = require("./DirectoryWatcher");
		}
		options = options || {};
+		const key = directory.includes(".yarn/$$virtual")
+                ? "yarn_virtual" // Shared watcher for virtual packages
+                : directory + " " + JSON.stringify(options);
-		const key = directory + " " + JSON.stringify(options);
		const watcher = this.directoryWatchers.get(key);
		if (watcher === undefined) {
			const newWatcher = new DirectoryWatcher(directory, options);
			this.directoryWatchers.set(key, newWatcher);
			newWatcher.on("closed", () => {
				this.directoryWatchers.delete(key);
			});
			return newWatcher;
		}
		return watcher;
	}

This change prevents the file descriptor error – I assume the watchpack DirectoryWatcher doesn’t play nicely with PnP. I’m not sure if it really even makes sense to have filesystem watchers for virtual packages because they don’t exist on disk and are immutable. So I modified the caching mechanism so there’s only a single DirectoryWatcher for all the virtual packages, which avoids the file descriptor exhaustion problem.

+3
rtsao on Jun 9, 2020

I’m experiencing this same issue while configuring the Sanddance project inside of a react-cosmos project using yarn 2. Will post again if I can make a minimal public repository, but I’m able to trigger this condition every time on a private project.

Update

  • I’m able to reproduce the issue by running any application on another port (Jupyter notebook, Python Flask Server, an instance of node-http-proxy). The moment I close another app, the webpack-dev-server is able to recompile successfully.
+1
hydrosquall on Feb 23, 2020

Also, maybe of some importance: Sometimes, after I made a change (one change, like a new line), there are 2 recompilations happening subsequently. 😕 I don’t have this behavior in “node_modules” mode.

+1
nkalinov on Feb 4, 2020

That’s very curious - we’re using PnP at work with a pretty large Webpack app of multiple thousands of files, but we haven’t seen this error at all.

The only thing I can think of would be if the fsevents package failed to build (because then watchpack would fallback on the fs.watch function which is known to be vulnerable to file descriptor exhaustion). But if that was the case you should have a message at build time telling you that fsevents fails to build, and a log file.

If by any chance you managed to make a minimal repro I can take a look (I understand it might be difficult considering it likely happens only on your proprietary codebase), but otherwise I have no idea what might cause this, at least for now 🙁

+1
arcanis on Feb 4, 2020
Read more comments on GitHub
← berry: [Bug?] Virtual paths cause duplicate module instantiation
berry: [Bug] Workspaces foreach always exits with zero even on error →