next.js: jest-worker breaks building the project with "write EPIPE" error on macOS

This option in next.config.js seems to resolve the issue for us:

module.exports = {
  experimental: {
    cpus: 1,
  },
};

It affects how new Worker from jest-worker is created:

https://github.com/vercel/next.js/blob/d60c95cb57497396d534eb8dd6fce9a192fc22fe/packages/next/build/index.ts#L675

experimental.cpus is also used for other things like setting the parallelism for webpack build, see e.g. this full-text search:

Specifically for jest-worker, it sets numWorkers in jest-worker here:

const workerPoolOptions: WorkerPoolOptions = {
  numWorkers: this._options.numWorkers ?? Math.max(cpus().length - 1, 1),
  //...
};

On my machine which is MBP 16’’ 2019 with a 6-core Intel i7, cpus().length is 12 so jest-worker would then use 11 as numWorkers. For experimental.cpus, I can use the values 1, 2 and 3 for the build to succeed ✅. 4 and above lead to that EPIPE error 🚫.

The same problem! My build is running in docker and periodically crashes with this error!

Next.js: 11.0.1 Node.js: 14.17.3 Yarn: 2.4.2 (PnP)

events.js:352
#9 98.36       throw er; // Unhandled 'error' event
#9 98.36       ^
#9 98.36 
#9 98.36 Error: write EPIPE
#9 98.36     at ChildProcess.target._send (internal/child_process.js:839:20)
#9 98.36     at ChildProcess.target.send (internal/child_process.js:710:19)
#9 98.36     at ChildProcessWorker.initialize (/app/.yarn/cache/jest-worker-npm-27.0.0-next.5-6f53fab806-7564f4c78c.zip/node_modules/jest-worker/build/workers/ChildProcessWorker.js:181:11)
#9 98.36     at ChildProcessWorker._onExit (/app/.yarn/cache/jest-worker-npm-27.0.0-next.5-6f53fab806-7564f4c78c.zip/node_modules/jest-worker/build/workers/ChildProcessWorker.js:275:12)
#9 98.36     at ChildProcess.emit (events.js:375:28)
#9 98.36     at ChildProcess.emit (domain.js:470:12)
#9 98.36     at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)
#9 98.36 Emitted 'error' event on ChildProcess instance at:
#9 98.36     at internal/child_process.js:843:39
#9 98.36     at processTicksAndRejections (internal/process/task_queues.js:77:11) {
#9 98.36   errno: -32,
#9 98.36   code: 'EPIPE',
#9 98.36   syscall: 'write'
#9 98.36 }

Downgrading to version 10.2.3 solves the problem.

And yes, we’re on yarn 2 and PnP.

Just want to report that we had to give up on PnP, unfortunately 😕. There is something quite strange going on between Next.js, webpack, Yarn PnP and who-knows-what. This is a cross-post from our internal issue:

---

After avoiding the EPIPE error, builds started showing PnP errors like this:

$ YARN_NODE_LINKER=pnp YARN_ENABLE_GLOBAL_CACHE=false yarn build
info  - Loaded env from /Users/borekb/dev/shoptet/sofa/packages/citygolf/.env
warn  - You have enabled experimental feature(s).
warn  - Experimental features are not covered by semver, and may cause unexpected or broken application behavior. Use them at your own risk.

warn  - Minimum recommended TypeScript version is v4.3.2, older versions can potentially be incompatible with Next.js. Detected: 4.1.5
info  - Checking validity of types
info  - Creating an optimized production build
Failed to compile.

ModuleNotFoundError: Module not found: Error: Can't resolve 'encoding' in '/Users/borekb/dev/shoptet/sofa/packages/.yarn/cache/node-fetch-npm-2.6.0-29c7a53447-394197397d.zip/node_modules/node-fetch/lib'


> Build error occurred
Error: > Build failed because of webpack errors
    at /Users/borekb/dev/shoptet/sofa/packages/.yarn/$$virtual/next-virtual-6178b5f3d3/0/cache/next-npm-11.0.1-c40bc7f380-552185654b.zip/node_modules/next/dist/build/index.js:15:924
    at async Span.traceAsyncFn (/Users/borekb/dev/shoptet/sofa/packages/.yarn/$$virtual/next-virtual-6178b5f3d3/0/cache/next-npm-11.0.1-c40bc7f380-552185654b.zip/node_modules/next/dist/telemetry/trace/trace.js:6:584)

I originally started dealing with those one by one – either installing it to our package.json files or adding them to .yarnrc.yml but then I stopped and explored this specific “error” coming from node-fetch.

Optional peer dependencies, PnP and webpack

node-fetch has this in their compiled output:

let convert;
try {
  convert = require('encoding').convert;
} catch (e) {}

That’s a correct way to try to load an optional peerDependency, and plain Node.js + Yarn PnP don’t have any problem with that: try creating a small project with index.js and a single dependency on node-fetch, this source code:

const fetch = require('node-fetch');
console.log('done');

and run it with yarn node index.js. It will work fine because PnP will throw an error inside a try/catch block, which will be swallowed by that try/catch and everything will work fine:

However, when run via webpack, this fails, so it’s for further exploration in [internal issue].

BTW, by default, next build in both apps will fail with the error above. I also tried adding this to .yarnrc.yml:

packageExtensions:
  'node-fetch@2.6.0':
    peerDependencies:
      encoding: '*'
    peerDependenciesMeta:
      encoding:
        optional: true

Then it fails with a different error:

I still think this isn’t right – it’s an optional peer dependency so it shouldn’t be required to be provided by one of its ancestors.

Loose mode as a workaround

Added in [commit hash], packages now use loose mode.

Build now passes but generates a lot of warnings:

These warnings can be silenced with PNP_DEBUG_LEVEL=0 in Yarn 3 (we’re currently on 2.4.1) but better would be to find out why we’re seeing failed webpack builds on a perfectly valid code.

---

Another strange thing is that trying to use webpack 4 produces very different output from webpack 5, about which I’ve posted https://github.com/vercel/next.js/discussions/27045.

All in all, Next.js 11 + Yarn PnP seems quite broken at this point.

We’ve been fighting this as well as part of our attempt to upgrade to Next.js 11. We’re not entirely sure what’s causing it but some observations are:

• It only happens with Yarn’s PnP mode, node-modules linker is fine.
• We tried Node.js 16 (normally, we’re on v14) – no difference.
• We’ve tried updating all dependencies in our packages to “latest” and to use next@v11.0.2-canary.7, this seems to almost solve the issue – we see the EPIPE error much less after that but it still happens (I’d say about 10% chance; before, it was like 95%).
• If I have .next folder from a previous successful run, yarn build will typically succeed. When I delete it (rm -rf .next), I get EPIPE error again.
• We’ve tried pinning jest-worker to a newer version – "jest-worker": "27.0.6" in the resolutions field in package.json – didn’t help.
• EPIPE with errno: -32 might be a clue but I don’t know what that is – I’ve seen EPIPE connected to memory leaks, ungracefully closed SSL connections and many other things…

This is still happening to me on next@11.1.2 node v16.2.0 and yarn 3.0.0.

Is there anyone who can test my modified PR(https://github.com/vercel/next.js/pull/28420) ? I checked it in my local project, but I want to see if it works in other projects as well.

I don’t think this is limited to mac btw. I’m seeing this on Windows as well.