terser-webpack-plugin: Process aborts with 'out of memory' when using 2.0.0

I created a project that reliably reproduces out-of-memory errors when using terser-webpack-plugin 2.3.2: https://github.com/cjlarose/terser-webpack-plugin-out-of-memory

If you’re using CircleCI or another execution environment where os.cpus().length returns many more CPUs than are allocated to your container, I recommend strongly setting the parallel option of TerserWebpackPlugin explicitly (setting it to the number of available cores corresponding to your resource class is a good rule-of-thumb).

// webpack.config.js
const path = require('path');
const TerserPlugin = require('terser-webpack-plugin');

module.exports = {
  entry: () => {
    ...
  },
  output: {
    path: path.resolve(__dirname, 'dist'),
    filename: '[name].bundle.js'
  },
  optimization: {
    minimizer: [
      new TerserPlugin({ parallel: 8 }),
    ],
  },
};

This will help avoid ENOMEM errors that happen because terser-webpack-plugin tries to fork too many processes. It will help avoid some instances of JavaScript heap out of memory errors, but not all of them. For projects that have many webpack entries (and especially if they’re big), setting parallel explicitly is not sufficient in avoiding JavaScript heap out of memory errors.

I proposed a fix in #206.

Same story here. After we upgraded terser-webpack-plugin from v1.4.1 to v2.2.1 the build process started to run out of memory. As a result, our CI builds started to fail. We have some big chunks though: 5 over 1MB, 2 over 2MB. Degrading back to v1.4.1 resolves the problem. Turning off parallel build resolves the problem as well.

The error that we get (it is reported 3 times in a row):

ERROR in c/92f867d3e514eb399b9d.js from Terser
Error: Call retries were exceeded
    at ChildProcessWorker.initialize (/home/circleci/workdir/project/node_modules/jest-worker/build/workers/ChildProcessWorker.js:193:21)
    at ChildProcessWorker.onExit (/home/circleci/workdir/project/node_modules/jest-worker/build/workers/ChildProcessWorker.js:263:12)
    at ChildProcess.emit (events.js:210:5)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12)

Below are the results summarized in a table:

Note: in all 3 cases above everything else remained unchanged, only 2 variables changed: the version and the parallel option of the plugin.

Release will be today, evening, want to add more tests (increase coverage)

I think I was able to solve this problem, big thanks @cjlarose for inspiration and great idea, honestly I don’t know how I didn’t get to this decision before. Now I run tests to put them here, I think today we release a patch version and close that problem.

I want to warn that this reduces consumption, but for real big projects, for example 5000+ entries, you need increase node memory usage

Looks terser leaking

Terser 4.4.3 released. I think this can be closed now. I have identified a new way to save memory and will open an issue here.

We came to the conclusion that this is being caused by the worker-farm dependency, which detects the CPU count using require('os').cpus().length.

On our medium Circle CI instances this returns 36, but there are 2 CPUs on that resource class:

Adding this to our onCreateWebpackConfig function made the problem go away.

config.optimization.minimizer[0].options.parallel = false

BTW, worker-farm seems a little abandoned: only two commits ever, and no maintenance at all in more than three years. For the maintainers here it might be worth considering a migration to node-worker-farm, which while it does seem to have the same CPU count detection issue, will probably be more worthwhile for one of us to submit a bug report to about it.

I guess it could also be the case that require('os').cpus().length is perfectly correct for every use case except this plugin. Maybe it could be as simple as overriding that maxConcurrentWorkers default if process.env.CIRCLECI === true?

@evilebottnawi found the leak! It is a tiny leak unless you pass a gigantic chunk to Terser. I’ll plug it when I can and ship.

@filipesilva sorry for not responding to your comment, I’ve cloned your repo and all but didn’t reply at the time 😃 thanks for looking into this.

Cool then, I think its indeed true that terser v2 consumes more memory. Now I wonder if we can help something for this. I think it’s also very related to https://github.com/terser/terser since this is just a plugin that wraps terser to webpack. I dont know much about terser internal codebase so I guess we can only be patient 😉

I think it definitely is a terser issue, because normally our build without terser is ~2mins with terser is ~9mins & in this case – out of memory issue – if I disable terser’s sourcemap option (or even disable minification completely) everything works fine. But if I enable sourcemap in terser I get the out of memory issue.

Also it always fail on this step

 98% [0] chunk asset optimization TerserPlugin