webpack: High memory usage from DefaultStatsFactoryPlugin
Bug report
What is the current behavior?
DefaultStatsFactoryPlugin seems to use a lot of memory and causes out-of-memory issues in my build. I’m limiting the process to 8GB and this plugin seems to be using >6GB when the process dies.
When I comment out the plugin, along with the other DefaultStats* plugins added in WebpackOptionsApply.js, the process uses 3GB max.
If the current behavior is a bug, please provide the steps to reproduce.
My setup has approximately 300 pages, all exposed via module federation and using other modules (both federated and not).
Unfortunately I don’t have a way to provide the original code because it is in a private repository and creating a simple reproduction repository is not easy as the problem arises when using hundreds of files but I’ll start creating a repo to showcase the issue.
I just want to start the conversation about what could be done in cases where DefaultStatsFactoryPlugin uses too much memory. I’m not asking in StackOverflow or other channels because I believe this is something in webpack’s core implementation and might require decisions by the core team.
I think it might be related to how I’m using module federation and what info is tracked from the sources. I do have some Heap Snapshots and Allocation timeline profiles that I can send but I believe this may not be a memory leak as all the information in the stats object might be needed at the same time for some of the other steps Webpack performs.
What is the expected behavior?
One option may be to allow more control over how DefaultStatsFactoryPlugin gathers data to minimize the memory footprint by either having a configuration that disables it (knowing that it may affect other plugins) or by filtering the modules that it is tracking.
Other relevant information: webpack version: 5.0.0 Node.js version: 10.22 Operating System: Mac OS, Linux Additional tools:
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 3
- Comments: 15 (7 by maintainers)
Commits related to this issue
- Initial commit. Reproducing webpack/webpack#11659 — committed to juanger/webpack_memory_issues by juanger 4 years ago
We found similar behavior in our project (absurd memory usage, dev server crashing after just a couple of recompiles, extremely slow compilation), and found them to be apparently caused by eslint-webpack-plugin. That is, we remove it from the configuration, and our memory usage and performance returns to normal. If you happen to be using it, worth a look. Admittedly a long shot since it sounds like you’re on TS, but it was certainly the cause of our issues.
@alexander-akait I just wanted to say what I did to overcome this.
I split my federated modules so they don’t have as many files as before and each federated module is bundled in a separate process. I do see an improvement with the latest optimizations though so thank you very much!