core: NextFederationPlugin is crashing on bigger project

Hi @ScriptedAlchemy!

Recently we faced an issue with NextFederationPlugin where NextJS/Webpack would crash if we tried to expose all the project pages (~30 pages).

Some observations so far:

Build does not crash only in build and client-side mode (conditional webpack configuration). I assume it is because the client only is only 50% of memory consumption during the build and fits in standard (4Gb?) nodejs allocation
Standard memory settings can hold 20 out of 30 pages, although extremely slow
If I bump memory past 6Gb, it will compile, but dev mode is extremely slow - producing a change is ~15 seconds vs. 1 second without the plugin
I tried to split up packages into multiple NextFederationPlugin instances, but the outcome is the same (also, it does not support multiple instances?)
Even with a smaller set of exposed components, eventually, dev mode will run out of memory.
We use Nextv13, although I installed canary to generate a trace

While clientside and build time only works for the short term, SSR would help determine 404 (if a page is not exposed) and for SEO purposes.

Please find the trace tree attached.

    Operating System:
      Platform: darwin
      Arch: arm64
      Version: Darwin Kernel Version 22.2.0: Fri Nov 11 02:03:51 PST 2022; root:xnu-8792.61.2~4/RELEASE_ARM64_T6000
    Binaries:
      Node: 18.12.1
      npm: 8.19.2
      Yarn: 1.22.19
      pnpm: N/A
    Relevant packages:
      next: 13.2.5-canary.28
      eslint-config-next: 13.2.4
      react: 18.2.0
      react-dom: 18.2.0

tree_apr3.txt

Please advise what should we look into?

About this issue

Original URL
State: closed
Created a year ago
Reactions: 6
Comments: 42 (21 by maintainers)

Most upvoted comments

seeing increased memory usage and occasional heap limit reached on build, we only have a handful of components so I don’t think the size is an issue (maybe dependencies, though?)

wassgha on Aug 16, 2023

@ScriptedAlchemy facing the same issue with project of pages ~20 as well. It throws the below heap memory error during my build. Interestingly dev server works fine but its notably slow. Currently using nextjs-mf@7.0.6

(node:1761756) [DEP_WEBPACK_MODULE_ID] DeprecationWarning: Module.id: Use new ChunkGraph API
(Use `node --trace-deprecation ...` to show where the warning was created)
info  - Creating an optimized production build .
<--- Last few GCs --->

[1761756:0x5f02ae0]    38185 ms: Mark-Compact 4066.5 (4142.5) -> 4062.2 (4142.5) MB, 786.2 / 0.0 ms  (average mu = 0.639, current mu = 0.166) allocation failure; scavenge might not succeed
[1761756:0x5f02ae0]    39653 ms: Mark-Compact 4078.2 (4142.5) -> 4074.0 (4167.0) MB, 1244.2 / 0.0 ms  (average mu = 0.409, current mu = 0.153) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xbf9890 node::Abort() [/home/user/.nvm/versions/node/v19.9.0/bin/node]
 2: 0xaf6859  [/home/user/.nvm/versions/node/v19.9.0/bin/node]
 3: 0xddaf80 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/home/user/.nvm/versions/node/v19.9.0/bin/node]
 4: 0xddb336 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/home/user/.nvm/versions/node/v19.9.0/bin/node]
 5: 0xfd9db5  [/home/user/.nvm/versions/node/v19.9.0/bin/node]
 6: 0xfed4e5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/user/.nvm/versions/node/v19.9.0/bin/node]
 7: 0xfc924f v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/user/.nvm/versions/node/v19.9.0/bin/node]
 8: 0xfca297 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/user/.nvm/versions/node/v19.9.0/bin/node]
 9: 0xfa99fa v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/home/user/.nvm/versions/node/v19.9.0/bin/node]
10: 0x13bd76f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/home/user/.nvm/versions/node/v19.9.0/bin/node]
11: 0x18443f9  [/home/user/.nvm/versions/node/v19.9.0/bin/node]

Also after I upgrading to v7.0.6 started seeing this message at the top on dev and build

(node:1761756) [DEP_WEBPACK_MODULE_ID] DeprecationWarning: Module.id: Use new ChunkGraph API
(Use `node --trace-deprecation ...` to show where the warning was created)

Any clue?

UPDATE: it has now started crashing in dev server as well after I added couple more pages. Should this be a critical issue?

gitneeraj on Aug 8, 2023

Hi @ScriptedAlchemy, thanks again for all of your work.

I’m testing the release v6.5.2-rc6.0 and I’m facing a memory issue: FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory when the app is built.

Doesn’t happen if I remove the plugin
Doesn’t happen with v6.5.1
No issue in dev mode
Uses pages dir

Plugin is currently tried locally on our main big app, with ~150 pages (I know it’s crazy). And it seems to be because of the app’s size because there’s no problem with only a few pages built.

I’m happy to bring more details if you want me to try some things in order to have a stable v7.

(edit: same thing with stable release (v7.0.0))

yspychala on Jul 3, 2023

Okay im nearing the stage where i think i can release v7 - however this is a major overhaul so please test it, theres likely to be bugs ill have to hotfix

ScriptedAlchemy on Apr 19, 2023

Increase memory.

Might be resolved when I replace one of the plugins with the data manifest plugin we just shipped but no guarantees.

ScriptedAlchemy on Mar 18, 2024

Not to say that something isnt wrong with my plugin… buuut 150 pages?! Split that with federation!! So theres no build big enough to even worry about memory hahaha. thats impressive size

ScriptedAlchemy on Jul 10, 2023

Really appreciate the help so far @ScriptedAlchemy! Looking forward to testing it.

matthewbae on Apr 20, 2023

I have a similar issue. I just posted a related issue about memory issues for a single exposed component. It’s slightly different because my memory consumption is for a single exposed page. But, in my testing, I saw a linear increase in memory as I exposed more pages, and I can only assume that having more exposed pages exacerbates the problem.

I originally exposed multiple pages, but even just trying to build would crash because of out-of-memory errors. Turns out icon libraries would blow the memory past 16gbs for multiple exposed pages. Even without the icons library, I encountered the same memory issues you saw.

Definitely interested in seeing both types of problems resolved.

matthewbae on Apr 9, 2023

facing the same issue, In my nextjs project I have around 10 pages, but have 300+ dependencies (actually using components as module hosted on bit.cloud), before I was getting out of memory error, but by increasing heap size to 16192, I’m able to build project locally but it hangs when running on bitbucket pipeline.

piyushsharma21 on Aug 15, 2023

@ScriptedAlchemy .next folder size is increasing up to 7 to 9gb when i apply module federation, any fix for this, please help me on this, if i comment webpack config in next.config.js its taking upto 700 to 900 MB

I have the same problem. Only able to run the build with --max-old-space-size set to 16192 otherwise it crashes.

The .next folder size is increasing up to 2.5GB when I enable module federation. Without the .next directory is 240 MB

bobsingor on Aug 8, 2023

@ScriptedAlchemy .next folder size is increasing up to 7 to 9gb when i apply module federation, any fix for this, please help me on this, if i comment webpack config in next.config.js its taking upto 700 to 900 MB

purna-hm on Jul 18, 2023

@ialpert I’m very happy that it’s resolved your longstanding issue. Thank you for your patience

ScriptedAlchemy on Jul 8, 2023

Quick update. I’ve been able to build the app by increasing the Node’s allocated memory with --max-old-space-size.

It was set to 8192 and now is 16192 (which is huge!). Since the app builds successfully without nextjs-mf and since it’s ok with v6, I’d rather not have to do this.

I also tried to remove half of the pages of the app, and, without the memory fix, the build fails anyway.

I can provide some logs if I’m told which ones to share.

yspychala on Jul 7, 2023

Actually on second thought I tried to build production and faced the following:

Failed to compile.

./node_modules/@module-federation/nextjs-mf/src/internal-delegate-hoist.js:8:0
Module not found: Package path ./?pop is not exported from package /.../node_modules/react (see exports field in /.../node_modules/react/package.json)

https://nextjs.org/docs/messages/module-not-found

./node_modules/@module-federation/nextjs-mf/src/internal-delegate-hoist.js:15:0
Module not found: Package path ./jsx-runtime?pop is not exported from package /.../node_modules/react (see exports field in /.../node_modules/react/package.json)

https://nextjs.org/docs/messages/module-not-found

./node_modules/@module-federation/nextjs-mf/src/internal-delegate-hoist.js:22:0
Module not found: Package path ./jsx-dev-runtime?pop is not exported from package /.../node_modules/react (see exports field in /.../node_modules/react/package.json)

https://nextjs.org/docs/messages/module-not-found

6.4.0 builds just fine

ialpert on May 3, 2023

Okay! Thanks for the patience.

I’ve pushed out the first beta release of v7

This contains the contents of the large PR

Note it’s on the “next” tag

https://www.npmjs.com/package/@module-federation/nextjs-mf/v/6.4.1-beta.0

ScriptedAlchemy on Apr 28, 2023

So far, its muuuch better.

The stats plugin still needs work but I’m going to let that roll over to after merge release.

Overall my build times and responsiveness is way better now (like 2x)

Website loads are also much faster now thanks to module hoisting

ScriptedAlchemy on Apr 22, 2023

yes. this next plugin has historically been complicated. its been a huge workaround, i think that rewrite which removes child compiler will simplify things greatly and let us focus on improvements instead of base support.

ScriptedAlchemy on Apr 10, 2023

I do thin in the next major release, im able to remove child compilers which i suspect adds enormous overheads.

heres what i need to generate from stats:

{
  "sharedModules": [],
  "federatedModules": [
    {
      "remote": "home_app",
      "entry": "static/chunks/remoteEntry.js",
      "sharedModules": [
        {
          "chunks": [
            "static/chunks/node_modules_antd_es_index_js-_68e70-26097fa26c2daec1-fed.js"
          ],
          "provides": [
            {
              "shareScope": "default",
              "shareKey": "antd",
              "requiredVersion": "^4.22.3",
              "strictVersion": true,
              "singleton": false,
              "eager": false
            }
          ]
        }
      ],
      "exposes": {
        "./SharedNav": [
          {
            "components_SharedNav_tsx": [
              "static/chunks/components_SharedNav_tsx-2d770589888512b3-fed.js"
            ],
            "pages_home_test-shared-nav_tsx": [
              "static/chunks/pages_home_test-shared-nav_tsx-ed5cb44cc0da577c-fed.js"
            ]
          }
        ],
        "./menu": [
          {
            "components_menu_tsx": [
              "static/chunks/components_menu_tsx-a2794b447f6e1fcc-fed.js"
            ]
          }
        ]
      },
      "remoteModules": {
        "checkout/pages/checkout/[...slug]": "webpack/container/remote/checkout/pages/checkout/[...slug]",
        "checkout/pages/checkout/[pid]": "webpack/container/remote/checkout/pages/checkout/[pid]",
        "checkout/pages/checkout/exposed-pages": "webpack/container/remote/checkout/pages/checkout/exposed-pages",
        "checkout/pages/checkout/index": "webpack/container/remote/checkout/pages/checkout/index",
        "checkout/pages/checkout/test-check-button": "webpack/container/remote/checkout/pages/checkout/test-check-button",
        "checkout/pages/checkout/test-title": "webpack/container/remote/checkout/pages/checkout/test-title",
        "shop/useCustomRemoteHook": "webpack/container/remote/shop/useCustomRemoteHook",
        "shop/pages/shop/exposed-pages": "webpack/container/remote/shop/pages/shop/exposed-pages",
        "shop/pages/shop/index": "webpack/container/remote/shop/pages/shop/index",
        "shop/pages/shop/test-webpack-png": "webpack/container/remote/shop/pages/shop/test-webpack-png",
        "shop/pages/shop/test-webpack-svg": "webpack/container/remote/shop/pages/shop/test-webpack-svg",
        "shop/pages/shop/products/[...slug]": "webpack/container/remote/shop/pages/shop/products/[...slug]",
        "checkout/CheckoutTitle": "webpack/container/remote/checkout/CheckoutTitle",
        "checkout/ButtonOldAnt": "webpack/container/remote/checkout/ButtonOldAnt",
        "shop/WebpackSvg": "webpack/container/remote/shop/WebpackSvg",
        "shop/WebpackPng": "webpack/container/remote/shop/WebpackPng"
      }
    }
  ]
}

ScriptedAlchemy on Apr 9, 2023

Okay for stats, i need module reasons to track any shared or remote modules. This is not critical but does improve perf when loading code since i can ssr js tags.

I likely dont need that much stats though. So we can most likely trim it down. Take a look at the federation stats json that i emit, thats essentially the info i need to collect, nothing else.

ScriptedAlchemy on Apr 9, 2023