next.js: NextJs compiling extremely slow

I’ve experienced the issue with @mui/material and my own npm package @gorazdo/tomui The problem comes from a lot of modules to process (11k) like in this article Unfortunately, this solution doesn’t work. The problem is the regex (which is a completely mysterious thing for a non-rust developer. I’ve even tried Rust Regex online tool ), with no luck.

My Solution (3-5 times faster)

decreased 3-5 times dev build/rebuild/fast-refresh time decreased amount of modules by 10 times from 11k to 1k modules.

I’ve managed it using the following config:

    modularizeImports: {
      '@gorazdo/tomui': {
        transform: '@gorazdo/tomui/{{member}}',
      },
      '@mui/material': {
        transform: '@mui/material/{{member}}',
      },
      '@mui/icons-material/?(((\\w*)?/?)*)': {
        transform: '@mui/icons-material/{{ matches.[1] }}/{{member}}',
      },
    },

But I had to update imports manually for useTheme, ThemeProvider, styled, lighten and darken in my case

from:

import { Box, lighten, Typography, styled } from '@mui/material'
// for lighten and styled the regex didn't work neither the simple {{member}} way. so it is manually changed

to

import { Typography, Box } from '@mui/material'
import { styled, lighten } from '@mui/material/styles' 

To fix my library I had to use default export for each file which is going to be modularized.

Questions

1. How to deal with this regex to make it work with @mui/material ?
2. How to deal with modularization of files that have only named exports?
3. Can we automate modularization somehow?

Experiencing this issue?

To provide a trace which only includes metadata of the application:

• npm install next@canary (or yarn add next@canary)
• Run development like normal and reproduce the issue you’re experiencing
• Stop Next.js (generally ctrl + c)
• Send the file .next/trace file here or send it to https://twitter.com/timneutkens if you don’t want to share it publicly. You can use https://gist.github.com to create a private url for the file. - If you have a custom .babelrc please provide it - If you have a custom next.config.js please provide it - if you are using TailwindCSS please provide tailwind.config.js

⚠️ The metadata in .next/trace includes full file paths to all JS/CSS. It does not include the file contents. ⚠️ The main thing we need to help investigate your application is .next/trace, package.json and such are not helpful in investigating the issue.

Reply to earlier messages

but shouldn’t the potential solutions be posted here to help everyone that has this issue?

So far all I’ve done is help investigate the slowdowns for the people that reached out. There’s no specific solution as it highly depends on the app and I’m going to share the findings. . I’ve been working on a way that we can investigate these slowdowns without needing the application code itself but it requires a bunch of steps that up till a few days ago were quite involved.

Here’s a list of what I’ve found so far:

• @stevensturkop hasn’t provided a trace yet
• @PhillipLangMartinez
  • Fast Refresh is behaving as expected, no particular slowdown, will benefit from the work we’re doing for SWC and webpack optimizations though
  • The main problem they’re experiencing is about on-demand-entries. Next.js compiles the JS/CSS/etc for pages on-demand when you request them, so if you request http://localhost:3000/about Next.js will at that point start compiling the JS/CSS for pages/about.js, this is intentional to ensure that you can have thousands of pages in a Next.js app and it would not slow down your app, otherwise you’d pay this compilation penalty at bootup
    • This will also benefit from the work we’re doing integrating SWC with full backwards compat. I’ve had Phillip try the new flag and it’s resulted in at least 50% faster initial compilation and ~60% faster compiling of pages. Note that this highly depends on if you have a lot of node_modules or a lot of application code, the application code will be compiled faster but node_modules are not passed through babel currently so they are not affected by the work we’re doing to add SWC at this point in time.
  • In order to investigate this one further I’m adding timings for filesystem reads to find if that’s what is causing issues with a large amount of JS as they have a bunch of large node_modules like react-bootstrap included
• @kartikrao
  • 90% of the time spent compiling is related to TailwindCSS, it’s slowing down each CSS file by spending about 2 minutes and 30 seconds, it seems that the application is not using TailwindCSS JIT, so I’d recommend enabling that for all users that are using TailwindCSS: https://tailwindcss.com/docs/just-in-time-mode
• @f16z
  • Machine fs.readFile seems to be slow, it takes over 100ms per file, we expanded the tracing to include timing for individual files
  • Initial compile + compiling pages/index.js reads and compiles about 6867 files. What I’ve found is that 5557 of those are coming from @material-ui/icons/esm, assuming that the app is not using 5.5K icons I’d recommend changing how they are imported so that not all files have to be compiled. Based on debugging this trace we’re adding a separate chart that will allow us to visualize the dependency graph based on a trace file
• @billymcintosh can you provide a trace based on the above instructions

I had a similar issue and it was also due to @material-ui/icons and loading all of them. If you import them directly from your application code, the tree-shaking works properly. However, in our case, we were importing them from an internal package installed in the node_modules of our application which resulted in all icons being imported (only in the server bundle).

We solved it by following this documentation and adding a babel plugin to automatically rename our named imports to default imports when building and publishing our internal package.

I think we should consider this issue closed. @anthonyalayo

I’ve added the modularizeImports MUI transforms and tried 10 other things mentioned in other places, but still the DX with NextJS is very bad for many people due to extremely slow page loads. The DX in other tools in the ecosystem (e.g. vite, even create-react-app) don’t suffer from this, so I don’t think this issue should be closed.

The development experience with NextJS is advertised greatly by Vercel, so it is frustrating when it does not live up to its promises.

This is how the compiling look like during a page navigation in my fairly large project:

Here’s some screenshots when I tried adding dependencies one by one in a new project with next 13.1.6:

After adding date-fns:

After adding modularizeImport on date-fns:

So the modularizeImports definitely works, but only a small number of libraries organizes their imports in a way that is compatible.

After adding @mui/x-data-grid (@mui/material, @mui/icons-material, date-fns, @mui/x-data-grid in total)

With ~1400 modules compiled and taking 1.7 seconds for page load in an empty project just by adding a few dependencies, it’s easy to see how it gets out of hand in a large project.

I wonder if a lot of the slow page loading is due to prefetching of pages? In the following screenshot I hovered my mouse over a few links in my sidebar before navigating which seems to create a waterfall effect and cause delay.

If NextJS cannot improve the page load performance due to Webpack, I still think it can help with tooling to improve the DX. For example:

• Option to disable prefetching globally in development
• Debug option to log which modules was compiled
• Automatically detect libraries compatible with modularizeImport and do it automatically …etc

Happy to share private repo with NextJS developers if it helps solving this issue.

Our app www.drive.com.au is the largest automotive site in Australia.

Creating a production build with next build takes > 20 minutes on a machine with 6 cores and 32 GB of RAM. Our builds have slowed significantly since updating to Next 11.x

@timneutkens - Happy to provide access to config and code, if its of wider benefit to the community, we are also happy to pitch in if helpful.

I admire you helping one to one, but shouldn’t the potential solutions be posted here to help everyone that has this issue?

Good news! I approximately reduced my next dev from 6500 to 2500 modules for the first page I load!

TLDR: explode your imports!

For the long story:

I tried at first babel-plugin-import (https://github.com/inclusion-numerique/mediature/commit/a5dc625fdd072c43438cf70f60f420201e80d62d) to allow using merged imports for MUI… but it didn’t reduce the modules built.

Then I tried the way of Next.js with:

    modularizeImports: {
      '@mui/material': {
        transform: '@mui/material/{{member}}',
      },
      '@mui/icons-material/?(((\\w*)?/?)*)': {
        transform: '@mui/icons-material/{{ matches.[1] }}/{{member}}',
      },
    },

But nothing changed too.

Weeks after I was just upset by this painful slow compilation of module just to load basic pages (sometimes 40 seconds).

I decided to go the hard way, I replaced for example all my:

import { Grid, Typography } from '@mui/material';

by:

import Grid from '@mui/material/Grid';
import Typography from '@mui/material/Typography';

I also replaced for example:

import { LocalizationProvider } from '@mui/x-date-pickers';
import { frFR as coreFrFR } from '@mui/material/locale';
// import { DataGrid, frFR as dataGridFrFR } from '@mui/x-data-grid';
import { frFR as datePickerFrFR } from '@mui/x-date-pickers';

by:

import { LocalizationProvider } from '@mui/x-date-pickers/LocalizationProvider';
import { frFR as coreFrFR } from '@mui/material/locale';
// import { frFR as dataGridFrFR } from '@mui/x-data-grid/locales/frFR';
import { frFR as datePickerFrFR } from '@mui/x-date-pickers/locales/frFR';

import { fr as frDateLocale } from 'date-fns/locale';
import { format as formatDate, formatDistance, formatRelative, isDate } from 'date-fns';

by:

import frDateLocale from 'date-fns/locale/fr';
import formatDate from 'date-fns/format';
import formatDistance from 'date-fns/formatDistance';
import formatRelative from 'date-fns/formatRelative';
import isDate from 'date-fns/isDate';

(for my examples it was spread across files but I tried to give you an overview of the steps (https://github.com/inclusion-numerique/mediature/commit/f1c4850da4427de2e369931d0ede7bc5ba81b4aa), just think about big librairies and try to see if there is way to do the “tree-shaking” manually).

Note that I don’t get why using Babel or Next.js optimization didn’t help on MUI. From what I read Webpack tree-shaking is only enabled when NODE_ENV=production, is it the reason? For me tree-shaking is also important in development to not bring everything to the table of compilation.

If Babel/Next.js optimizations worked for you even with next dev, I’m fine to admit I did something wrong, but no clue what it is. Is it possible it’s due to pnpm? (just asking, I saw a lot of weird issues with it due to symlinks)

Speaking of pnpm, I also find a way to save some modules compilation (due to duplication of packages). I use a monorepo and some of my packages are using MUI. Since they were not align on some rules of pnpm a dependency with the exact same version in 2 packages would not be merged into one (it’s deduplicated). It’s more specific to pnpm so if you want this, I tried to explain a bit how to debug/fix this: https://github.com/pnpm/pnpm/discussions/6055#discussioncomment-5018827

Now my first load of a page is 2500 modules in about 12 seconds, which is way better even if still not perfect.

Hope this helps!

In case it helps somebody, in our case the line causing the extremely slow compilation was this one:

//tailwind.config.js
/** @type {import('tailwindcss').Config} */
module.exports = {
  // eslint-disable-next-line global-require, import/no-extraneous-dependencies
  presets: [require('tailwind-preset')],
  content: [
    './src/app/**/*.{js,ts,jsx,tsx}',
    './src/pages/**/*.{js,ts,jsx,tsx}',
    './src/components/**/*.{js,ts,jsx,tsx}',
    './src/layout/**/*.{js,ts,jsx,tsx}',
    '../../packages/ui/**/*.{js,ts,jsx,tsx}', // THIS ONE!
  ],
};

In our case we are using turborepo and when adding the package/ui the compilation time goes from 2-5 seconds to >500 seconds

Is there a public tool available to parse the .next/trace file? Would love to dive deeper without bugging Tim 😃

Edit: https://github.com/vercel/next.js/blob/canary/scripts/trace-to-tree.mjs is the tool (and it’s awesome!)

In case it helps somebody, in our case the line causing the extremely slow compilation was this one:

//tailwind.config.js
/** @type {import('tailwindcss').Config} */
module.exports = {
  // eslint-disable-next-line global-require, import/no-extraneous-dependencies
  presets: [require('tailwind-preset')],
  content: [
    './src/app/**/*.{js,ts,jsx,tsx}',
    './src/pages/**/*.{js,ts,jsx,tsx}',
    './src/components/**/*.{js,ts,jsx,tsx}',
    './src/layout/**/*.{js,ts,jsx,tsx}',
    '../../packages/ui/**/*.{js,ts,jsx,tsx}', // THIS ONE!
  ],
};

In our case we are using turborepo and when adding the package/ui the compilation time goes from 2-5 seconds to >500 seconds

fixed it for me, thanks!

Hey Steven, could you send me a message on https://twitter.com/timneutkens, then we can investigate it without code access.

I’ve had the same problem for the last 8 months. It seems many people are having the same problem. In my opinion this should be a priority to get fixed. I have tried almost everything: deactivating windows defender, using yarn instead of npm, but nothing has worked.

sometimes it takes 1 min to change pages in development. In production it’s super fast.

I can’t publish my code as well since it’s copyrighted.

{
  "name": "",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "next": "next",
    "dev": "nodemon server/index.js",
    "build": "next build",
    "start": "node server/index.js",
    "analyze": "cross-env ANALYZE=true next build",
    "analyze:server": "cross-env BUNDLE_ANALYZE=server next build",
    "analyze:browser": "cross-env BUNDLE_ANALYZE=browser next build"
  },
  "dependencies": {
    "@date-io/date-fns": "^1.3.13",
    "@dnd-kit/core": "^3.1.1",
    "@dnd-kit/sortable": "^4.0.0",
    "@material-ui/core": "^4.12.3",
    "@material-ui/icons": "^4.9.1",
    "@material-ui/lab": "^4.0.0-alpha.60",
    "@material-ui/pickers": "^3.3.10",
    "@next/bundle-analyzer": "^11.0.1",
    "@sendgrid/mail": "^7.4.0",
    "@stripe/stripe-js": "^1.12.1",
    "@tippyjs/react": "^4.2.5",
    "aws-sdk": "^2.809.0",
    "axios": "^0.21.0",
    "bcryptjs": "^2.4.3",
    "bootstrap": "^4.4.1",
    "chroma-js": "^2.1.1",
    "config": "^3.3.2",
    "cookie": "^0.4.1",
    "cookie-parser": "^1.4.5",
    "cookie-session": "^1.4.0",
    "cors": "^2.8.5",
    "cross-env": "^7.0.3",
    "d3-ease": "^2.0.0",
    "date-fns": "^2.21.1",
    "express": "^4.17.1",
    "express-session": "^1.17.1",
    "express-validator": "^6.6.1",
    "geoip-lite": "^1.4.2",
    "intersection-observer": "^0.12.0",
    "js-cookie": "^2.2.1",
    "jsonwebtoken": "^8.5.1",
    "moment": "^2.29.1",
    "mongoose": "^5.10.11",
    "next": "10.2.3",
    "next-absolute-url": "^1.2.2",
    "node-fetch": "^2.6.1",
    "nprogress": "^0.2.0",
    "passport": "^0.4.1",
    "passport-google-oauth20": "^2.0.0",
    "password-validator": "^5.1.1",
    "pdfjs-dist": "^2.7.570",
    "prismjs": "^1.23.0",
    "quill": "^1.3.7",
    "randomcolor": "^0.6.2",
    "react": "16.14.0",
    "react-beautiful-dnd": "^11.0.5",
    "react-bootstrap": "^1.4.0",
    "react-contenteditable": "^3.3.5",
    "react-dom": "16.14.0",
    "react-dropzone": "^11.2.3",
    "react-quill": "^1.3.5",
    "react-select": "^3.1.1",
    "react-spring": "^8.0.27",
    "react-toastify": "^6.1.0",
    "react-zoom-pan-pinch": "^1.6.1",
    "request-ip": "^2.1.3",
    "roughjs": "^4.3.1",
    "sass": "^1.27.0",
    "socket.io": "^3.1.2",
    "socket.io-client": "^3.1.2",
    "string-strip-html": "^4.3.5",
    "stripe": "^8.137.0",
    "validator": "^13.5.2",
    "web-push": "^3.4.4"
  },
  "devDependencies": {
    "eslint": "^7.16.0",
    "eslint-plugin-react": "^7.21.5"
  }
}

In case it helps somebody, in our case the line causing the extremely slow compilation was this one:

//tailwind.config.js
/** @type {import('tailwindcss').Config} */
module.exports = {
  // eslint-disable-next-line global-require, import/no-extraneous-dependencies
  presets: [require('tailwind-preset')],
  content: [
    './src/app/**/*.{js,ts,jsx,tsx}',
    './src/pages/**/*.{js,ts,jsx,tsx}',
    './src/components/**/*.{js,ts,jsx,tsx}',
    './src/layout/**/*.{js,ts,jsx,tsx}',
    '../../packages/ui/**/*.{js,ts,jsx,tsx}', // THIS ONE!
  ],
};

In our case we are using turborepo and when adding the package/ui the compilation time goes from 2-5 seconds to >500 seconds

fixed it for me, thanks!

Hi. but, how you found workaround to not include paths from one of packages if you really need this?

hello there, to fix you problem you have to make a component (put all you code here like the “App” component in react) and then put it like this : import App from ‘./App’ export default function Home() { // return the App component } update just the App

You can visualize the trace yourself using canary/scripts/trace-to-tree.mjs if you’re curious.

node scripts/trace-to-tree.mjs /path/to/trace

Option to disable prefetching globally in development

Prefetching is disabled in development for pages. For app it’s enabled but only on hover. If you want to completely disable prefetching you can pass prefetch={false}.

Debug option to log which modules was compiled

See above.

Automatically detect libraries compatible with modularizeImport and do it automatically

This is potentially breaking which is why we don’t have a default config.

Overall the issues with large amounts of modules was one the reasons we started building Turbopack: https://turbo.build/pack. Which is currently in alpha. It’s a completely new architecture to allow processing many more modules quickly, including for next build.

“dev”: “next dev --turbo”, using turbopack worked for me https://nextjs.org/docs/architecture/turbopack

https://github.com/vercel/next.js/issues/29559#issuecomment-1323379937

THIS!!! Thank you sir, this solved a huge headache for me. I copied my settings from some example config provided by turbo I believe and this was a nightmare.

@Vallo not without changing it, I also had to do the same for a library I was using.

Enabling swcMinify might speed things up.

// next.config.js
    swcMinify: true,

Is there a way to lazily compile linked pages in the background once the current page is done?

Hey @ctotheameron, I had a look at the trace file and it seems that the slowdown is coming from vanilla-extract. If you have a look at the output of our trace file interpreter: https://gist.github.com/timneutkens/52f5cf1d1991b2789fbaa81885638c00 (you can run it yourself, it’s this script: https://github.com/vercel/next.js/blob/canary/scripts/trace-to-tree.mjs)

It seems the large majority of the time spent is caused by the plugin added: https://gist.github.com/ctotheameron/4737a779386aac592d3f7fde13016d83#file-next-config-js-L2-L3

I’ve experienced the issue with @mui/material and my own npm package @gorazdo/tomui The problem comes from a lot of modules to process (11k) like in this article Unfortunately, this solution doesn’t work. The problem is the regex (which is a completely mysterious thing for a non-rust developer. I’ve even tried Rust Regex online tool ), with no luck.

My Solution (3-5 times faster)

decreased 3-5 times dev build/rebuild/fast-refresh time decreased amount of modules by 10 times from 11k to 1k modules.

I’ve managed it using the following config:

    modularizeImports: {
      '@gorazdo/tomui': {
        transform: '@gorazdo/tomui/{{member}}',
      },
      '@mui/material': {
        transform: '@mui/material/{{member}}',
      },
      '@mui/icons-material/?(((\\w*)?/?)*)': {
        transform: '@mui/icons-material/{{ matches.[1] }}/{{member}}',
      },
    },

But I had to update imports manually for useTheme, ThemeProvider, styled, lighten and darken in my case

from:

import { Box, lighten, Typography, styled } from '@mui/material'
// for lighten and styled the regex didn't work neither the simple {{member}} way. so it is manually changed

to

import { Typography, Box } from '@mui/material'
import { styled, lighten } from '@mui/material/styles' 

To fix my library I had to use default export for each file which is going to be modularized.

Questions

1. How to deal with this regex to make it work with @mui/material ?
2. How to deal with modularization of files that have only named exports?
3. Can we automate modularization somehow?

Hey @Carduelis , managed to find a regex that works with hooks and stuff? Thanks!