next.js: Slower Response Times when updating from 10.0.7 to 10.2.0 or 11.0.1 or 12.0.1 (with Disabled automatic image optimization)

Ok, so here we go.

Custom Next.js configuration:

• inject polyfills

• @sentry import alias

• TerserPlugin config

• next-plugin-custom-babel-config and next-transpile-modules (because of monorepo)

• bundle analyzer

• custom static import plugin

• some next.js configs

• next.config.js

    const Webpack = require('webpack')
    const namespace = process.env.NAMESPACE
    const buildIdArgument = process.env.BUILD_ID
    const withStaticImport = require('./buildtools/plugin-static')
    const withBundleAnalyzer = require('@next/bundle-analyzer')({
        enabled: process.env.ANALYZE === 'true',
    })
    const path = require('path')
    const withTranspileModules = require('next-transpile-modules')
    const withPlugins = require('next-compose-plugins')
    const withCustomBabelConfigFile = require('next-plugin-custom-babel-config')
  
    function withCustomWebpack(config2 = {}) {
        const { webpack } = config2
  
        config.webpack = (config, options) => {
            const { isServer, dev, buildId } = options
  
            const originalEntry = config.entry
            config.entry = async () => {
                const entries = await originalEntry()
  
                if (entries['main.js'] && !entries['main.js'].includes('./src/polyfills/polyfills.ts')) {
                    entries['main.js'].unshift('./src/polyfills/polyfills.ts')
                }
  
                return entries
            }
  
            if (!dev) {
                config.plugins.push(new Webpack.DefinePlugin({ 'process.env.SENTRY_RELEASE': JSON.stringify(buildId) }))
            }
  
            // In `pages/_app.js`, Sentry is imported from @sentry/node. While
            // @sentry/browser will run in a Node.js environment, @sentry/node will use
            // Node.js-only APIs to catch even more unhandled exceptions.
            //
            // This works well when Next.js is SSRing your page on a server with
            // Node.js, but it is not what we want when your client-side bundle is being
            // executed by a browser.
            //
            // Luckily, Next.js will call this webpack function twice, once for the
            // server and once for the client. Read more:
            // https://nextjs.org/docs#customizing-webpack-config
            //
            // So ask Webpack to replace @sentry/node imports with @sentry/browser when
            // building the browser's bundle
            if (!isServer) {
                config.resolve.alias['@sentry/node'] = '@sentry/browser'
            }
  
            if (!dev) {
                if (config.optimization && config.optimization.minimizer) {
                    for (const plugin of config.optimization.minimizer) {
                        if (plugin.constructor.name === 'TerserPlugin') {
                            plugin.options.sourceMap = true
                            plugin.options.terserOptions = {
                                ...plugin.options.terserOptions,
                                warnings: false,
                                compress: {
                                    ...plugin.options.terserOptions.compress,
                                    drop_console: !isServer,
                                },
                            }
                            break
                        }
                    }
                }
            }
            return webpack(config, options)
        }
  
        return config
    }
  
    const plugins = [
        [
            withCustomBabelConfigFile({
                babelConfigFile: path.resolve('../../babel.config.js'),
            }),
        ],
        [withTranspileModules(['@wh'])],
        [withStaticImport],
        [withBundleAnalyzer],
        [withCustomWebpack],
    ]
  
    const config = {
        generateBuildId: async () => {
            return buildIdArgument || 'local'
        },
        publicRuntimeConfig: {
            namespace: namespace,
        },
        productionBrowserSourceMaps: true,
        poweredByHeader: false,
        assetPrefix: '', // we do not set this to our cache server on purpose, as it doesn't play nicely with plugin-static - we rather use proxy rules to redirect /_next/ to the cache server
        typescript: {
            ignoreDevErrors: true,
            ignoreBuildErrors: false,
        },
    }
  
    module.exports = withPlugins(plugins, config)
  

• polyfills.ts

    if (window.Promise && !window.Promise.finally) {
        window.Promise.finally = require('core-js/features/promise/finally')
    }
  
    require('intersection-observer')
  
    // next.js suggests to export undefined https://github.com/zeit/next.js/issues/7959 but this causes a babel warning. so we now export null and it seems to work
    export default null
  

• babel.config.js

    module.exports = function (api) {
        api.cache(true)
  
        const presets = ['next/babel']
  
        const plugins = [
            [
                'module-resolver',
                {
                    root: ['.'],
                    alias: {
                        '@my/alias': './some/path',
                    },
                },
            ],
            [
                'styled-components',
                {
                    ssr: true,
                },
            ],
            '@babel/plugin-proposal-optional-chaining',
            '@babel/plugin-proposal-nullish-coalescing-operator',
        ]
  
        return {
            presets,
            plugins,
        }
    }
  

• plugin-static.js

     * Thanks to Georges Haidar
     *
     * Registers 1) a module loader rule and 2) a handy alias for file-loader both
     * of which allow importing static assets in client-side code.
     *
     * 1) Module loader rule
     * This will allow importing images assets like any other module like so:
     *
     * ```
     * import logo from "../static/logo.svg";
     * ```
     *
     * The value of logo will be a URL to the static asset
     *
     * 2) `static!` alias for file-loader
     *
     * The `static!` prefix can be used in import statements to treat a module as a
     * static asset and return a URL to it instead of bundling it and loading it
     * like a regular js/json module.
     *
     * ```
     * import honeybadger from "honeybadger-js";
     * ```
     * The effect of this line would be that webpack bundles the node module and
     * the value of honeybadger will be the default export from it.
     *
     * What if instead we wanted a reference to this module as a url that can be
     * given to a `<script />` tag? In that case we can write the following:
     *
     * ```
     * import honeybadgerSrc from "static!honeybadger-js";
     * ```
     *
     * honeybadgerSrc is now a URL that can be passed to a script's src attribute
     *
     * 3) the query params ?inline can be used to pass files to the url-loader instead to inline them
     *
     * @param {*} nextConfig
     */
    module.exports = (nextConfig = {}) => {
        return Object.assign({}, nextConfig, {
            webpack(config, options) {
                const { set } = require('lodash')
  
                const { isServer } = options
                const staticConfig = {
                    context: '',
                    emitFile: true,
                    name: '[name].[hash].[ext]',
                    publicPath: '/_next/static/assets/', // do not use assetPrefix from next on purpose, as it will not work as expected
                    outputPath: `${isServer ? '../' : ''}static/assets/`,
                }
  
                set(config, 'resolveLoader.alias.static', `file-loader?${JSON.stringify(staticConfig)}`)
  
                config.module.rules.push({
                    test: /\.(jpe?g|png|svg|gif|ico|webp|woff|woff2)$/,
                    oneOf: [
                        {
                            resourceQuery: /inline/,
                            use: [
                                {
                                    loader: 'url-loader',
                                },
                            ],
                        },
                        {
                            use: [
                                {
                                    loader: 'file-loader',
                                    options: staticConfig,
                                },
                            ],
                        },
                    ],
                })
  
                if (typeof nextConfig.webpack === 'function') {
                    return nextConfig.webpack(config, options)
                }
  
                return config
            },
        })
    }
  

Custom express server:

• unleash integration using unleash-client
• /health endpoint for kubernetes livenessProbe and readinessProbe
• dns caching with dnscache
• prometheus endpoint using prom-client
• compression using compression
• parsing cookies using cookie-parser
• custom 301 redirects (advanced features like removing multiple slashes, with/without trailing slash based on route, case-sensitivity of path, stripping parts of paths)
• rewriting paths to lowercase with some exceptions
• access logging
• removing double and trailing slashes for next.js
• set custom cookies with set-cookie, partially based on unlease feature toggle states
• rewrites for app-embedded webviews (renders a different SSG/SSR next.js route)
• rewrite for some page to convert path params to query params and some alias pages
• logout route which clears cookies
• feature toggle endpoint which provides unleash feature toggles to clients
• some email verification stuff
• static assets with express.static
• SIGINT and SIGTERM handling for kubernetes graceful shutdown

_document.tsx:

• styled-components style collection in `getInitialProps`

    static async getInitialProps(ctx: ExpressNextDocumentContext): Promise<DocumentInitialProps> {
        // tslint:disable-next-line:no-unsafe-any
        const sheet = new ServerStyleSheet()
        const view = ctx.renderPage
  
        try {
            ctx.renderPage = () =>
                view({
                    // tslint:disable-next-line:no-unsafe-any
                    enhanceApp: (App) => (props) => sheet.collectStyles(<App {...props} />),
                })
  
            const initialProps = await Document.getInitialProps(ctx)
            return {
                ...initialProps,
                styles: (
                    <React.Fragment>
                        {initialProps.styles}
                        {sheet.getStyleElement()}
                    </React.Fragment>
                ),
            }
        } finally {
            // tslint:disable-next-line:no-unsafe-any
            sheet.seal()
        }
    }
  

• custom next/document head without preload to improve LCP

  HeadWithoutPreload.tsx

    import { Head } from 'next/document'
  
    // remove <link rel="preload"> for next.js chunks - improves FCP/LCP a lot, but worsens TTI a little
    // https://github.com/vercel/next.js/discussions/11120#discussioncomment-109894
    export class HeadWithoutPreload extends Head {
        getPreloadDynamicChunks() {
            return []
        }
  
        getPreloadMainLinks() {
            return []
        }
    }
  

• preconnects, dns-prefetches, favicons

• inlined <style> for self-hosted font

• core-js polyfilling for old browsers with <script noModule /> referencing core-js-bundle/minified using the plugin-static mentioned above

_app.tsx:

• getInitialProps
  • set some global state based on user agent
  • fetch some common api data for the that is shown in the application header on all pages
  • check if current page is SSR or SSG and throw on api errors for SSG (since it’s build time)
  • for SSR fetch user data if logged in
  • for SSR render getInitialProps for current component and measure getInitialProps time and add it as custom header to response
• componentDidMount
  • apply custom scroll restoration
  • set some state for providers for SSG pages
  • some other small stuff
• componentDidCatch: capture exception with Sentry
• render
  • a lot of providers, e.g.
    • react-redux Provider (legacy, only used for one page anymore)
    • styled-components theme provider
    • feature toggles
    • etc.
  • some global styled-components styles
  • NProgress
  • global container for react-toastify
  • loader for some low-priority external scripts using useEffect
  • custom page layout component based on rendered component

Note that we use styled-system (actually a custom fork of it), which means there is some runtime overhead for generating all the css classes on top of the styled-components overhead.

If you need to know anything in more detail, let me know. Thanks for helping!

It would be super helpful if you could help us further narrow down the canaries where these issues started appearing and getting worse for you. I wouldn’t recommend running the canaries in production, but perhaps you could do a local build with a canary or minor release, get a good sample of response times, and bisect to the next version?

It sounds like the first regression was between 10.0.7 and 10.2.0, and there was another between 10.2.0 and 11.0.1, so it would be very useful if you could help narrow those down to separate canaries. Thank you!

Maybe instead of a reduced test case, what about listing out some things that not all applications would use

Ok, I‘ll do a write-up in the next few days.

So much for now: we‘re mixing SSR (with getInitialProps) and SSG, but since SSG ist extremely fast anyways, we don‘t really see changes there. We don‘t use ISR. We use styled-components which could have an impact on SSR render time (collecting styles in _document).

Based on the paths above, I guess this website is https://www.willhaben.at/iad ? Not sure whether this information is useful for coming up with a theory of what is making it slower, but at least it shows the general complexity of the pages…

Well this exact url is served from a legacy system, but yes, this is the domain of our application. /iad/immobilien and /iad/gebrauchtwagen is served from our next.js application.

Unfortunately it’s a closed-source project. If you need more detailed information about something, please let me know.

A reproduction would be helpful, we can’t investigate based on this report.