next.js: Async generateMetadata hangs the app without any visible sign of loading

Link to the code that reproduces this issue or a replay of the bug

https://github.com/wiwo-dev/generatemetadata-loading-issue https://codesandbox.io/p/github/wiwo-dev/generatemetadata-loading-issue/main

To Reproduce

  1. Run the application using the command: yarn run dev.

  2. Navigate to the root path (/). Here, you’ll find two lists of “users” retrieved from jsonplaceholder. The list at the top has the problem as it uses generateMetadata. The one at the bottom doesn’t use generateMetadata.

  3. The top list contains links to /user/[userId], and these links have a generateMetadata function that fetches data. Upon clicking these links, for the initial 3 seconds, there is no visible loading indication, making users believe that something might be wrong with the link.

  4. In contrast, if you click on any of the links in the bottom list, which leads to /user-no-metadata/[userId] that doesn’t use generateMetadata, you’ll notice that a Loading state is triggered by Suspense, providing a clear loading indication.

Current vs. Expected behavior

Current behaviour When a user navigates to a page containing a generateMetadata function that fetches data from a slowly working API, there’s a problem. The application appears to hang, providing no indication that it’s actively loading. Notably, the fetch operation within generateMetadata doesn’t trigger React’s Suspense, and the browser also fails to display any loading indicators.

To simulate a slow API response, a setTimeout is intentionally used within the getUsers function.

I encountered this issue while working on a project with a tens of thousands number of subpages, and it’s highly likely that a given subpage will be opened for the first time and only once, with new entries continually being added.

Expected behaviour It should trigger the Suspense or show any kind of loading indicator in the browser so the user knows that something is happening under the hood and wait for the page to be displayed.

Verify canary release

  • I verified that the issue exists in the latest Next.js canary release

Provide environment information

Operating System:
      Platform: darwin
      Arch: arm64
      Version: Darwin Kernel Version 22.4.0: Mon Mar  6 21:00:41 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T8103
    Binaries:
      Node: 18.7.0
      npm: 8.15.0
      Yarn: 1.22.15
      pnpm: 6.11.0
    Relevant Packages:
      next: 13.4.20-canary.36
      eslint-config-next: N/A
      react: 18.2.0
      react-dom: 18.2.0
      typescript: 5.1.3
    Next.js Config:
      output: N/A

Which area(s) are affected? (Select all that apply)

Metadata (metadata, generateMetadata, next/head)

Additional context

Is there a way to display a loading state or any form of indication that something is loading while still utilising the generateMetadata function to fetch metadata information from a slow API?

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 52
  • Comments: 47 (15 by maintainers)

Commits related to this issue

Most upvoted comments

👋 Hi everyone! We heard your requests.

We’re introducing Streaming Metadata as the enhancement of the recent canary releases to improve metadata rendering performance.

Introduction

This feature is designed to improve page loading time for users, especially when dealing with slow metadata generation. With Streaming Metadata enabled:

  • The generateMetadata returned metadata is treated as suspenseful data, allowing the page content to render immediately.
  • Metadata is then injected asynchronously into the page on the client side once it’s resolved.

Since metadata is critical for SEO, we’ve ensured that bots continue to receive fully rendered metadata in the HTML. For human users, who primarily care about page content, metadata can be added later without impacting their experience.

Usage

The following new options in next.config.js are for configuring Streaming Metadata:

  • experimental.streamingMetadata (boolean): Enables the Streaming Metadata feature (disabled by default)
  • experimental.htmlLimitedBots optional (RegExp): Specifies a regex to identify bot user agents that should opt out of the streaming metadata feature and continue receiving fully blocking metadata in the HTML. (We have a list of common bots which have limited ability to handle metadata rendering)

By default, only the Google bots which can act like headless browsers will receive streaming metadata when you enable the feature. Other bots will receive blocking metadata like the ones configured in htmlLimitedBots. If you noticed any bots also need to consume blocking metadata, please let us know to add it into default list.

If you don’t need to customize the bots user agents for the blocking metadata and default config is enough for you, then only need to have one flag enabled in your next.config.js

module.exports = {
  experimental: {
    streamingMetadata: true,
  },
}

Notes

We’re still working on optimizations for PPR mode. However, if you’re not using PPR, you can try out this feature now to accelerate pages with slow generateMetadata() calls.

We’d love to hear your thoughts! Please feel free to share any comments, suggestions, or concerns in this thread. We’d love to hear your feedbacks! Thank you! 🚀

We’re also seeing this issue, which is a pretty big issue with for us. We have slow endpoints that generate our metadata and right now it feels like a tradeoff between bad SEO and using this feature. Is there a workaround to get metadata on the page without blocking or getting a loading ui/suspense boundary for page navigation?

I don’t know if it helps but it seems the rsc is called twice in a row. If we have generateMetadata on, the first request takes as long as the metadata does to return, which doesn’t seem to be caught by any suspense/loading ui boundary. If we take out the generateMetadata, both requests are still fired, but the first one is nearly instant with the second one taking some time but that is correctly picked up by the suspense loading boundary.

This should be a very commonly used feature, and it is surprising that the official team has not provided any fixes.

Same issue here. At least I expect generateMetadata to trigger suspense, and also I would like to bypass calling generateMetadata on client-side navigation, show the page instantly, and update metadata in the background.

Hey, I wanted to give a few updates on this.

  • like @borispoehland, if the data is cache-able you should use unstable_cache to cache it
  • however that still will cause the first hit to be slow, since it won’t be cached, this is a known limitation. When we release PPR, you’ll be able to statically pregenerate the metadata and combine it with a dynamic/ssr page.
  • if the data truly can’t be cached, we’re also planning on making metadata streamable, which means that we would not block on it during render, allowing you to serve the response asap and potentially the metadata 6s+ after, like some cases in this thread.

The problem is very clear in the following code snippet using app router.

  1. Page got stuck for 2 seconds in loading state while generateMetadata and TestContent are running in parallel
  2. When generateMetadata finish, <Suspense> start running for 2 seconds waiting TestContent to finish.
  3. The page renders.

The first point is the problem, because <Suspense> must start running from the beginning and the full page with its metadata should render together.

import { Suspense } from "react"

export async function generateMetadata() {
    await new Promise(resolve => setTimeout(resolve, 2000))
    return {
        title: 'Test',
        description: 'Test',
    }
}

async function TestContent() {
    await new Promise(resolve => setTimeout(resolve, 4000))
    return  <h1>Test</h1>
}

export default async function Test() {
    return (
        <Suspense fallback={<div>loading</div>}>
            <TestContent />
        </Suspense>
    )
}

Neither latest nor canary are working as expected. With this issue is impossible to create an elegant, efficient and SEO friendly page with app router. Please help.

I’m working on a pretty big next.js app that’s on pages router. Our pages would look somewhat like this:

my-app/
├─ src/
│  ├─ pages/
│  │  ├─ foo/
│  │  │  ├─ bar/
│  │  │  │  ├─ index.tsx (ssr with `getServerSideProps`)
│  │  │  ├─ index.tsx (ssr with `getServerSideProps`)

Our biggest pain with pages router is that getServerSideProps of /foo/bar would run every time when the user navigates from /foo page to /bar even though bar page shares a lot of data with foo page. This resulted in a sluggish navigation experience where the user will receive no UX feedback for ~500ms (that’s how long our roundtrip time takes). However, pages router had routeChangeStart and routeChangeComplete event listeners which we could use to know if the user requested a client side navigation. We used that to show a loader to the user in the /foo page before /foo/bar page loaded. It was not ideal, as ideal would be able to show the loader in /foo/bar page but it was still better than not showing the user any feedback at all when we’re waiting for the server response to continue with the navigation.

We were super excited when app router was announced with persisted layouts. We waited two years to give it some time for app router to be stable. We recently did an app router migration spike and we found that navigation from /foo to /foo/bar was super fast because of two reasons:

  1. We can create a layout.tsx file for /foo that would persist when we navigate to /foo/bar so the bar page only has to load data necessary to render bar
  2. Because of Suspense, we were able to take the user to the /foo/bar page as soon as the user clicked/pressed the link to the page and show a loader while the data was loaded for the bar page.

But when we experimented with app router we only tested the layouts and pages, we did not test metadata as the docs did not mention anything related to generateMetadata function blocking client side navigations. It only mentions that the initial load will be blocked before the data can be streamed. It does not mention anything about suspense loader getting blocked during client side transitions.

On the initial load, streaming is blocked until generateMetadata has fully resolved, including any content from loading.js

We’re working on a very big app. Our team spent a few weeks migrating some of our pages from pages router to app router. With app router, our pages look somewhat like this:

my-app/
├─ src/
│  ├─ app/
│  │  ├─ foo/
│  │  │  ├─ bar/
│  │  │  │  ├─ page.tsx (has generateMetadata() and generateViewport())
│  │  │  ├─ page.tsx (has generateMetadata() and generateViewport())
│  │  │  ├─ layout.tsx

Things were looking good until we added the generateMetadata function. After adding the function to bar page the user receives no feedback at all until the server response comes back after navigation. The user is now just waiting there not knowing what happened after clicking/pressing on a link for approxmiately ~400ms when their connection is fast, when it’s slow, the user is waiting without any feeback for ~2s. This is terrible for UX. Unlike pages router, we don’t have something like routeChangeStart and routeChangeComplete in the app router, so we can’t show any loading UX even on the current page. Now we’re caught between the pages router and the app router world with little to no hope.

@feedthejim none of the options you suggested would work for us. It’s a self-hosted app. Our data is very fast and cached in a different layer and not in the render layer for freshness reasons. We can’t use unstable_cache, PPR or use cache. We don’t want to bother setting up our infra with streaming either. We just want to be able to server render a fully dynamic page with persisted layouts and not have generateMetadata and generateViewport functions block the Suspense loader. Our UX was a lot better in pages router even though we were loading more data than necessary during client-side navigations to /foo/bar page.

Same issue here. At least I expect generateMetadata to trigger suspense, and also I would like to bypass calling generateMetadata on client-side navigation, show the page instantly, and update metadata in the background.

I do hope this issue gets attention, as @ArianHamdi’s comment is exactly what I hoped/expected the functionality to be.

@Matt-roki , in the meantime, I was able to implement a work-around inspired by https://github.com/vercel/next.js/issues/45418#issuecomment-1914076788 that has seemed to resolve our issues.

Great news, thank you @huozhi! One small source of confusion: would setting htmlLimitedBots override default ones? It feels a bit scary to edit this setting after reading https://nextjs.org/docs/app/api-reference/functions/generate-metadata#streaming-metadata.

It’d be also great to know what the default list is and under what principles it can be updated. For example, if I discover a regional bot (e.g. in a country with 50M people), would it go to the ‘official’ Next.js list via a PR? Or is built-in list dedicated to global bots only?

Hi everyone, we’ve shipped the streaming metadata as the default behavior to canary release, you don’t need to have that experimental configuration streamingMetadata anymore. If you still need to customized the html limited bots, you can directly update htmlLimitedBots in next.config.js which is a stable configuration now.

@sean-ahn Do you have any special search engine that not covered in the default list that needs to be addressed? We can add them into the list. Ideally you don’t have to override them most of the time and defaut list can cover them.

👋 Hi everyone! We heard your requests.

We’re introducing Streaming Metadata as the enhancement of the recent canary releases to improve metadata rendering performance.

Introduction

This feature is designed to improve page loading time for users, especially when dealing with slow metadata generation. With Streaming Metadata enabled:

  • The generateMetadata returned metadata is treated as suspenseful data, allowing the page content to render immediately.
  • Metadata is then injected asynchronously into the page on the client side once it’s resolved.

Since metadata is critical for SEO, we’ve ensured that bots continue to receive fully rendered metadata in the HTML. For human users, who primarily care about page content, metadata can be added later without impacting their experience.

Usage

The following new options in next.config.js are for configuring Streaming Metadata:

  • experimental.streamingMetadata (boolean): Enables the Streaming Metadata feature (disabled by default)
  • experimental.htmlLimitedBots optional (RegExp): Specifies a regex to identify bot user agents that should opt out of the streaming metadata feature and continue receiving fully blocking metadata in the HTML. (We have a list of common bots which have limited ability to handle metadata rendering)

By default, only the Google bots which can act like headless browsers will receive streaming metadata when you enable the feature. Other bots will receive blocking metadata like the ones configured in htmlLimitedBots. If you noticed any bots also need to consume blocking metadata, please let us know to add it into default list.

If you don’t need to customize the bots user agents for the blocking metadata and default config is enough for you, then only need to have one flag enabled in your next.config.js

module.exports = { experimental: { streamingMetadata: true, }, }

Notes

We’re still working on optimizations for PPR mode. However, if you’re not using PPR, you can try out this feature now to accelerate pages with slow generateMetadata() calls.

We’d love to hear your thoughts! Please feel free to share any comments, suggestions, or concerns in this thread. We’d love to hear your feedbacks! Thank you! 🚀

We ran into issues with metadata tags not being set correctly when a bot requests the page.

"next": "15.2.0-canary.20",

Steps to reproduce: - Enable streamingMetadata setting - Set open graph meta tags - Use a valid bot (discord, twitter, etc…) and post a link so it tries to generate open graph

Expected: - Bot correctly displays OG embed with title, description and image

Actual: - Bot can’t read any of the metadata tags


We checked manually as well by getting a page with metadata with the discordbot user agent (Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)) and we can confirm the metadata tags do not get set on <head>. We tried this both on a staging environment and a local environment.

We can also confirm that user agent is being passed correctly when we trigger discord to embed the link. We further tested it by passing a custom regex in htmlLimitedBots option but that doesn’t work as well

Edit: We turned off streamingMetadata and surprisingly we run into the same issue, the meta tags are missing on <head>

We’re introducing Streaming Metadata as the enhancement of the recent canary releases to improve metadata rendering performance.

yesss, this makes me so happy! e-comm style apps with slow metadata has been one of my biggest gripes with Next.js as it absolutely tanks INP for link navigation. I just migrated the entirety of emojis.com to the latest canary with streamingMetadata and results are positive!

Thank you for the update @feedthejim .

While those enhancements seem worthy, they also don’t seem like quick wins. Did you consider a solution similar to what @ArianHamdi wrote above (at least in the short-term):

I would like to bypass calling generateMetadata on client-side navigation, show the page instantly, and update metadata in the background.

I think that would go a long way to alleviating this issue, as we would be paying the cost on first-load only, whereas now it’s per-page.

Same issue, couldn’t generateMetadata be called compile time as well with a revalidation?