next.js: `next/og` cause memory leak in production standalone build

Link to the code that reproduces this issue

https://github.com/Innei/next-og-oom-repro

To Reproduce

  1. build project
  2. run as standalone build node server.js
  3. open /og
  4. open devtools and force refresh the page(ignore cache) more times.
  5. watch the system monitor, and the next app memory usage is increasing.

The initial memory usage about to 50M, and refresh /og about 10 times, got 300M CleanShot 2024-05-07 at 8  26 37@2x

CleanShot 2024-05-07 at 8  30 54@2x

I can provide some ways to try to troubleshoot memory issues.

Add memory dump code in .next/standalone/server.js


process.title = 'next-og-oom'

const v8 = require('v8')
const fs = require('fs')

function createHeapSnapshot() {
  const snapshotStream = v8.getHeapSnapshot()
  const timestamp = new Date().toISOString().replace(/[:\.]/g, '-')
  const fileName = `/tmp/${timestamp}.heapsnapshot`
  const fileStream = fs.createWriteStream(fileName)
  snapshotStream.pipe(fileStream).on('finish', () => {
    console.log('Heap snapshot saved to', fileName)
  })
}

process.on('SIGUSR2', () => {
  console.log('SIGUSR2 received, creating heap snapshot...')
  createHeapSnapshot()
})

And then refresh the page and refresh the page several times and observe the app memory usage afterwards. When memory overflows and is not freed, hit the heap of memory at that point with kill -SIGUSR2 <pid>.

As you can see from the following dump, it’s the ImageResponse-related modules that are leaking memory. ImageResponse and FigmaImageResponse that means @vercel/og causes memory leak?

Link to https://github.com/vercel/next.js/issues/44685#issuecomment-2098076162.

Current vs. Expected behavior

  1. Memory does not free up and increases more and more, finally oom.

expected:

Memory can be freed up.

Provide environment information

Operating System:
  Platform: darwin/linux
  Arch: arm64/amd64
  Version: Darwin Kernel Version 23.4.0: Fri Mar 15 00:12:49 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6020/ or Linux
  Available memory (MB): 32768
  Available CPU cores: 12
Binaries:
  Node: 18.18.0/20.x
  npm: 10.2.4
  Yarn: 1.22.21
  pnpm: 9.1.0
Relevant Packages:
  next: 14.2.3 // Latest available version is detected (14.2.3).
  eslint-config-next: N/A
  react: 18.3.1
  react-dom: 18.3.1
  typescript: 5.4.5
Next.js Config:
  output: standalone

Which area(s) are affected? (Select all that apply)

Image (next/image)

Which stage(s) are affected? (Select all that apply)

Other (Deployed)

Additional context

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 13
  • Comments: 26 (9 by maintainers)

Most upvoted comments

The fix is patched in v14.2.13, please upgrade to the new version 🙏

Self assigned - will take a look!

@khuezy If you dig a little deeper, there is an open issue in resvg-wasm package which has identified a memory leak: https://github.com/yisibl/resvg-js/issues/216#issuecomment-2188877664

We have applied the above patch to @vercel/og manually in our project and yet we keep experience heavy memory leak. The above solution is still better than no patch, but this package doesn’t seem to be the inherent cause of the leak.

@huozhi I’m not sure if I should comment here or open a new issue. Please redirect me if needed!

We bumped up to v14.2.13, but continue to see the memory leak running nextjs in a long-lived docker container. The upgrade did slow the memory leak a bit, but it still appears to be present. This app only has a few server-side routes, all of which are using the @vercel/og package.

image

I don’t think it’s related to Docker images - I’m deploying on fly.io and they only use the Dockerfile to pull down the dependencies, then they run the stack raw on firecracker. I have a typescript temporal worker stack running on node-22.5.1 and memory has been stable… so I think this is a reactjs or nextjs leak (probably the latter)

@ethos-seth please import from next/og instead of @vercel/og. If you’re still having the issue, please file a new issue with repro where you monitored it has memory leak

Hey @Eric-Arellano, I’ve seen a few issues on the next/image optimization leakage too. But my app continues to grow in memory over time, when it does garbage collect the memory drops but is always a little higher than the previous GC, so over time is continues to grow.

I have a rewrite and a custom image loader that routes all /_next/img to a non-js server than handles the image optimization and there still seems to be a leak. So there might be multiple instances of leaks somewhere in the javascript, reactjs, or nextjs stack.

Thanks @shuding! This approach worked well for us https://github.com/orgs/vercel/discussions/6117#discussioncomment-8776252. After applying the patch, our memory stopped growing as aggressively.

Screenshot 2024-08-05 at 19 42 46

Thanks @shuding! This approach worked well for us https://github.com/orgs/vercel/discussions/6117#discussioncomment-8776252. After applying the patch, our memory stopped growing as aggressively.

Screenshot 2024-08-05 at 19 42 46

That fix the memory leak for me too. Thanks!

Version: node:20.12.0-alpine

Try with 18-alpine, starting from version 19, I have the same memory leak issue, but everything is fine with version 18. 🤷‍♂️

Is it normal for the memory to grow like yours after the patch?

(Coworker of Frank). No, I don’t think it’s normal. Our usage stats suggest it is not due to more usage over time, but rather we suspect a memory leak. We’re still trying to figure out what causes it, with two leading hypotheses:

Possible fix already sitting on your doorstep for free since March 13: https://github.com/orgs/vercel/discussions/6117#discussioncomment-8776252

Hi @chipcop106

What’s the metrics tracing platform? Are you using Sentry or Datadog? 👀

I’m using AWS ECS, which is the metrics of the docker’s container.

Here is our memory chart when using the Next 14.2.3. I confirmed a memory leak issue with the open graph and Twitter images. From 27/6, we disable this function, and this chart becomes normal.

image

Solution

Please don’t use dynamically generated metadata images using tsx, js, or ts until fixed.