gotenberg: Memory leak route "POST /forms/chromium/convert/url"

Appreciation

We are running Gotenberg in production. A lot of appreciation from our team for this wonderful project enabling many URL->PDF conversions per day for our endusers.

Context

We are running it on AWS FARGATE with the following configuration:

{
    "cpu": 0,
    "essential": true,
    "stopTimeout": 120,
    "command": [
      "gotenberg",
      "--api-timeout=480s",
      "--chromium-max-queue-size=20",
      "--chromium-auto-start=true",
      "--chromium-clear-cache=true",
      "--chromium-clear-cookies=true",
      "--libreoffice-disable-routes=true",
      "--webhook-disable=true",
      "--prometheus-disable-route-logging=true",
      "--prometheus-disable-collect=true",
      "--log-level=debug"
    ]
  }

Gotenberg version 8.

CPU and RAM on the high-end. image

The only url being hit is “POST /forms/chromium/convert/url” (and /health). The urls (supplied by our endusers so unknown content and we can’t share) do probably contain some heavy HTML.

Our memory consumption on AWS increases slowly over time. Idem the risk of getting “convert URL to PDF: convert to PDF: context deadline exceeded” error. The dips are automatic or manual restarts.

image

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 41

Most upvoted comments

Seems stable enough. I see growth and then I assume the sudden drop is chromium restarting

Screenshot 2025-01-07 at 10 57 31 AM

Before this I was not using flag and memory would just continue to grow until pod was restarted.

I am using --chromium-restart-after=100 on 8.15.3 and seeing much better memory usage

@gulien I would need some long term testing of around 2 weeks, to provide the actual “gain”.

Release 8.16.0 brings a default value for the restart strategy (10). Thanks all for the help!

@gulien Here a before and after deployment, marker in the center is the deployment. Ignore the time around 7th, there was an internal issue

I think due to the short life time, I don’t see that much.

Image Image

Hey @gulien, I can confirm that memory usage is much more stable after upgrading to 8.15.3. I will monitor the situation over the next few days, but it all looks promising.

Also updated to 8.15.3-cloudrun using --chromium-restart-after=50

Hey, sure. I was going to do that this week or next week.

@michalschroeder it might leak zombie processes.

Hey, we have a similar issue. We are using the POST /forms/chromium/convert/html endpoint to generate PDF. We will try to use some of the advice you have mentioned here but just wanted to leave it here for context how it looks on our end. Memory usage has always been increasing, but when we migrated to v8 in early November (from the v7.10.2 to the latest v8), memory usage increased significantly.

image

Thanks @Fank! I guess it’s about the same load as before? Are you able to quantify the “gain” somehow (like, 10%)?

@Fank @ldlamarc as a workaround, would the solution described in #987 (comment) make sense for you?

Restart after x time is probably something where my first reflex would be to try to do it on the container orchestration side (AWS).

The images gotenberg/snapshot:fixchromium-memory-leak and gotenberg/snapshot:fixchromium-memory-leak-cloudurn have been rebased with the main branch.

I’ve also deployed this version to the demo instance to monitor the memory leaking, although it did not really appear on this particular instance.

I am using --chromium-restart-after=50 now and will let you know how it will looks like after some days.

I noticed something similar image

We are only using the chromium convert API to generate PDFs out of HTML files. The longer an instance is running the more memory it will consume.

The is how we are running gotenberg:

          gcloud run deploy gotenberg
          --image=gotenberg/gotenberg:8-cloudrun
          --args=gotenberg
          --args="--api-port-from-env=PORT"
          --memory=1Gi

If an instance uptime is too long it hits the memory limit: image

Due to the autoscaling of google cloud run, its a rare case but still happens sometimes.

Hello @ldlamarc,

Chromium tends to have memory leaks. Last time, it was because Gotenberg was automatically restarting it (it was also leading to zombie processes - even with tini). However, with latest versions, it may have change again.

Maybe try using --chromium-restart-after=10 (or any value that makes sense for you) to see if it mitigates the issue.