gotenberg: Memory leak route "POST /forms/chromium/convert/url"
Appreciation
We are running Gotenberg in production. A lot of appreciation from our team for this wonderful project enabling many URL->PDF conversions per day for our endusers.
Context
We are running it on AWS FARGATE with the following configuration:
{
"cpu": 0,
"essential": true,
"stopTimeout": 120,
"command": [
"gotenberg",
"--api-timeout=480s",
"--chromium-max-queue-size=20",
"--chromium-auto-start=true",
"--chromium-clear-cache=true",
"--chromium-clear-cookies=true",
"--libreoffice-disable-routes=true",
"--webhook-disable=true",
"--prometheus-disable-route-logging=true",
"--prometheus-disable-collect=true",
"--log-level=debug"
]
}
Gotenberg version 8.
CPU and RAM on the high-end.
The only url being hit is “POST /forms/chromium/convert/url” (and /health). The urls (supplied by our endusers so unknown content and we can’t share) do probably contain some heavy HTML.
Our memory consumption on AWS increases slowly over time. Idem the risk of getting “convert URL to PDF: convert to PDF: context deadline exceeded” error. The dips are automatic or manual restarts.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 41
Seems stable enough. I see growth and then I assume the sudden drop is chromium restarting
Before this I was not using flag and memory would just continue to grow until pod was restarted.
I am using
--chromium-restart-after=100on8.15.3and seeing much better memory usage@gulien I would need some long term testing of around 2 weeks, to provide the actual “gain”.
Release 8.16.0 brings a default value for the restart strategy (10). Thanks all for the help!
@gulien Here a before and after deployment, marker in the center is the deployment. Ignore the time around 7th, there was an internal issue
I think due to the short life time, I don’t see that much.
Hey @gulien, I can confirm that memory usage is much more stable after upgrading to 8.15.3. I will monitor the situation over the next few days, but it all looks promising.
Also updated to
8.15.3-cloudrunusing--chromium-restart-after=50Hey, sure. I was going to do that this week or next week.
@michalschroeder it might leak zombie processes.
Hey, we have a similar issue. We are using the
POST /forms/chromium/convert/htmlendpoint to generate PDF. We will try to use some of the advice you have mentioned here but just wanted to leave it here for context how it looks on our end. Memory usage has always been increasing, but when we migrated to v8 in early November (from thev7.10.2to the latestv8), memory usage increased significantly.Restart after x time is probably something where my first reflex would be to try to do it on the container orchestration side (AWS).
The images
gotenberg/snapshot:fixchromium-memory-leakandgotenberg/snapshot:fixchromium-memory-leak-cloudurnhave been rebased with themainbranch.I’ve also deployed this version to the demo instance to monitor the memory leaking, although it did not really appear on this particular instance.
I am using
--chromium-restart-after=50now and will let you know how it will looks like after some days.I noticed something similar
We are only using the chromium convert API to generate PDFs out of HTML files. The longer an instance is running the more memory it will consume.
The is how we are running gotenberg:
If an instance uptime is too long it hits the memory limit:
Due to the autoscaling of google cloud run, its a rare case but still happens sometimes.
Hello @ldlamarc,
Chromium tends to have memory leaks. Last time, it was because Gotenberg was automatically restarting it (it was also leading to zombie processes - even with tini). However, with latest versions, it may have change again.
Maybe try using
--chromium-restart-after=10(or any value that makes sense for you) to see if it mitigates the issue.