gotenberg: docx conversion on debian does not work | Libreoffice crash

Hi,

thank you for such an great tool! It works fine in my development environment (ubuntu wsl and ubuntu).

But on our production system I get only an “internal Server Error”.


  _____     __           __
 / ___/__  / /____ ___  / /  ___ _______ _
/ (_ / _ \/ __/ -_) _ \/ _ \/ -_) __/ _ '/
\___/\___/\__/\__/_//_/_.__/\__/_/  \_, /
                                   /___/

A Docker-powered stateless API for PDF files.
Version: 8.0.2
-------------------------------------------------------
[SYSTEM] modules: api chromium libreoffice libreoffice-api libreoffice-pdfengine logging pdfcpu pdfengines pdftk prometheus qpdf webhook
[SYSTEM] pdfengines: libreoffice-pdfengine pdfcpu pdftk qpdf
[SYSTEM] prometheus: collecting metrics
[SYSTEM] libreoffice-api: LibreOffice ready to start
[SYSTEM] api: server listening on port 3000
[SYSTEM] chromium: Chromium ready to start
{"level":"debug","ts":1706091729.0305188,"logger":"api.formslibreofficeconvert","msg":"form fields: map[]","trace":"d7073679-33f7-4943-9731-65a26ed76780"}
{"level":"debug","ts":1706091729.0306876,"logger":"api.formslibreofficeconvert","msg":"form files: map[dummy.docx:/tmp/51178158-7b89-4f40-89d4-7a55af6d0faa/615b0125-c4cd-4770-a33f-dd8677df2123/dummy.docx]","trace":"d7073679-33f7-4943-9731-65a26ed76780"}
{"level":"debug","ts":1706091729.030958,"logger":"api.formslibreofficeconvert","msg":"process lock acquired","trace":"d7073679-33f7-4943-9731-65a26ed76780"}
{"level":"debug","ts":1706091729.031098,"logger":"libreoffice-api.libreoffice","msg":"start process"}
{"level":"debug","ts":1706091729.0316796,"logger":"libreoffice-api.libreoffice.usrliblibreofficeprogramsoffice.bin","msg":"start unix process: /usr/lib/libreoffice/program/soffice.bin --headless --invisible --nocrashreport --nodefault --nologo --nofirststartwizard --norestore -env:UserInstallation=file:///tmp/709497b2-2c99-483f-95f0-bddeea39f0bb/10ade9be-20e8-4259-9505-d3ef4fa8a6c5 --accept=socket,host=127.0.0.1,port=42351,tcpNoDelay=1;urp;StarOffice.ComponentContext"}
{"level":"debug","ts":1706091729.1251678,"logger":"libreoffice-api.libreoffice.usrliblibreofficeprogramsoffice.bin.stderr","msg":"terminate called after throwing an instance of 'std::runtime_error'"}
{"level":"debug","ts":1706091729.1252174,"logger":"libreoffice-api.libreoffice.usrliblibreofficeprogramsoffice.bin.stderr","msg":"  what():  osl::Thread::create failed"}
{"level":"debug","ts":1706091729.1448293,"logger":"libreoffice-api.libreoffice.usrliblibreofficeprogramsoffice.bin","msg":"unix process already killed"}
{"level":"debug","ts":1706091729.1448631,"logger":"api.formslibreofficeconvert","msg":"process lock released","trace":"d7073679-33f7-4943-9731-65a26ed76780"}
{"level":"debug","ts":1706091729.1452055,"logger":"api.formslibreofficeconvert","msg":"'/tmp/51178158-7b89-4f40-89d4-7a55af6d0faa/615b0125-c4cd-4770-a33f-dd8677df2123' context's working directory removed","trace":"d7073679-33f7-4943-9731-65a26ed76780"}
{"level":"error","ts":1706091729.1452672,"logger":"api","msg":"convert to PDF: process first start: start process: execute LibreOffice: unix process error: wait for unix process: signal: aborted (core dumped)","trace":"d7073679-33f7-4943-9731-65a26ed76780","remote_ip":"172.17.0.1","host":"localhost:3000","uri":"/forms/libreoffice/convert","method":"POST","path":"/forms/libreoffice/convert","referer":"","user_agent":"GuzzleHttp/7","status":500,"latency":117286886,"latency_human":"117.286886ms","bytes_in":174496,"bytes_out":21}

Our production server is an debian with 4.9.0.19-amd64 kernel. Docker version is 19.03.12.

According to the documentation I checked the ressources but there is no limitation. I think 32GB memory should be enough. 😉

So I tried to track down the problem. If I start the container with --libreoffice-auto-start enabled, the container didn’t start.


  _____     __           __
 / ___/__  / /____ ___  / /  ___ _______ _
/ (_ / _ \/ __/ -_) _ \/ _ \/ -_) __/ _ '/
\___/\___/\__/\__/_//_/_.__/\__/_/  \_, /
                                   /___/

A Docker-powered stateless API for PDF files.
Version: 8.0.2
-------------------------------------------------------
[SYSTEM] modules: api chromium libreoffice libreoffice-api libreoffice-pdfengine logging pdfcpu pdfengines pdftk prometheus qpdf webhook
[SYSTEM] chromium: Chromium ready to start
[SYSTEM] pdfengines: libreoffice-pdfengine pdfcpu pdftk qpdf
{"level":"debug","ts":1706091982.3399339,"logger":"libreoffice-api.libreoffice","msg":"start process"}
[SYSTEM] prometheus: collecting metrics
{"level":"debug","ts":1706091982.3410268,"logger":"libreoffice-api.libreoffice.usrliblibreofficeprogramsoffice.bin","msg":"start unix process: /usr/lib/libreoffice/program/soffice.bin --headless --invisible --nocrashreport --nodefault --nologo --nofirststartwizard --norestore -env:UserInstallation=file:///tmp/7469d30f-d126-4c27-94ab-e54f9de264be/34df5b55-398d-4cbd-bfa8-705e9c4c5e8f --accept=socket,host=127.0.0.1,port=43647,tcpNoDelay=1;urp;StarOffice.ComponentContext"}
{"level":"debug","ts":1706091982.3984182,"logger":"libreoffice-api.libreoffice.usrliblibreofficeprogramsoffice.bin.stderr","msg":"terminate called after throwing an instance of 'std::runtime_error'"}
{"level":"debug","ts":1706091982.3984692,"logger":"libreoffice-api.libreoffice.usrliblibreofficeprogramsoffice.bin.stderr","msg":"  what():  osl::Thread::create failed"}
[FATAL] starting libreoffice-api: launch supervisor: start process: execute LibreOffice: unix process error: wait for unix process: signal: aborted (core dumped)
{"level":"debug","ts":1706091982.4568202,"logger":"libreoffice-api.libreoffice.usrliblibreofficeprogramsoffice.bin","msg":"unix process already killed"}

The are some tickets which suggest to increase --libreoffice-start-timeout. I increased it up to 400 seconds but still no luck.

Next I checked the LibreOffice Version with docker exec -it gotenberg libreoffice --version and got ERROR 4 forking process.

For me, it seems like that libreoffice is unable to start. Thats why no conversion is possible. But how to fix it?

At the moment I’m out of ideas how to get gotenberg running.

I would be very appreciate for any hints. 😃

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 36

Most upvoted comments

I got a quick response from our server provider. We are using Debian 9. The highest possible docker version is 19.03.15. But the problem still exists with 19.03.15. I will try to reproduce the error on an other system and will report as soon I have an result.

I have first results:

  • on an new installed debian 9 (stretch) VM, Gotenberg fails. Same error as in our production environment.
  • on a new installed debian 10 (buster) VM, there is no error. Also if the docker components have the same version like the debian 9 machine

So this error occurs only on debian 9 (stretch). I will try to find a workaround an will report it here, if there is any. If not, it’s ok for me too. We will upgrade the production os asap. Maybe a note in the documentation would be an option.

I mean, in your case, you may just spin up more instances by default. Best option is to check que gotenberg_libreoffice_requests_queue_size overtime in a day, and find out what could be the best number of instances to handle all your scenarios. The same metric may be used to auto-scale your number of instances, but maybe that’s overkill for your use case.

I understand your reasoning. Thanks for the input. I will just roll back because each of my servers have 64 cores and 256g memory at my disposal. I can just tune docker to have no limits. Which is much easier than configuring a whole orch setup (which will end up using the same amount of resources anyway).

Thanks again, and hopefully AlexKvrlp gets his stuff fixed up.

EDIT: I will spin up a prometheus instance so i can see what my load really looks like (a lot of these conversions are small 25k docx files).

Hi,

In all instances (using your backport, and vanilla), the logged error is convert to PDF: context deadline exceeded. The timeout is already 90s … i just increased it to 120s to eliminate that from being the issue (not having enough time, 120 should be more than enough for our small pdf conversion jobs).

I did see something on SO about someone testing routes in Go, and if too many came in, it would throw that error almost instantly… it seemed related to the context timeout being tied to ALL of them, and not just a single one.

During the right times of day, it would be possible for me to send enough docs to the api that it might just be hitting that 90s for all in queue (and not 90 per item).

I dont know Go well enough to see if that is the case with your code.

Lastly, i can fall back to 7.4.2, i dont want to take up too much time if im an edge case.

EDIT Actually… reviewing the curl logs of the failed calls, the upload content seems too small. I will take a look at the files being sent… maybe this is a goose chase and something else is wrong on my end.

I got a quick response from our server provider. We are using Debian 9. The highest possible docker version is 19.03.15. But the problem still exists with 19.03.15. I will try to reproduce the error on an other system and will report as soon I have an result.