php-vips: SEGFAULT after the inclusion of FFI::shutdown() to release memory consumption

Running libvips 8.13.3 on Debian bullseye with PHP 8.1 under Apache using jcupitt/vips:2.10

Calling FFI::shutdown() at the end of a script after an Image::Thumbnail() operation results in the process SEGFAULT-ing.

The call to shutdown() is required as without it the memory consumption climbs and is never released back to the system (at least from our experiments).

There are more details and a full reproduction case, including Dockerfile to match the environment: https://github.com/ingenerator/libvips-segfault-repro

Thanks to @acoulton: Failing tests proving either case here: https://github.com/ingenerator/libvips-segfault-repro/actions/runs/3444654670/jobs/5747485403

And a further attempt to debug the source of the SEGFAULT here: https://github.com/ingenerator/libvips-segfault-repro/pull/3

We have followed as much of the documentation as we can find around usingphp-vips appropriately however, it may be that it is not suited to running in this environment, in this way (ie mpm-prefork / multiuser). While happy to spend a little more time debugging, if you have any suggestions of where to go next with it? It would be good to know if it is at least intended to work this way?

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 22 (15 by maintainers)

Most upvoted comments

So, no solution to growing memory consumption over multiple requests?

My PHP-FPM workers grow over 2GB after less than 200 requests. Using php-vips 1.0.10 to shrink JPEG-XL images (from about 4000*3000 to 2400 by larger side), overlay them with PNG image via composite2 and encode them to legacy JPEG to output to browser.

vips_cache_set_max(1), vips_cache_set_max_files(1) do not help .

Had to set php-fpm pm.max_requests to 200 to respawn worker processes otherwise PHP-FPM gets oom_killed…

PHP 8.2.13 with php-vips binary extension libvips 8.15 (built from sources) libjxl 0.8.2 (built from sources) libspng 0.7.3 Debian 12

DiamondMax on Dec 14, 2023

I think this is a memory fragmentation issue. You might consider using a different memory allocator such as jemalloc to prevent memory fragmentation in long-running, multi-threaded, glibc-based Linux processes. See https://github.com/lovell/sharp/issues/955 to learn more about memory allocation and why the choice of a memory allocator is important.

You could also consider switching to a Alpine-based Docker image, as the memory allocator in musl, upon which Alpine Linux is based, is generally considered very good in terms of lack of fragmentation and returning freed memory.

For example, with these changes: https://github.com/ingenerator/libvips-segfault-repro/compare/test-php-vips-pull-171...kleisauke:alpine-based

I see:

Result summary:

  - # 1 HTTP:200 Memory: 10.19MB / 33.6GB
  - # 2 HTTP:200 Memory: 15.7MB / 33.6GB
  - # 3 HTTP:200 Memory: 15.84MB / 33.6GB
  - # 4 HTTP:200 Memory: 15.96MB / 33.6GB
  - # 5 HTTP:200 Memory: 15.99MB / 33.6GB
  - # 6 HTTP:200 Memory: 16.02MB / 33.6GB
  - # 7 HTTP:200 Memory: 16.02MB / 33.6GB
  - # 8 HTTP:200 Memory: 16.05MB / 33.6GB
  - # 9 HTTP:200 Memory: 16.07MB / 33.6GB
  - # 10 HTTP:200 Memory: 16.08MB / 33.6GB
  Final Memory: 15.54MB / 33.6GB

So, much less memory usage than glibc.

kleisauke on Nov 14, 2022