ruby-vips: memory leak when writing

I apologize if this is a very basic question. I’m using ruby-vips in what I think is a very simplistic way to process a few thousand TIFF files and convert them to PNG. It does convert files to PNG but fairly rapidly gobbles up all available memory:


requite 'vips'

for file in Dir.foreach('image_dir') do |filename|
  next if File.extname(file) != '.tif'
  png = tiff.sub '.tif', '.png'
  img = VIPS::Image.new tiff
  img.write png
end

Is there something I’m doing wrong here? I noticed that if I comment out the write it is able to successfully read in all the images, so it appears to be caused by writing the PNG file?

About this issue

Original URL
State: closed
Created 9 years ago
Comments: 51 (49 by maintainers)

Commits related to this issue

tune GC a bit see https://github.com/jcupitt/ruby-vips/issues/67#issuecomment-172341601 — committed to libvips/ruby-vips by jcupitt 8 years ago
more GC tuning on ruby2.1 and later, do an incremental GC on every write on older ruby, full GC every 100 writes see https://github.com/jcupitt/ruby-vips/issues/67 — committed to libvips/ruby-vips by jcupitt 8 years ago
tune GC a bit see https://github.com/jcupitt/ruby-vips/issues/67#issuecomment-172341601 — committed to oleksandrbyk/olek-ruby-vips by oleksandrbyk 8 years ago
more GC tuning on ruby2.1 and later, do an incremental GC on every write on older ruby, full GC every 100 writes see https://github.com/jcupitt/ruby-vips/issues/67 — committed to oleksandrbyk/olek-ruby-vips by oleksandrbyk 8 years ago

Most upvoted comments

@ioquatix How did you arrive at the 30 MiB buffer size and shouldn’t the value in release be negative?

Btw. a shortcut for specifying KiB, MiB etc.: 1<<10 KiB, 1<<20 MiB, 1<<30 GiB.

felixbuenemann on Mar 24, 2020

You could try to use ruby-vips8, it has much better memory management. I’ve successfully used it to do batch processing of thousands of images.

felixbuenemann on May 13, 2016

Hey all- Thanks you for your hard works on the ruby-vips gem.

I’m also seeing a strange case when I load pdfs… I’m converting each page to its own image. After browsing through this thread I removed the write to see if that’s where the memory was globbing but it’s actually in the read. It looks like every call to pdfload is storing the whole pdf in memory. Not sure if this is a GC issue or this is the right thread to post it.

    page_count = Vips::Image.pdfload("test.pdf", access: :sequential).get("n-pages")
    page_count.times do |p|
      im = Vips::Image.pdfload("test.pdf", page: p, n: 1, access: :sequential)
      im.write_to_file("buffer/test.#{p}.png")
    end

In this case, I don’t actually see a GC happen. The file is 298 pages and 5.5mb

starting mem: 39mb
page: 0 - Memory: 52.38671875mb
page: 1 - Memory: 57.9375mb
page: 2 - Memory: 63.46875mb
-snip-
page: 297 - Memory: 1701.53125mb

jalevin on May 27, 2020

Okay, I was playing around with it a bit more. It turns out if you inform the GC of the buffer size, it will do a better job.

module GC
	extend FFI::Library
	ffi_lib FFI::CURRENT_PROCESS
	
	attach_function :rb_gc_adjust_memory_usage, [:ssize_t], :void
end

def acquire(img)
	GC.rb_gc_adjust_memory_usage(30*1024*1024)
	ObjectSpace.define_finalizer(img, self.method(:release))
end

def release
	GC.rb_gc_adjust_memory_usage(-30*1024*1024)
end

What we need is a way to tell the GC the correct buffer size/memory usage.

ioquatix on Mar 24, 2020