celluloid: Celluloid::Future retaining arguments causing them to not get GC'd

I’m not sure if this is a bug with Ruby (I’m on ruby 2.1.2p95 but the same behavior exists on 2.0.0.-p247) or Celluloid, but due to a memory leak with Futures we can’t use Celluloid at all for our use case because memory can grow unbounded quickly and uses up all the RAM on the machine. I’ve tested 0.15.2 and 0.16.0 and it appears on both. Here’s a very simple repro:

# Setup
require 'celluloid'

def Process.rss ; `ps -o rss= -p #{Process.pid}`.chomp.to_i ; end
def Process.pretty_rss ; Process.rss.to_s.gsub(/(\d)(?=(\d{3})+(\..*)?$)/,'\1,') + " kilobytes" ; end

class Foo
  include Celluloid
  def do_something(arg1)
    return true
  end
end

Storing futures in an array, then iterating back over them causes unbounded memory growth:

puts(Process.pretty_rss)
20.times do
  f = Foo.new
  futures = []
  100.times { futures << f.future.do_something("a" * 50_000) }
  futures.length.times {|i| futures[i].value }
  f.terminate()
  futures = []
  puts(Process.pretty_rss)
end

# => 28,384 kilobytes
# => 33,824 kilobytes
# => 39,624 kilobytes
# => 43,396 kilobytes
# => 48,128 kilobytes
# => 53,320 kilobytes
# => 55,900 kilobytes
# => 56,724 kilobytes
# => 59,920 kilobytes
# => 60,844 kilobytes
# => 60,156 kilobytes
# => 64,408 kilobytes
# => 67,520 kilobytes
# => 66,960 kilobytes
# => 70,604 kilobytes
# => 74,948 kilobytes
# => 76,040 kilobytes
# => 75,472 kilobytes
# => 76,280 kilobytes
# => 75,528 kilobytes
# => 79,584 kilobytes

If you call .value on the future when you make it, growth is contained:

puts(Process.pretty_rss)
20.times do
  f = Foo.new
  futures = []
  100.times { futures << f.future.do_something("a" * 50_000).value }
  f.terminate()
  futures = []
  puts(Process.pretty_rss)
end

# => 80,384 kilobytes
# => 79,728 kilobytes
# => 80,536 kilobytes
# => 79,912 kilobytes
# => 80,720 kilobytes
# => 80,124 kilobytes
# => 80,932 kilobytes
# => 80,308 kilobytes
# => 79,756 kilobytes
# => 80,564 kilobytes
# => 79,940 kilobytes
# => 80,748 kilobytes
# => 80,124 kilobytes
# => 80,932 kilobytes
# => 80,308 kilobytes
# => 79,756 kilobytes
# => 80,492 kilobytes
# => 80,036 kilobytes
# => 80,844 kilobytes
# => 80,220 kilobytes

Edited by @niamster: test case with a proof https://gist.github.com/niamster/7bbc8a0fcde30f122327

About this issue

  • Original URL
  • State: closed
  • Created 10 years ago
  • Comments: 55 (48 by maintainers)

Commits related to this issue

Most upvoted comments

I believe that I am still seeing this issue with 0.17.3 and MRI. Thought I was going crazy until I found this thread! In our use case, we are instantiating many Actors over time (and terminating) so that we have a constant “alive” number of actors.

I’ve implemented the suggestion by @niamster to dereference instance variables in a finalizer, but this is just a partial workaround. Is there any way to explicitly mark the actor or “celluloidized object” for garbage collection?

Our alternative is to used a fixed number of Actor objects and essentially rerun initialization-like behaviors as opposed to killing one off and creating a new one. This is OK, but requires some messier code, and use of a Mutex and ConditionVariable for signalling (so we don’t reinitialize while .async operations are occurring).

it shows different result if I do Foo.new out of the loop.

def objcount(klass)
  c = 0
  ObjectSpace.each_object(klass) {
    c += 1
  }
  puts "#{klass}: " + c.to_s
end

f = Foo.new
200.times do
  r = f.future.do_something(1)
  r.value
end
ObjectSpace.garbage_collect
objcount(Celluloid::Future)
Celluloid::Future: 1
D, [2015-01-07T22:47:29.229500 #5813] DEBUG -- : Terminating 1 actor...

The below is similar to the original.

200.times do
  f = Foo.new
  r = f.future.do_something(1)
  r.value
end
ObjectSpace.garbage_collect
objcount(Celluloid::Future)
Celluloid::Future: 200
D, [2015-01-07T22:45:19.248581 #5596] DEBUG -- : Terminating 200 actors...