puma: Performance degradation with multithreading and connection pools

I maintain several of the Ruby framework samples for TechEmpower’s FrameworkBenchmarks. The latest Round 14 Preview 1.1 numbers reflect very favorably on all the perf work that’s been going on in the Ruby community over the past few months and years. However, there is an anomaly in Puma’s numbers that may be worth digging into!

Context

Round 14 Preview 1.1 was executed on a machine with 80 hardware threads (and some ungodly amount of RAM).

• Puma, Unicorn, and Passenger were configured to run with 100 processes.
• Puma was additionally configured to run with four threads (min and max) per process.
• Under Puma, Sequel and ActiveRecord were configured to use connection pools with four slots.
• Under Unicorn and Passenger, Sequel and ActiveRecord were configured to run single-threaded.
• The wrk benchmark was executed against these instances with a concurrency of 256.

I’ll be talking about YARV/MRI 2.4 throughout the rest of this report unless otherwise specified. (There is a JRuby/TorqueBox benchmark but it has some other undiagnosed performance issues.)

The Good

On most of the tests, Puma wins handily thanks to its increased concurrency, even on CPU-bound tasks. For example, JSON serialization under Sinatra:

And single SELECT queries against PostgreSQL under Roda/Sequel (note the FE column):

We can see in that last example that Puma pulls ahead despite the overhead of Sequel automatically checking connections in to and out from its pool. The route being called looks like this (Roda automatically serializes the resulting Hash object to JSON):

# Test type 2: Single database query
static_get '/db' do
  World.with_pk(rand(10_000).succ).values
end

The Bad

However, the wheels start to come off as soon as we start executing more than one query in a route. For example, multiple SELECT queries against PostgreSQL under Roda/Sequel:

Oops! That’s not good at all. Here’s roughly what that code looks like:

# Test type 3: Multiple database queries
static_get '/queries' do
  Array.new(20) do
    World.with_pk(rand(10_000).succ).values
  end
end

Note: I am able to reproduce this “anomaly” locally, and wrapping a Sequel::Database#synchronize around the loop (to ensure a single database connection is checked out from the pool and used for all 20 queries) improves the performance of this route by 10%, still 30% behind Unicorn and Passenger on my machine.

This performance disparity can’t be explained by the overhead of Sequel’s connection pool as shown in the prior example of single-query performance. We can see it in ActiveRecord, too; multiple SELECT and UPDATE queries performed against MySQL under Sinatra/ActiveRecord:

Here’s roughly the code for that route:

# Test type 5: Database updates
get '/updates' do
  worlds =
    ActiveRecord::Base.connection_pool.with_connection do
      Array.new(20) do
        world = World.find(rand(10_000).succ)
        world.update(:randomnumber=>rand(10_000).succ)
        world.attributes
      end
    end

  json worlds
end

The Ugly

It seems like the issue manifests from a “tight” loop of queries occurring in a single request thread with a connection checked out from a pool. When the queries are distributed across multiple request threads, Puma plays nice with a connection pool. When Sequel or ActiveRecord are single-threaded, there are no performance issues whatsoever. That’s true even for Puma: single-threaded Puma beats Unicorn and Passenger in these benchmarks (on my machine)!

This is particularly pernicious because it’s exactly the use case where you would expect Puma’s multithreading to help you, not hurt you: lots of IO-bound tasks. In the real world, we could easily solve this problem by rewriting these routes to perform a single SELECT .. WHERE .. IN query. However, I think these benchmarks raise an interesting question: Is this a pathological case that can’t be fixed without changing the application, or does it demonstrate an edge case that could be quietly afflicting thousands of Ruby applications running under Puma?

Updates

1. (25-Mar) Interestingly, a SELECT .. WHERE .. IN is also slower (fewer requests/second) on Puma—multi-threaded (1,443) cf. single-threaded (2,200). So the issue doesn’t appear to be strictly related to a “tight” loop of IO-bound tasks.

2. (25-Mar) I was able to reproduce my findings under Ruby 2.2.6 and Puma 2.15.3—multi-threaded (256) cf. single-threaded (387)—so the performance anomaly (if there truly is one) has been around for some time.

3. (25-Mar) Using a connection pool with fewer slots than Puma’s thread depth seems to give decent compromise numbers. I can get within 20% of single-threaded performance using a connection pool size of floor(2*ln(x)), but that’s just a wild guess at a formula based on what I’m seeing at a variety of Puma thread depths.