puma: Puma 4.3.6 - Rails 6.0.3.4 - TLS brings in memory leak

Describe the bug Creating a “hello world” rack application and configuring puma to run using TLS, excessive memory is used by the application.

Puma config:

threads 1, 1
workers 1
ssl_bind '0.0.0.0', 3000, {
    key: "certificates/certificate.key",
    cert: "certificates/certificate.crt",
    verify_mode: 'none',
    no_tlsv1: true,
    no_tlsv1_1: true
  }

bind 'tcp://0.0.0.0:3500'

# Specifies the `pidfile` that Puma will use.
pidfile ENV.fetch("PIDFILE") { "server.pid" }

CLI : docker run --rm -p 3000:3000 -p 3500:3500 puma_tls_memory_leak

To Reproduce

I’ve created a repo and a dockerfile to be able to reproduce, using your hello world rack : https://github.com/Melchizedech/rails_puma_tls_memory_leak

Running while :; do curl -k --location 'https://localhost:3000' -v ; done will increase the RAM used by the application from 25MB (RSS) to 60MB (RSS). Then every minute or so, another extra MB (RSS) will be consumed.

If we instead run while :; do curl --location 'http://localhost:3000/' -v ; done, the RAM used by the app increase from 25 to 32MB (RSS) and eventually, the app will start consuming extra MB of RAM.

I definitely think there is an issue on Puma with TLS.

Here is the graph of one application that we use, it is a small Rails app that receives close to no user traffic. The “real” traffic it receives comes from the liveness probe.

Cluster A : App in a production cluster

Cluster B : App in another production cluster

Here are some screenshots from the ram consumption of a brand new rails app only running the API mode, while we hit a non-existing route, behind a similar configuration.

Just after boot : App after having booted in production mode

After 6 min of constant cURL App after 6min of stress test on a non-existing route using HTTPS

After 28min of constant cURL App after 28min of stress test on a non-existing route using HTTPS While I was testing the brand new rails API to reproduce the issue (before swwitching for a Rack hello world app), I tried to use jemalloc, I’ve seen a strong improvement, though, it was still getting more RAM used (but a much, much, slower rate).

Expected behavior I would expect the RAM consumption to not be increasing forever when I use TLS on Puma.

Desktop (please complete the following information):

OS: ArchLinux, (and also Debian via Dockerization)
Puma Version : 4.3.6 (but could also reproduce on 5.0.2)

Any help to debug/investigate that is welcomed.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 17 (7 by maintainers)

Most upvoted comments

Hi guys, just to chime in here. The issue I was having with leaking memory was not related to what is described in this issue.

what was it related to?

It was actually related to a memory leak in this gem: https://github.com/typhoeus/ethon

l33z3r on Feb 16, 2022

I can try to put that app without TLS in place on one of our staging environment and give you the result in several days.

Using such a solution won’t work. (TL;DR: this is an app that requires encrypted traffic in our internal network. traffic between [outside, internal] <=> nginx will be encrypted but not nginx <=> puma). We will most likely use service mesh or a similar technology to remove the need for TLS within puma.

Here is three screenshots for puma 4.3.6 with HTTP At boot time puma_http_beg

After 7min puma_http_7min

After 27min puma_http_27min

here are 3 screenshots of puma 5.0.2 with HTTPS

At boot time puma5_beg

After 6 minutes puma5_6min

After 33 minutes puma5_33min

These screenshots are using the same hello world rack app. only difference was a change in the gemfile

diff --git a/Gemfile b/Gemfile
index 8418cc4..781766f 100644
--- a/Gemfile
+++ b/Gemfile
@@ -3,4 +3,4 @@ git_source(:github) { |repo| "https://github.com/#{repo}.git" }
 
 ruby '2.7.2'
 
-gem 'puma', '~> 4.1'
+gem 'puma', '~> 5.0'
diff --git a/Gemfile.lock b/Gemfile.lock
index 50eda8b..b55ce66 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -2,14 +2,14 @@ GEM
   remote: https://rubygems.org/
   specs:
     nio4r (2.5.4)
-    puma (4.3.6)
+    puma (5.0.2)
       nio4r (~> 2.0)
 
 PLATFORMS
   ruby
 
 DEPENDENCIES
-  puma (~> 4.1)
+  puma (~> 5.0)
 
 RUBY VERSION
    ruby 2.7.2p137

Melchizedech on Oct 13, 2020

Hi guys, just to chime in here. The issue I was having with leaking memory was not related to what is described in this issue.

l33z3r on Nov 1, 2021

The app is receveing traffic from the liveness probe.

The real user traffic has not been simulated. Though I can give it another go if needed.

On October 15, 2020 1:23:44 PM UTC, Nate Berkopec notifications@github.com wrote:

@Melchizedech Was that app receiving any traffic? I assume your memory graphs from the TLS-on app was constantly receiving traffic.

– You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/puma/puma/issues/2425#issuecomment-709321599

– Sent from my Android device with K-9 Mail. Please excuse my brevity.

Melchizedech on Oct 15, 2020

Okay, I’ll deploy this hello world rack application with TLS, and I’ll let it live for a day or two. I’ll come back with the monitoring results.

Melchizedech on Oct 14, 2020