puma: Puma errors and performs badly when receiving a non-HTTPS request while TLSv1.3-configured

The bug: Timeout / Puma throwing an error when querying Rails through a reverse proxy.

This is similar to https://github.com/puma/puma/issues/1708 but I thought it was worth summarizing/detailing again, in a separate issue (also because the context/usage is different).

~$ bundle exec rails s -b 'ssl://localhost:3000?key=$SSL_KEY&cert=$SSL_CERT'
=> Booting Puma
=> Rails 6.0.2.1 application starting in development
=> Run `rails server --help` for more startup options
W, [2020-02-13T21:27:50.354171 #6604]  WARN -- : Meta::Organization: overriding method 'invalid?'!
Puma starting in single mode...
* Version 4.3.1 (ruby 2.6.5-p114), codename: Mysterious Traveller
* Min threads: 0, max threads: 16
* Environment: development
* Listening on ssl://localhost:3000?key=/Users/[REDACTED]/app/private/ssl-dev/dev-key.key&cert=/Users/[REDACTED]/app/private/ssl-dev/dev-certificate.crt
Use Ctrl-C to stop
Error in reactor loop escaped: System error: Undefined error: 0 - 0 (Puma::MiniSSL::SSLError)
/Users/[REDACTED]/.rvm/gems/ruby-2.6.5/gems/puma-4.3.1/lib/puma/minissl.rb:43:in `read'
/Users/[REDACTED]/.rvm/gems/ruby-2.6.5/gems/puma-4.3.1/lib/puma/minissl.rb:43:in `engine_read_all'
/Users/[REDACTED]/.rvm/gems/ruby-2.6.5/gems/puma-4.3.1/lib/puma/minissl.rb:54:in `read_nonblock'
/Users/[REDACTED]/.rvm/gems/ruby-2.6.5/gems/puma-4.3.1/lib/puma/minissl.rb:128:in `read_and_drop'
/Users/[REDACTED]/.rvm/gems/ruby-2.6.5/gems/puma-4.3.1/lib/puma/minissl.rb:145:in `close'
/Users/[REDACTED]/.rvm/gems/ruby-2.6.5/gems/puma-4.3.1/lib/puma/client.rb:145:in `close'
/Users/[REDACTED]/.rvm/gems/ruby-2.6.5/gems/puma-4.3.1/lib/puma/reactor.rb:266:in `rescue in block in run_internal'
/Users/[REDACTED]/.rvm/gems/ruby-2.6.5/gems/puma-4.3.1/lib/puma/reactor.rb:218:in `block in run_internal'
/Users/[REDACTED]/.rvm/gems/ruby-2.6.5/gems/puma-4.3.1/lib/puma/reactor.rb:157:in `each'
/Users/[REDACTED]/.rvm/gems/ruby-2.6.5/gems/puma-4.3.1/lib/puma/reactor.rb:157:in `run_internal'
/Users/[REDACTED]/.rvm/gems/ruby-2.6.5/gems/puma-4.3.1/lib/puma/reactor.rb:313:in `block in run_in_thread'

SSH Outputs (reverse proxy forwarding):

debug1: client_input_channel_open: ctype forwarded-tcpip rchan 2 win 2097152 max 32768
debug1: client_request_forwarded_tcpip: listen  port 80, originator [REDACTED_IP] port 53504
debug1: connect_next: host localhost ([::1]:3000) in progress, fd=7
debug1: channel 0: new [REDACTED_IP]
debug1: confirm forwarded-tcpip
debug1: channel 0: connected to localhost port 3000

The browser ends-up with a gateway time-out response from Nginx (reverse-proxy server):

504 Gateway Time-out
nginx/1.17.6

The error above is only thrown on timeout, or on canceling the request.

Puma config:

$ rails runner "require 'puma/minissl'; require 'puma/puma_http11'; Puma::Server.class; Puma::MiniSSL.check; puts Puma::MiniSSL::OPENSSL_VERSION"
> OpenSSL 1.1.1d  10 Sep 2019

$ rails runner "require 'puma/minissl'; require 'puma/puma_http11'; Puma::Server.class; Puma::MiniSSL.check; puts Puma::MiniSSL::OPENSSL_LIBRARY_VERSION"
> OpenSSL 1.1.1d  10 Sep 2019

$ which openssl
> /usr/local/opt/openssl@1.1/bin/openssl

$ openssl version -a
> OpenSSL 1.1.1d  10 Sep 2019
built on: Sat Sep 28 13:18:07 2019 UTC
platform: darwin64-x86_64-cc
options:  bn(64,64) rc4(16x,int) des(int) idea(int) blowfish(ptr)
compiler: clang -fPIC -arch x86_64 -O3 -Wall -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -D_REENTRANT -DNDEBUG
OPENSSLDIR: "/usr/local/etc/openssl@1.1"
ENGINESDIR: "/usr/local/Cellar/openssl@1.1/1.1.1d/lib/engines-1.1"
Seeding source: os-specific

To Reproduce

1. Clean install the latest openssl (also tried with libressl) on OSX:

brew install openssl@1.1
ln -s /usr/local/opt/openssl@1.1/lib/libssl.1.1.dylib /usr/local/lib/libssl.1.1.dylib
ln -s /usr/local/opt/openssl@1.1/lib/libssl.1.1.dylib /usr/local/lib/libssl.dylib
ln -s /usr/local/opt/openssl@1.1/lib/libcrypto.1.1.dylib /usr/local/lib/libcrypto.1.1.dylib
ln -s /usr/local/opt/openssl@1.1/lib/libcrypto.1.1.dylib /usr/local/lib/libcrypto.dylib

1. Clean install latest ruby version, specifying openssl dir

rvm --autolibs=disable install 2.6.5 --with-openssl-dir=/usr/local/etc/openssl@1.1/

1. Setting up reverse proxy server, using:

https://github.com/robzhu/nginx-local-tunnel
https://github.com/jwilder/nginx-proxy
https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion

It works fine. Of course, both 80 and 443 are open and accessible. LetsEncrypt certificate is valid.

1. Gemfile as follow:

ruby '2.6.5'
gem 'puma', '4.3.1'
gem 'pg', '1.2.2'
gem 'rails', '~> 6.0.2.1'
gem 'openssl', '~> 2.1.0'

1. Generate Self-Signed Certificates

openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
  -keyout dev-key.key -out dev-certificate.crt -subj /CN=example.com

1. Run Rails

export SSL_KEY="/[REDACTED]/railsapp/private/ssl-dev/dev-key.key"
export SSL_CERT="/[REDACTED]/railsapp/private/ssl-dev/dev-certificate.crt"
~$ bundle exec rails s -b 'ssl://localhost:3000?key=$SSL_KEY&cert=$SSL_CERT'

1. Run proxy server

ssh -NR :80:localhost:3000 -p 2222 user@my.proxy.server.tld -v

Expected behavior

• Proxy server works fine (tried with a node.js app).
• Querying rails using https locally works fine. I expect to be able to query the rails app using https, through the proxy server.

I tried many different configurations, clean install, ways of installing OpenSSL/LibreSSL, using ngrok, Serveo, setting up owner reverse proxy (current setup described), different ways of generating the SSL certificates. No chance so far.

I also tried forking and updating Puma/reactor.rb accordingly to the suggestion of @MSP-Greg in https://github.com/puma/puma/issues/1708 (https://github.com/puma/puma/issues/1708#issuecomment-552151295). In that case it doesn’t throw anything right away, but ends up timing-out and crashing. I am not comfortable playing with OpenSSL & co, so I am still not sure whether this is an issue from Puma, or if something could be fixed in my setup.

Desktop:

• OSX Catalina 10.15.3
• Chrome Version 80.0.3987.100 (Official Build) (64-bit)

Question: Referring to https://github.com/puma/puma/issues/1708#issuecomment-523434704 , would attempting to disable/prevent Tls 1.3 on Nginx be of any help?