puma: Bundler error prevents phased-restarting puma after upgrading to rails 7

Describe the bug

We saw two different errors

158586] ! Unable to start worker
[4158586] /usr/lib/ruby/gems/3.0.0/gems/bundler-2.3.9/lib/bundler/definition.rb:481:in `materialize' [4158594] + Gemfile in context: /var/local/kenhub/kenhub.d/878/Gemfile

which should have raised this kind of error raise GemNotFound, "Could not find #{missing_specs.map(&:full_name).join(", ")}" in any of the sources"

and

[16442] ! Unable to start worker
[16442] /usr/lib/ruby/gems/3.0.0/gems/bundler-2.3.9/lib/bundler/runtime.rb:309:in `check_for_activated_spec!'

which should have raised this kind of error Gem::LoadError.new "You have already activated #{activated_spec.name} #{activated_spec.version}, but your Gemfile requires #{spec.name} #{spec.version}. #{suggestion}"

Why didn’t we see these error messages in the puma logs and how can we fix them?

Puma config:

#!/usr/bin/env puma

# The directory to operate out of.
#
# The default is the current directory.
#
directory "/var/local/kenhub/kenhub"

# Set the environment in which the rack's app will run. The value must be a string.
#
# The default is "development".
#
environment ENV["RAILS_ENV"] || "development"

# Store the pid of the server in the file at "path".
#
pidfile "/var/run/puma/puma.pid"

# Use "path" as the file to store the server info state. This is
# used by "pumactl" to query and control the server.
#
state_path "/var/run/puma/puma.state"

# Redirect STDOUT and STDERR to files specified. The 3rd parameter
# ("append") specifies whether the output is appended, the default is
# "false".
#
stdout_redirect "/var/log/kenhub/puma.log", "/var/log/kenhub/puma.log", true

# Configure "min" to be the minimum number of threads to use to answer
# requests and "max" the maximum.
#
# The default is "0, 16".
#
# threads 0, 16
threads 8, 8


# Bind the server to "url". "tcp://", "unix://" and "ssl://" are the only
# accepted protocols.
#
# The default is "tcp://0.0.0.0:9292".
#
bind 'unix:///var/run/puma/puma.sock'


# === Cluster mode ===

# How many worker processes to run.  Typically this is set to
# to the number of available cores.
#
# The default is "0".
#
workers ENV["PUMA_WORKERS"] || 16

# Allow workers to reload bundler context when master process is issued
# a USR1 signal. This allows proper reloading of gems while the master
# is preserved across a phased-restart. (incompatible with preload_app)
# (off by default)

# see https://github.com/puma/puma/blob/master/docs/deployment.md#restarting
prune_bundler                  

To Reproduce

  • update rails code to rail 7
  • run /usr/bin/bundle exec pumactl -F config/puma.rb phased-restart

Expected behavior

phased-restart should restart all puma instances correctly

Desktop (please complete the following information):

  • OS: Ubuntu 20.04 with systemd
  • Puma Version 5.6.2

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 33 (32 by maintainers)

Commits related to this issue

Most upvoted comments

I vote for keeping it simple

@nateberkopec

re the ‘loaded extensions’, I’m in favor of adding that logging behind some kind of “verbose mode” switch

Ok. I’ll look at the current switches, may add one.

For everyone, see PR #2903. In Ruby master, the io/wait methods being used in Puma have been moved into the native Ruby IO class. So, many years from now, when Ruby 3.2 is the Puma minimum required Ruby version…

there’s no explicit dependency in puma, just relying on stdlib, right?

Puma gemspec:

s.required_ruby_version = Gem::Requirement.new(">= 2.2")

There is no io-wait gem available for Ruby 2.2, as first release (0.1.0) was ‘>= 2.3.0’. So, we can’t add an explicit dependency. Maybe next major release…

Did you get a chance to try out the repro?

No yet, but soon.

@nateberkopec we found a workaround, and also managed to reproduce the issue hopefully.

The workaround for us was to install io-wait:0.2.1 globally using gem install io-wait:0.2.1. We’re still not sure why the global gem interferes with the local one, and the local one gets installed and exists in the folders.

The repro can be found at https://github.com/gingerlime/puma-phased-restart-bundler-error