puma: BUNDLE_GEMFILE is incorrectly set to parent Gemfile in worker processes (when not setting BUNDLE_GEMFILE explicitly)

Describe the bug

Before #1893, if you used prune_bundler and started bundle exec puma without explicitly setting BUNDLE_GEMFILE, workers would not have a value set for the environment variable BUNDLE_GEMFILE. This is desirable when you want workers to use the Gemfile located in the application directory, especially if that directory is different from the directory that you started the puma master process in (such as with a phased restart).

After #1893, workers have BUNDLE_GEMFILE unexpectedly set to the path of the Gemfile used by the master process. I think this forces bundler in the workers to use that Gemfile instead of the one in the worker’s working directory as it would if the environment variable was unset.

I think the intent of that PR was that if you start your server with BUNDLE_GEMFILE=Gemfile.rails6 bundle exec puma, that workers would have BUNDLE_GEMFILE set to the same value. But it ignores the fact that Bundler itself sets a value for ENV['BUNDLE_GEMFILE'] even if it wasn’t specified at startup time. Workers now end up inheriting this value, when they didn’t before.

One consequence of this bug is that it is no longer possible to delete from your server the Gemfile that was used when you booted the puma master process. If you do, workers fail with the next phased restart. I imagine there are other bugs related to being able to change gems in the Gemfile for new versions of your application, but I haven’t been able to reproduce those.

Puma config:

port 3000
workers 1
prune_bundler
directory '/usr/src/releases/current'

on_worker_boot do
  pp "BUNDLE_GEMFILE: #{ENV['BUNDLE_GEMFILE']}"
end

To Reproduce

I wrote a script that reproduces the error here: https://github.com/cjlarose/puma-phased-restart-could-not-find-gem-errors/tree/regression-1893 (I reused a repo from an unrelated issue)

Before #1893 , workers boot and print BUNDLE_GEMFILE: . After that PR, workers boot and print BUNDLE_GEMFILE: /usr/src/releases/1582850222334837300/Gemfile, referring to the path of the Gemfile that was used by the puma master process.

Expected behavior

If BUNDLE_GEMFILE was unspecified when you started the master process, it should remain unspecified in the worker processes.

Desktop (please complete the following information):

  • OS: Linux
  • Puma Version 8c9b3ebc7a86bdc5488704a0225b76409b9f555f

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 17 (17 by maintainers)

Commits related to this issue

Most upvoted comments

@AlexWayfer That summary sounds correct. That’s how I typically use puma when I want to update my application in a environment that uses mutable infrastructure (reusing existing live servers).

What if Gemfile v2 has puma update?

This doesn’t work. I think what happens is workers will try to activate the new version of puma, but since an old version is already activated, RubyGems throws an exception. Workers continue to die and spawn and die and spawn. This is actually true of any change in Gemfile v2 that changes the version of puma or any gem that is a runtime dependency of your puma master process (nio4r especially), but also anything else you choose to run in your puma master process (such as puma_worker_killer). This is a separate problem documented in #2018 and again in #1875.

Is it something for manual resolution?

Yes, typically. This is something devs might have to communicate to operators, unfortunately. If there’s a puma upgrade, we have to perform normal or “hot” restarts, not “phased” restarts.

Hello all!

I glad to work on #2120 and resolve this… stuff with Bundler.

Please, describe for me, what to do and what you’re expecting.

As I understood, you want to:

  1. Have a commit aaa with Gemfile v1
  2. Start a puma with workers and prune_bundler (with bundle exec or not? 🤔 without it’d be hard to resolve some git dependencies)
  3. Pull a commit bbb with Gemfile v2
  4. Phased restart puma with updated bundle for workers (master process is untouched)

Am I right?

The main question for me: is it OK? What if Gemfile v2 has puma update? And master process should be restarted? Is it something for manual resolution (“if we’re updating bundle with puma let’s restart puma completely”)?

BUNDLER_ORIG_ORIG_BUNDLE_GEMFILE

Haha, I don’t have a better suggestion as you two have both dug into this issue far more than I have. I agree that this is not a great solution.

I’d also point out that #2120 is related here and will have to get merged eventually when working.

Oh yeah, that’s my bad. The issue is in master, but not yet in a released version of puma.