puma: BUNDLE_GEMFILE is incorrectly set to parent Gemfile in worker processes (when not setting BUNDLE_GEMFILE explicitly)
Describe the bug
Before #1893, if you used prune_bundler
and started bundle exec puma
without explicitly setting BUNDLE_GEMFILE
, workers would not have a value set for the environment variable BUNDLE_GEMFILE
. This is desirable when you want workers to use the Gemfile located in the application directory, especially if that directory is different from the directory that you started the puma master process in (such as with a phased restart).
After #1893, workers have BUNDLE_GEMFILE
unexpectedly set to the path of the Gemfile used by the master process. I think this forces bundler
in the workers to use that Gemfile instead of the one in the worker’s working directory as it would if the environment variable was unset.
I think the intent of that PR was that if you start your server with BUNDLE_GEMFILE=Gemfile.rails6 bundle exec puma
, that workers would have BUNDLE_GEMFILE
set to the same value. But it ignores the fact that Bundler
itself sets a value for ENV['BUNDLE_GEMFILE']
even if it wasn’t specified at startup time. Workers now end up inheriting this value, when they didn’t before.
One consequence of this bug is that it is no longer possible to delete from your server the Gemfile that was used when you booted the puma master process. If you do, workers fail with the next phased restart. I imagine there are other bugs related to being able to change gems in the Gemfile for new versions of your application, but I haven’t been able to reproduce those.
Puma config:
port 3000
workers 1
prune_bundler
directory '/usr/src/releases/current'
on_worker_boot do
pp "BUNDLE_GEMFILE: #{ENV['BUNDLE_GEMFILE']}"
end
To Reproduce
I wrote a script that reproduces the error here: https://github.com/cjlarose/puma-phased-restart-could-not-find-gem-errors/tree/regression-1893 (I reused a repo from an unrelated issue)
Before #1893 , workers boot and print BUNDLE_GEMFILE:
. After that PR, workers boot and print BUNDLE_GEMFILE: /usr/src/releases/1582850222334837300/Gemfile
, referring to the path of the Gemfile that was used by the puma master process.
Expected behavior
If BUNDLE_GEMFILE
was unspecified when you started the master process, it should remain unspecified in the worker processes.
Desktop (please complete the following information):
- OS: Linux
- Puma Version 8c9b3ebc7a86bdc5488704a0225b76409b9f555f
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 17 (17 by maintainers)
Commits related to this issue
- Replace deprecated `Bundler.with_clean_env` with `with_unbundled_env` Also add tests for `GEM_HOME` environment variable preserving. Also drop `BUNDLE_GEMFILE` preserving, resolve #2133 — committed to AlexWayfer/puma by AlexWayfer 4 years ago
- Replace deprecated `Bundler.with_clean_env` with `with_unbundled_env` Also add tests for `GEM_HOME` environment variable preserving. Also drop `BUNDLE_GEMFILE` preserving, resolve #2133 Also disabl... — committed to AlexWayfer/puma by AlexWayfer 4 years ago
- Replace deprecated `Bundler.with_clean_env` with `with_unbundled_env` Also add tests for `GEM_HOME` environment variable preserving. Also drop `BUNDLE_GEMFILE` preserving, resolve #2133 Also disabl... — committed to AlexWayfer/puma by AlexWayfer 4 years ago
- Replace deprecated `Bundler.with_clean_env` with `with_unbundled_env` Also add tests for `GEM_HOME` environment variable preserving. Also drop `BUNDLE_GEMFILE` preserving, resolve #2133 Also disabl... — committed to AlexWayfer/puma by AlexWayfer 4 years ago
- Replace deprecated `Bundler.with_clean_env` with `with_unbundled_env` Also add tests for `GEM_HOME` environment variable preserving. Also drop `BUNDLE_GEMFILE` preserving, resolve #2133 Also disabl... — committed to AlexWayfer/puma by AlexWayfer 4 years ago
@AlexWayfer That summary sounds correct. That’s how I typically use puma when I want to update my application in a environment that uses mutable infrastructure (reusing existing live servers).
This doesn’t work. I think what happens is workers will try to activate the new version of puma, but since an old version is already activated, RubyGems throws an exception. Workers continue to die and spawn and die and spawn. This is actually true of any change in
Gemfile
v2 that changes the version ofpuma
or any gem that is a runtime dependency of your puma master process (nio4r
especially), but also anything else you choose to run in your puma master process (such aspuma_worker_killer
). This is a separate problem documented in #2018 and again in #1875.Yes, typically. This is something devs might have to communicate to operators, unfortunately. If there’s a puma upgrade, we have to perform normal or “hot” restarts, not “phased” restarts.
Hello all!
I glad to work on #2120 and resolve this… stuff with Bundler.
Please, describe for me, what to do and what you’re expecting.
As I understood, you want to:
aaa
withGemfile
v1puma
with workers andprune_bundler
(withbundle exec
or not? 🤔 without it’d be hard to resolve some git dependencies)bbb
withGemfile
v2puma
with updated bundle for workers (master
process is untouched)Am I right?
The main question for me: is it OK? What if
Gemfile
v2 haspuma
update? Andmaster
process should be restarted? Is it something for manual resolution (“if we’re updating bundle withpuma
let’s restartpuma
completely”)?Haha, I don’t have a better suggestion as you two have both dug into this issue far more than I have. I agree that this is not a great solution.
I’d also point out that #2120 is related here and will have to get merged eventually when working.
Oh yeah, that’s my bad. The issue is in
master
, but not yet in a released version of puma.