chruby: A way to not set GEM_HOME if the installed ruby is under $HOME

As a compromise and to get the changes released faster, I am thinking of adding a separate dev.sh file which can be loaded along with chruby.sh and that changes how GEM_HOME/GEM_PATH are set, in order to support testing rubies of the same version but with different configurations.

As per @dentarg’s use case of wanting to switch between aarm64 and x86-64 on Apple Silicon, I could also add a multi_arch.sh file which takes into account the current architecture.

Testing the same ruby version with different configurations or switching between aarch64 and x86-64 on specific hardware are niche use-cases vs. your more common Ruby app development. Providing opt-in solutions seems like a good compromise. Depending on whether these opt-in solutions become popular, they might become the default behavior in 1.0.0.

@eregon like we have discussed on multiple occasions, changing the path of the GEM_HOME is a breaking change and would cause chruby users to suddenly lose all of their installed gems after upgrading. Since chruby is widely used this would cause a great deal of confusion and frustration, so I decided to push that change back to 1.0.0 where we could safely break with backwards compatibility. Likewise, rubygems also has to maintain backwards compatibility and not change default behavior too much, otherwise that could possibly cause issues with downstream users and Linux distributions which expect gems to be installed into specific directories.

I would have accepted #451, however I am very hesitant about adding additional bifurcating logic. I could envision scenarios where you are debugging an issue for a use and you need to determine if the gems are being installed into the correct location, so you then have to determine if the ruby’s gem directory is writable or not in order to determine if chruby is going to use the ruby’s gem dir or ~/.gem/.... This would likely result in more debugging, troubleshooting, and complexity.

Since chruby is loaded into user’s shells, and because it has to run under Bash 3+ and Zsh, I have to be very cautious about changing any functionality and debate every change, especially those that might break or change default behavior for users. chruby also has an explicit policy of not accepting workaround fixes for upstream issues, which means I scrutinize issues and whether they could be better solved by rubygems or upstream Ruby. I also have to balance the needs of regular users vs the more exotic edge-cases which Ruby maintainers discover while testing Rubies. chruby is not an easy project to contribute to and requires a great deal of patience and compromising.

I am now considering a different approach to handling GEM_HOME. Extracting the logic which sets GEM_HOME and GEM_PATH out as an additional function hook that could be overrode by additional opt-in configuration files much like how auto.sh is implemented. Additionally, we could extract that logic entirely into an additional gem_home.sh file and make isolating gems in ~/.gem/rubies/$ruby/ opt-in. This would provide an absolute bare minimum user experience of just switching the rubies, but not setting GEM_HOME by default. This would work seamlessly for users who have all rubies installed into ~/.rubies; which means their ruby gem directories are writable by default. If the GEM_HOME code is extracted entirely into an opt-in gems.sh or gem_home.sh file, and users have rubies installed into /opt/rubies (such as myself) , those users would then need to decide if they want gem isolated in ~/.gem/rubies/$ruby/ or allow rubygems to pick the gem installation directory for the user; this could potentially mean rubygems install gems of different /opt/rubies rubies into ~/.gem, but maybe some users might want that? By moving the logic out into functions, this opens the door for customization and different opt-in configurations.

Example Code

# potentially defined in a `gems.sh` or `gem_home.sh` file
function chruby_gems_set()
{
		export GEM_HOME="$HOME/.gem/$RUBY_ENGINE/$RUBY_VERSION" # could be extracted into another function in case people want to configure the `GEM_HOME` template string
		export GEM_PATH="$GEM_HOME${GEM_ROOT:+:$GEM_ROOT}${GEM_PATH:+:$GEM_PATH}"
		export PATH="$GEM_HOME/bin:$PATH"
}

# potentially defined in a `gems.sh` or `gem_home.sh` file
function chruby_post_hook() { chruby_set_gem_home }

function chruby_set()
{
    # add the $ruby/bin directory to PATH and query the ruby's information
    # ...
    chruby_post_hook
}

Sorry for not working on the 1.0.0 branch. I have been extremely busy over the last four-six years, with both commercial work (2014-2020) and other Open Source work (2020-2023). I will start working on 1.0.0 again; I just added relisting of rubies directories.

Testing the same ruby version with different configurations or switching between aarch64 and x86-64 on specific hardware are niche use-cases vs. your more common Ruby app development. Providing opt-in solutions seems like a good compromise. Depending on whether these opt-in solutions become popular, they might become the default behavior in 1.0.0.

I have to disagree that multiple architechtures are niche use cases. there are searches for it all over the web since apple silicone and rosetta have become main staple development machines. it has become a necessity even for many

I have not yet heard of other people having this issue besides you or other truffleruby developers.

?? – I’m no truffleruby developer 😃 Just a developer doing things with Ruby.

I’ve been a devote and happy chruby user for 10 years (yes, since start almost), but sadly, I no longer recommend friends and co-workers to use the released version of chruby. I now recommend people using eregon’s branch. It would be great to change that.

I can verify that in my work on the YJIT team, and especially for speed.yjit.org, we have difficulties when shared dirs contain built native extensions. We handle this by deleting all gems in all shared directories on every build, as well as all built Ruby dirs we’ll be using for that build. In general, my experience is that Rubygems/Bundler/etc deal poorly with trying to match up specific built native extensions to specific Rubies. So for benchmarking we use a Big Hammer to handle the situation: we delete everything, so there is clearly nothing stale or inappropriately shared.

This problem isn’t unique to chruby, though we do have it when using chruby. We’re a hard case - we build a lot of prerelease Rubies that all have the same version string, so RUBY_VERSION checking does nothing for us. We’re prone to get crashes, slowdowns and other problems very regularly if we leave built gems sitting around. So we basically treat shared-across-multiple-Rubies gem dirs as a bug, and delete their contents when there’s reason to care.

To preempt the same question: we do indeed work around the problem, by deleting everything. We need a fresh gem install of all native extensions for every build regardless, so saving old copies wouldn’t do us any good. If we wanted shorter runs that didn’t take multiple hours, or to reuse the same prerelease Ruby for multiple runs, we’d work around it by fiddling with dirs like GEM_HOME every time we changed versions. But if we need to manually manage our gem-related env vars every time we switch Rubies, chruby becomes a much less valuable tool.

I do however plan on adding additional function hooks to allow configuring how chruby sets GEM_HOME or GEM_PATH. These changes would be incremental and could be released before 1.0.0.

That would be nice. The defaults needs to change though, because the current GEM_HOME/GEM_PATH set by chruby are incorrect for all dev Rubies, and all non-CRuby. In other words, it only works for CRuby releases, in the case they are always built with the same flags (e.g., --enable-shared) and arch for a given release version on the same machine. It also fails for CRuby releases on platforms with multiple archs as said in https://github.com/postmodern/chruby/issues/422#issuecomment-1232791999 and https://github.com/postmodern/ruby-install/issues/413#issuecomment-1244762132.

I have not yet heard of other people having this issue besides you or other truffleruby developers.

For instance:

• CRuby master contributors using chruby (and notably Shopify devs). I believe that’s one of the main reasons why https://github.com/ruby/ruby/pull/5474 was added. I know at least @byroot, @peterzhu2118, @noahgibbs, and the YJIT team were affected by this.
• Quoting https://bugs.ruby-lang.org/issues/18249#note-9:

  For dev builds, Ruby switchers has responsibility to handle the ABI compatibility breakage.

• It’s important for caching, e.g., https://bugs.ruby-lang.org/issues/18249#note-15. With this bug in chruby people disable gem caching, which is wasteful.
• https://github.com/postmodern/chruby/issues/422#issuecomment-1232791999
• https://twitter.com/schneems/status/1401248352562515968
• https://twitter.com/jardonamron/status/1401250028778229770
• https://twitter.com/tenderlove/status/1401253239064993792
• https://twitter.com/jardonamron/status/1401253961701609472
• A subset of the many many issues related to GEM_HOME on this repo: https://github.com/postmodern/chruby/issues?q=is%3Aissue+GEM_HOME I remember commenting on some of them.
• Several direct messages on Slack, unfortunately I cannot link that.

I am just curious, if this is such a critical issue, why can’t truffleruby just workaround the problem?

TruffleRuby already spent significant amount of effort to workaround this chruby bug. Notably there is an ABI check when loading C extensions, so at least it fails early.

What workaround are you thinking of? I don’t think truffleruby can workaround chruby setting the GEM_HOME incorrectly. Whoever sets the GEM_HOME is responsible to set it correctly (or not set it), so that’s chruby, isn’t it?

Due to this, I’ve been forced to switch to @eregon’s fork of chruby that have #431: https://github.com/eregon/chruby/tree/do-no-set-gem-home

It was needed because I have a computer with Apple silicon (M2), and I want to be able run Ruby apps either on arm64 or x86_64 (“Intel”).

Using ruby-build it was very easy to install Rubies for different architectures to different directories (and configure your shell for it), but I can’t have them install native extensions (like for Nokogiri) to the same directory (GEM_HOME)

arm64 $ env | grep GEM
GEM_ROOT=/Users/dentarg/.arm64_rubies/2.7.6/lib/ruby/gems/2.7.0
GEM_HOME=/Users/dentarg/.gem/ruby/2.7.6
GEM_PATH=/Users/dentarg/.gem/ruby/2.7.6:/Users/dentarg/.arm64_rubies/2.7.6/lib/ruby/gems/2.7.0

intel $ env | grep GEM
GEM_ROOT=/Users/dentarg/.rubies/2.7.6/lib/ruby/gems/2.7.0
GEM_HOME=/Users/dentarg/.gem/ruby/2.7.6
GEM_PATH=/Users/dentarg/.gem/ruby/2.7.6:/Users/dentarg/.rubies/2.7.6/lib/ruby/gems/2.7.0

As an example, before switching to the chruby fork (using chruby 0.3.9 installed from Homebrew), I did bundle install for my app on arm64 (ruby 2.7.6p219 (2022-04-12 revision c9c2245c0a) [arm64-darwin21]), then I also did bundle install for it on Intel (ruby 2.7.6p219 (2022-04-12 revision c9c2245c0a) [x86_64-darwin21]). When I tried to run my tests on arm64 I got this back:

arm64 $ b e rake
Traceback (most recent call last):
  13: from /Users/dentarg/.gem/ruby/2.7.6/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:6:in `<main>'
  12: from /Users/dentarg/.gem/ruby/2.7.6/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:6:in `select'
  11: from /Users/dentarg/.gem/ruby/2.7.6/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:21:in `block in <main>'
  10: from /Users/dentarg/.gem/ruby/2.7.6/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:21:in `require'
   9: from /Users/dentarg/starkast/wikimum/test/unit/markup_test.rb:4:in `<top (required)>'
   8: from /Users/dentarg/starkast/wikimum/test/unit/markup_test.rb:4:in `require_relative'
   7: from /Users/dentarg/starkast/wikimum/lib/services/markup.rb:3:in `<top (required)>'
   6: from /Users/dentarg/starkast/wikimum/lib/services/markup.rb:3:in `require'
   5: from /Users/dentarg/.gem/ruby/2.7.6/gems/html-pipeline-2.14.2/lib/html/pipeline.rb:3:in `<top (required)>'
   4: from /Users/dentarg/.gem/ruby/2.7.6/gems/html-pipeline-2.14.2/lib/html/pipeline.rb:3:in `require'
   3: from /Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri.rb:10:in `<top (required)>'
   2: from /Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri.rb:10:in `require_relative'
   1: from /Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri/extension.rb:7:in `<top (required)>'
/Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri/extension.rb:7:in `require_relative': cannot load such file -- /Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri/2.7/nokogiri (LoadError)
  14: from /Users/dentarg/.gem/ruby/2.7.6/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:6:in `<main>'
  13: from /Users/dentarg/.gem/ruby/2.7.6/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:6:in `select'
  12: from /Users/dentarg/.gem/ruby/2.7.6/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:21:in `block in <main>'
  11: from /Users/dentarg/.gem/ruby/2.7.6/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:21:in `require'
  10: from /Users/dentarg/starkast/wikimum/test/unit/markup_test.rb:4:in `<top (required)>'
   9: from /Users/dentarg/starkast/wikimum/test/unit/markup_test.rb:4:in `require_relative'
   8: from /Users/dentarg/starkast/wikimum/lib/services/markup.rb:3:in `<top (required)>'
   7: from /Users/dentarg/starkast/wikimum/lib/services/markup.rb:3:in `require'
   6: from /Users/dentarg/.gem/ruby/2.7.6/gems/html-pipeline-2.14.2/lib/html/pipeline.rb:3:in `<top (required)>'
   5: from /Users/dentarg/.gem/ruby/2.7.6/gems/html-pipeline-2.14.2/lib/html/pipeline.rb:3:in `require'
   4: from /Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri.rb:10:in `<top (required)>'
   3: from /Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri.rb:10:in `require_relative'
   2: from /Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri/extension.rb:4:in `<top (required)>'
   1: from /Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri/extension.rb:30:in `rescue in <top (required)>'
/Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri/extension.rb:30:in `require': dlopen(/Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri/nokogiri.bundle, 0x0009): tried: '/Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri/nokogiri.bundle' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e))) - /Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri/nokogiri.bundle (LoadError)
rake aborted!

I know chruby 1.0.0 will bring changes to GEM_HOME (https://github.com/postmodern/chruby/pull/419) but I don’t think using basename is enough, as it will be the same basename regardless of the architecture.