bootsnap: ibf_dump_object_unsupported […] T_NONE (NotImplementedError)

See: Ruby upstream issue https://bugs.ruby-lang.org/issues/19419 See: Repo with reproduction case https://github.com/tisba/bootsnap-ruby-crash

Steps to reproduce

I’m having a hard time to produce a minimal version that reliably reproduces this error. Locally, with a Gemfile just containing aws-sdk-ec2, nokogiri and bootsnap I have seen it too, but cannot reproduce that reliably on my local machine yet. It seems to happen on a slightly more regular basis in our GH Action CI run, but also not 100% of the time.

According to the trace (see below) the NotImplementedError happens when loading clients_api.rb from the AWS SDK. Note that clients_api.rb is a MASSIVE file with >20,000 LOC. Maybe the size is (part of) the problem?

Is there something I can do to provide more useful information to debug this issue?

Expected behavior

It should not crash.

Actual behavior

It crashes.

System configuration

Bootsnap version: 1.16.0

Ruby version: 3.2.0, 3.2.1 (on aarch64, x86)

Rails version: 7.0.4.2 (not relevant)

Backtrace

In our CI it fails when we do a sanity check of the Rails app via RAILS_ENV=production bundle exec rails runner 'puts "Hello from #{Rails.env}!"':

/home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/compile_cache/iseq.rb:42:in `to_binary': ibf_dump_object_unsupported: 0x00007f11b8efff58 [0      ] T_NONE  (NotImplementedError)
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/compile_cache/iseq.rb:42:in `input_to_storage'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/compile_cache/iseq.rb:60:in `fetch'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/compile_cache/iseq.rb:60:in `fetch'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/compile_cache/iseq.rb:85:in `load_iseq'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/aws-sdk-ec2-1.364.0/lib/aws-sdk-ec2.rb:15:in `require_relative'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/aws-sdk-ec2-1.364.0/lib/aws-sdk-ec2.rb:15:in `<main>'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
	from /opt/hostedtoolcache/Ruby/3.2.0/x64/lib/ruby/gems/3.2.0/gems/bundler-2.4.3/lib/bundler/runtime.rb:60:in `block (2 levels) in require'
	from /opt/hostedtoolcache/Ruby/3.2.0/x64/lib/ruby/gems/3.2.0/gems/bundler-2.4.3/lib/bundler/runtime.rb:55:in `each'
	from /opt/hostedtoolcache/Ruby/3.2.0/x64/lib/ruby/gems/3.2.0/gems/bundler-2.4.3/lib/bundler/runtime.rb:55:in `block in require'
	from /opt/hostedtoolcache/Ruby/3.2.0/x64/lib/ruby/gems/3.2.0/gems/bundler-2.4.3/lib/bundler/runtime.rb:44:in `each'
	from /opt/hostedtoolcache/Ruby/3.2.0/x64/lib/ruby/gems/3.2.0/gems/bundler-2.4.3/lib/bundler/runtime.rb:44:in `require'
	from /opt/hostedtoolcache/Ruby/3.2.0/x64/lib/ruby/gems/3.2.0/gems/bundler-2.4.3/lib/bundler.rb:195:in `require'
	from /home/runner/work/foo/foo/config/application.rb:13:in `<main>'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.2/lib/rails/command/actions.rb:22:in `require_application!'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.2/lib/rails/command/actions.rb:14:in `require_application_and_environment!'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.2/lib/rails/commands/runner/runner_command.rb:33:in `perform'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.2/lib/rails/command/base.rb:87:in `perform'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.2/lib/rails/command.rb:48:in `invoke'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.2/lib/rails/commands.rb:18:in `<main>'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
	from /home/runner/work/foo/foo/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
	from bin/rails:5:in `<main>'

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 13
  • Comments: 26 (14 by maintainers)

Commits related to this issue

Most upvoted comments

Report submitted upstream: https://bugs.ruby-lang.org/issues/19419

+7
casperisfine on Feb 6, 2023

Ok, so Alan removed the write barrier and marked the patch as needing a backport, so this should be fixed in Ruby 3.2.2.

Unfortunately there’s not much bootsnap can do to work around this bug. On your end you may want to do some dirty hack such as:

GC.disable
require "aws-sdk-ec2"
GC.enable

I’ll close this as there isn’t much to do now.

+6
casperisfine on Feb 13, 2023

I don’t think it’s related to Nokogiri, if I replace it by rexml in the Gemfile I do get GC issues (a bit different):

$ while true; do \rm -rf tmp; BOOTSNAP_CACHE_DIR=tmp/ time bundle exec ruby crash.rb; done
/Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2/client_api.rb:7254:in `<module:ClientApi>': undefined method `new' for #<Seahorse::Model::Shapes::StringShape:0x0000000106ffd8c0 @metadata={}, @name="String"> (NoMethodError)

    DescribeVerifiedAccessTrustProvidersResult.add_member(:next_token, Shapes::ShapeRef.new(shape: NextToken, location_name: "nextToken"))
                                                                                       ^^^^
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2/client_api.rb:12:in `<module:EC2>'
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2/client_api.rb:10:in `<main>'
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2.rb:15:in `require_relative'
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2.rb:15:in `<main>'
	from /Users/byroot/.gem/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
	from /Users/byroot/.gem/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
	from crash.rb:2:in `<main>'
        0.97 real         0.55 user         0.31 sys
/Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2/client_api.rb:7247:in `<module:ClientApi>': undefined method `new' for #<Seahorse::Model::Shapes::StringShape:0x000000010bcfd7d8 @metadata={}, @name="String"> (NoMethodError)

    DescribeVerifiedAccessTrustProvidersRequest.add_member(:max_results, Shapes::ShapeRef.new(shape: DescribeVerifiedAccessTrustProvidersMaxResults, location_name: "MaxResults"))
                                                                                         ^^^^
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2/client_api.rb:12:in `<module:EC2>'
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2/client_api.rb:10:in `<main>'
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2.rb:15:in `require_relative'
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2.rb:15:in `<main>'
	from /Users/byroot/.gem/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
	from /Users/byroot/.gem/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
	from crash.rb:2:in `<main>'
        0.87 real         0.54 user         0.27 sys
/Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-core-3.170.0/lib/aws-sdk-core/structure.rb:9:in `[]=': no member 'shape' in struct (NameError)

        self[k] = v
            ^^^^^
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-core-3.170.0/lib/aws-sdk-core/structure.rb:9:in `block in initialize'
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-core-3.170.0/lib/aws-sdk-core/structure.rb:8:in `each'
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-core-3.170.0/lib/aws-sdk-core/structure.rb:8:in `initialize'
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2/client_api.rb:7253:in `new'
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2/client_api.rb:7253:in `<module:ClientApi>'
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2/client_api.rb:12:in `<module:EC2>'
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2/client_api.rb:10:in `<main>'
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2.rb:15:in `require_relative'
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2.rb:15:in `<main>'
	from /Users/byroot/.gem/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
	from /Users/byroot/.gem/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
	from crash.rb:2:in `<main>'
        0.84 real         0.49 user         0.29 sys
/Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2/client_api.rb:7249:in `<module:ClientApi>': undefined method `new' for #<Seahorse::Model::Shapes::StringShape:0x000000011cb3d900 @metadata={}, @name="String"> (NoMethodError)

    DescribeVerifiedAccessTrustProvidersRequest.add_member(:filters, Shapes::ShapeRef.new(shape: FilterList, location_name: "Filter"))
                                                                                     ^^^^
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2/client_api.rb:12:in `<module:EC2>'
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2/client_api.rb:10:in `<main>'
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2.rb:15:in `require_relative'
	from /Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2.rb:15:in `<main>'
	from /Users/byroot/.gem/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
	from /Users/byroot/.gem/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
	from crash.rb:2:in `<main>'
        0.94 real         0.55 user         0.31 sys
/Users/byroot/.gem/ruby/3.2.0/gems/aws-sdk-ec2-1.365.0/lib/aws-sdk-ec2/client_api.rb:7246:in `<module:ClientApi>': undefined method `new' for #<Seahorse::Model::Shapes::StructureShape:0x000000010aa943d0 @members={}, @members_by_location_name={}, @required=#<Set: {}>, @metadata={}, @name="SpotFleetRequestConfig"> (NoMethodError)
+1
casperisfine on Feb 13, 2023

but sounds really linked, when we don’t have this error, we have this one:

Yes. What is happening is that one reference isn’t properly marked, so when gc trigger the slot is recycled. If the slot is empty you’ll get that T_NONE error, but if it’s used by another object you can get weird NoMethodError like yours.

At this stage what could help:

  • A core dump, to be able to inspect the heap state.
  • A crash report with the C Level Backtrace but with gc.stress = true. This is very slow, but would properly pinpoint where the bug is.
+1
casperisfine on Feb 7, 2023
Read more comments on GitHub
← bootsnap: Fails to load iseq cache in Linux
bootsnap: incompatible with armv7l →