sentry-ruby: Stack level to deep error after upgrading to 4.4.0

In the version 4.4.0, we added support for capturing net/http requests as span/breadcrumbs. This feature requires patching the class, which can be done in 2 different but conflicting ways: method aliasing and module prepending. Here’s a great article from bearer explaining how these 2 approaches conflict with each other.

Fortunately, the community has been adopting the prepend approach for a while now + sentry-ruby requires Ruby 2.4+. So the chance of having the issue should be low.

How we’re going to help you upgrade

With that being said, if you’re reading this issue, chances are you’re having the conflict inside your app. So here’s our plan to help you upgrade:

  1. We’ll see if it’s possible to align your other dependencies toward the prepend approach. Most of the gems that patch this library should have provided an option to switch the mode to prepend. For example, rack-mini-profiler and its prepend_net_http_patch option.
  2. If the above option is not viable to you, we’ll decide whether which approach to take based on your feedback:
    • Provide a way to opt-out this patch.
    • Provide an option to switch modes like some other gems do.

Update

The version 4.4.1 should solve the issue.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 6
  • Comments: 26

Commits related to this issue

Most upvoted comments

FYI: This is very much not fixed in 4.4.1. We just tried a production deploy including v4.4.1 last night and this upgrade inadvertently took our entire site down.

Not sure how much it will help, but see attached log.

Still tracking.

Update: scout_apm mentioned that they plan to switch to the prepend approach too in the upcoming 5.0 version:

I am prepping a 4.1.0 version of the agent with several bug fixes, and then after that looking at 5.0. The big breaking change is switching to all prepend by default, and adding a configuration option to use alias if that causes compatibility issues with other gems.

While every ruby we support also has prepend, many other gems use alias_method, and that causes issues when mixed. Keep an eye out for this change, should be up pretty quick.

https://github.com/scoutapp/scout_apm_ruby/pull/402#issuecomment-846480983

# config/scout_apm.yml
common: &defaults
  disabled_instruments:
    - "NetHttp"

solves the issue for me

My config:

Sentry.init do |config|
  config.dsn = Rails.application.credentials[:sentry][:dsn]
  config.enabled_environments = %w[production staging]
  config.breadcrumbs_logger = [:sentry_logger] 
  config.excluded_exceptions = []
end

Given that https://github.com/scoutapp/scout_apm_ruby/pull/448 has been released in version 5.3.0. I’m closing this now.

If you have any similar issues, I recommend:

  1. Check if the conflicting package patches the component with alias_method
    • If it does, please file an issue there because they should migrate to prepend or at least provide an option for it.
  2. If it doesn’t, check if the version you are using uses alias_method
    • If your version uses alias_method, please upgrade to a version that doesn’t
    • If the version you’re using doesn’t use alias_method, but still causes this error with the SDK, please file a new issue

Following @adi-pen’s comment:

@st0012 I ran into the same issue due to the gem skylight. This comment helped.

I saw Skylight’s v5 announcement seems relevant:

“Skylight 5 also includes a shift to Module#prepend for adding instrumentation to Rails”

Updating to Skylight v5 worked for us - required update to Ruby 2.7.3 however. (https://github.com/publiclab/mapknitter/pull/1712) (error log had been this)

@jpaulomotta There’s a glimmer of hope from Scout on this one but it’s painfully slow going for some reason: https://github.com/scoutapp/scout_apm_ruby/pull/448

So this is still very much broken despite upgrading to the latest versions of scout_apm and sentry-*.

Our latest backtrace is here. I turned on some debugging information, which might make things easier to trace.

I was finally able to reproduce the problem on a non-production instance, and – trying to debug this a bit – it looks like the execution flow is this:

The request method in Net:HTTP is defined:

irb(main):004:0> Net::HTTP.instance_method(:request)
=> #<UnboundMethod: Net::HTTP(Sentry::Net::HTTP)#request(req, body=..., &block) /usr/local/bundle/gems/sentry-ruby-core-4.5.1/lib/sentry/net/http.rb:35>

… and that #request method calls its super, which is:

irb(main):006:0> Net::HTTP.instance_method(:request).super_method
=> #<UnboundMethod: Net::HTTP#request(request_with_scout_instruments)(*args, &block) /usr/local/bundle/gems/scout_apm-4.1.1/lib/scout_apm/instruments/net_http.rb:28>

that method, Net::HTTP.request_with_scout_instruments, then calls request_without_scout_instruments, which is:

irb(main):003:0> Net::HTTP.instance_method(:request_without_scout_instruments)
=> #<UnboundMethod: Net::HTTP(Sentry::Net::HTTP)#request_without_scout_instruments(request)(req, body=..., &block) /usr/local/bundle/gems/sentry-ruby-core-4.5.1/lib/sentry/net/http.rb:35>

… thereby creating the infinite loop.

I am not able to reproduce on a brand new rails app though, so I imagine the issue could just be with order in which these hooks are applied.

sentry-ruby 4.4.1 is out now, which should solve the issue. Sorry for the inconvenience.