rollbar-gem: Failsafe from rollbar-gem. OpenSSL::SSL::SSLError: \"SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)

Hi, we are getting errors like these in production, and wondering if this would be the best place to report it.

[Rollbar] Error processing the item: OpenSSL::SSL::SSLError, SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate). 
Item: {"access_token"=>"***", "data"=>{"level"=>"error", "environment"=>"production", 
"body"=>{"message"=>{"body"=>"Failsafe from rollbar-gem. OpenSSL::SSL::SSLError: \"SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)\" 
in /usr/local/lib/ruby/2.5.0/net/protocol.rb:44:in `connect_nonblock': error in process_item"}}, 
"notifier"=>{"name"=>"rollbar-gem", "version"=>"2.18.2"},
 "custom"=>{"orig_uuid"=>nil, "orig_host"=>nil}, "internal"=>true, "failsafe"=>true}}

A more detailed stacktrace:

              /usr/local/lib/ruby/2.6.0/net/
protocol.rb:  44:in `connect_nonblock'
              /usr/local/lib/ruby/2.6.0/net/
protocol.rb:  44:in `ssl_socket_connect'
                  /usr/local/lib/ruby/2.6.0/net/
http.rb: 996:in `connect'
                  /usr/local/lib/ruby/2.6.0/net/
http.rb: 930:in `do_start'
                  /usr/local/lib/ruby/2.6.0/net/
http.rb: 919:in `start'
                  /usr/local/lib/ruby/2.6.0/net/
http.rb:1470:in `request'
…ylight-core-4.1.2/lib/skylight/core/probes/
net_http.rb:  27:in `request'
…ms/elastic-apm-3.7.0/lib/elastic_apm/spies/
net_http.rb:  89:in `block in request'
…e/ruby/2.6.0/gems/elastic-apm-3.7.0/lib/
elastic_apm.rb: 290:in `with_span'
…ms/elastic-apm-3.7.0/lib/elastic_apm/spies/
net_http.rb:  79:in `request'
…ruby/2.6.0/gems/rollbar-2.19.3/lib/rollbar/notifier.rb: 555:in `block in do_post'
…ruby/2.6.0/gems/rollbar-2.19.3/lib/rollbar/notifier.rb: 616:in `handle_net_retries'
…ruby/2.6.0/gems/rollbar-2.19.3/lib/rollbar/notifier.rb: 555:in `do_post'
…ruby/2.6.0/gems/rollbar-2.19.3/lib/rollbar/notifier.rb: 533:in `send_item'
…ruby/2.6.0/gems/rollbar-2.19.3/lib/rollbar/notifier.rb: 204:in `process_item'
…ruby/2.6.0/gems/rollbar-2.19.3/lib/rollbar/notifier.rb: 244:in `block in process_from_async_handler'
…ruby/2.6.0/gems/rollbar-2.19.3/lib/rollbar/notifier.rb:  98:in `silenced'
               /usr/local/lib/ruby/2.6.0/forwardable.rb: 230:in `silenced'
…ruby/2.6.0/gems/rollbar-2.19.3/lib/rollbar/notifier.rb: 242:in `process_from_async_handler'
               /usr/local/lib/ruby/2.6.0/forwardable.rb: 230:in `process_from_async_handler'
…2.6.0/gems/rollbar-2.19.3/lib/rollbar/delay/sidekiq.rb:  19:in `perform'
…ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/processor.rb: 193:in `execute_job'
<truncated 17 additional frames>
…ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/processor.rb: 160:in `block in process'
…ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/processor.rb: 133:in `block (6 levels) in dispatch'
…ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/job_retry.rb: 110:in `local'
…ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/processor.rb: 132:in `block (5 levels) in dispatch'
…dle/ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/rails.rb:  43:in `block in call'
…esupport-5.2.3/lib/active_support/execution_wrapper.rb:  87:in `wrap'
…ems/activesupport-5.2.3/lib/active_support/reloader.rb:  73:in `block in wrap'
…esupport-5.2.3/lib/active_support/execution_wrapper.rb:  87:in `wrap'
…ems/activesupport-5.2.3/lib/active_support/reloader.rb:  72:in `wrap'
…dle/ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/rails.rb:  42:in `call'
…ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/processor.rb: 128:in `block (4 levels) in dispatch'
…ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/processor.rb: 254:in `stats'
…ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/processor.rb: 123:in `block (3 levels) in dispatch'
…uby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/job_logger.rb:  13:in `call'
…ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/processor.rb: 122:in `block (2 levels) in dispatch'
…ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/job_retry.rb:  78:in `global'
…ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/processor.rb: 121:in `block in dispatch'
…le/ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/logger.rb:  10:in `with'
…uby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/job_logger.rb:  33:in `prepare'
…ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/processor.rb: 120:in `dispatch'
…ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/processor.rb: 159:in `process'
…ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/processor.rb:  78:in `process_one'
…ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/processor.rb:  68:in `run'
…ndle/ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/util.rb:  17:in `watchdog'
…ndle/ruby/2.6.0/gems/sidekiq-6.0.1/lib/sidekiq/util.rb:  26:in `block in safe_thread'

Because of the existence of “Failsafe from rollbar-gem”, I assume that this issue is happening with the rollbar-gem, and just wanted to confirm.

Update: We haven’t seen logs show up in the rollbar dashboard since these errors started.

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 33
Comments: 20 (3 by maintainers)

Most upvoted comments

Should be working for everyone now.

waltjones on Jul 14, 2020

Contact support and ask for your quota back. I think they’ll be understanding.

waltjones on Jul 14, 2020

And there goes our monthly rollbar quota.

nkipreos on Jul 14, 2020

Hi guys, we are experiencing the same issue, any one here that has managed some kind of workaround for this?

nkipreos on Jul 14, 2020

@derekyau I recommend you remove your access_token from your comment

danielolivaresd on Jul 14, 2020

Please use 👍 instead of spam-posting with +1s 😃

d3mash on Jul 14, 2020

If your Sidekiq was flooded by the failsafe mechanism, you should try updating your gem. 2.23.0 supposedly fixes the issue, while 2.25.0 and 2.26.0 fixes other failsafe-related issues.

lmansur on Jul 14, 2020

FYI, I think this is the entry on Rollbar’s status page for this issue: https://status.rollbar.com/incidents/39rm7hwmplpm

danielolivaresd on Jul 14, 2020

From a sidekiq worker running heroku-18:

~ $ curl -vvv https://api.rollbar.com
* Rebuilt URL to: https://api.rollbar.com/
*   Trying 35.201.81.77...
* TCP_NODELAY set
* Connected to api.rollbar.com (35.201.81.77) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
~ $

ran this 2020-07-14 11:32:55 Eastern

bmcniel on Jul 14, 2020

This also caused our sidekiq queues to spike and create unlimited Rollbar delay jobs… we definitely need this fixed as soon as possible, thanks Rollbar.

Stack trace: Rollbar::Delay::Sidekiq {"access_token"=>"<REDACTED>", "data"=>{"level"=>"error", "environment"=>"production", "body"=>{"message"=>{"body"=>"Failsafe from rollbar-gem. OpenSSL::SSL::SSLError: \"SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)\" in /home/deployer/.rbenv/versions/2.6.5/lib/ruby/2.6.0/net/protocol.rb:44:in connect_nonblock’: error in process_item"}}, “notifier”=>{“name”=>“rollbar-gem”, “version”=>“2.24.0”}, “custom”=>{“orig_uuid”=>nil, “orig_host”=>nil}, “internal”=>true, “failsafe”=>true}}`

derekyau on Jul 14, 2020