rails: ActionCable Redis adapter crashes with Heroku Redis 6

Steps to reproduce

Any Rails project with ActionCable configured using the Redis adapter and a Redis url pointing to a Heroku Redis instance. Then upgrade (or already use) Redis 6 for this instance. Heroku requires TLS/SSL connections for Premium Redis 6 instances and I think that causes the problem. After rebooting the application and visiting any page that uses ActionCable, then the application crashes.

Expected behavior

I would expect the application (page) to not crash and ActionCable to function correctly (delivering the push notifications).

Actual behavior

The page crashes with an OpenSSL::SSL::SSLError exception:

2021-04-19T10:09:51.134544+00:00 app[web.1]: #<Thread:0x000055f0144a9e58@/app/vendor/bundle/ruby/2.6.0/gems/actioncable-6.0.3.6/lib/action_cable/subscription_adapter/redis.rb:151 run> terminated with exception (report_on_exception is true):
2021-04-19T10:09:51.134580+00:00 app[web.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:260:in `connect_nonblock': SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) (OpenSSL::SSL::SSLError)
2021-04-19T10:09:51.134584+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:260:in `connect'
2021-04-19T10:09:51.134586+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:302:in `connect'
2021-04-19T10:09:51.134587+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:354:in `establish_connection'
2021-04-19T10:09:51.134622+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:112:in `block in connect'
2021-04-19T10:09:51.134623+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:313:in `with_reconnect'
2021-04-19T10:09:51.134623+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:111:in `connect'
2021-04-19T10:09:51.134624+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:294:in `with_socket_timeout'
2021-04-19T10:09:51.134624+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:144:in `call_loop'
2021-04-19T10:09:51.134624+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/subscribe.rb:44:in `subscription'
2021-04-19T10:09:51.134627+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/subscribe.rb:14:in `subscribe'
2021-04-19T10:09:51.134627+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:3507:in `_subscription'
2021-04-19T10:09:51.134628+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:2326:in `block in subscribe'
2021-04-19T10:09:51.134630+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:69:in `block in synchronize'
2021-04-19T10:09:51.134632+00:00 app[web.1]: from /app/vendor/ruby-2.6.6/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
2021-04-19T10:09:51.134635+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:69:in `synchronize'
2021-04-19T10:09:51.134675+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:2325:in `subscribe'
2021-04-19T10:09:51.134678+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/actioncable-6.0.3.6/lib/action_cable/subscription_adapter/redis.rb:84:in `block in listen'
2021-04-19T10:09:51.134678+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:313:in `with_reconnect'
2021-04-19T10:09:51.134681+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:75:in `block in with_reconnect'
2021-04-19T10:09:51.134683+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:69:in `block in synchronize'
2021-04-19T10:09:51.134730+00:00 app[web.1]: from /app/vendor/ruby-2.6.6/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
2021-04-19T10:09:51.134731+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:69:in `synchronize'
2021-04-19T10:09:51.134731+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:74:in `with_reconnect'
2021-04-19T10:09:51.134732+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:81:in `without_reconnect'
2021-04-19T10:09:51.134735+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/actioncable-6.0.3.6/lib/action_cable/subscription_adapter/redis.rb:81:in `listen'
2021-04-19T10:09:51.134736+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/actioncable-6.0.3.6/lib/action_cable/subscription_adapter/redis.rb:155:in `block in ensure_listener_running'
2021-04-19T10:09:51.134910+00:00 app[web.1]: Exiting

System configuration

Rails version: 6.0.3.6

Ruby version: 2.6.6

Notes

I think this problem is caused because Heroku uses self-signed certificates as described here. The issue can be resolved by setting verify_mode: OpenSSL::SSL::VERIFY_NONE when initializing the Redis connection, but it is unclear to me how to do this conveniently for ActionCable, since the adapter initializes the connection instead of doing it manually. Setting VERIFY_NONE did work however when setting-up a custom connection through Redis.new.

Not sure if this is an actual bug, might be that I just missed a configuration option or a piece of documentation. I’m wondering if anybody else ran into this issue and knows how to fix it (my searches turned up empty). Or if some configuration option to ActionCable should be added.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 20 (8 by maintainers)

Most upvoted comments

I found the solution. You can disable SSL verify in cable.yml using the following config:

production:
  adapter: redis
  url: <%= ENV.fetch("REDIS_URL") { "redis://localhost:6379/1" } %>
  channel_prefix: sample_6_0_production
  ssl_params:
    verify_mode: <%= OpenSSL::SSL::VERIFY_NONE %>

However it seems that the ability to specify SSL configuration for the Redis connection for ActionCable is poorly documented (or at least I couldn’t find anything about it). Might it be worth to add this to the ActionCable guide? If so, then I will create a PR. If not, then this issue can be closed.

In the mean time, I hope this issue helps anybody running into this issue as well.

+10

kreintjes on Apr 22, 2021

In case anyone has the same problem, it was indeed a Heroku issue! Because RACK_ENV and RAILS_ENV were set to staging (we wanted to test before upgrading on prod), Heroku ignores what you have in cable.yml for some reason. Thanks again @zzak

annienero on Jun 16, 2021

@ryanstrickler @pixeltrix hi all, I’m running into this issue on Rails 5.2.6 despite setting:

  ssl_params:
    verify_mode: <%= OpenSSL::SSL::VERIFY_NONE %>

in my cable.yml config file.

I believe this is because in actioncable-5.2.6/lib/action_cable/subscription_adapter/redis.rb the ssl_params key does not get passed through to redis (see below where only certain keys from the config are sliced):

module ActionCable
  module SubscriptionAdapter
    class Redis < Base # :nodoc:
      prepend ChannelPrefix

      # Overwrite this factory method for Redis connections if you want to use a different Redis library than the redis gem.
      # This is needed, for example, when using Makara proxies for distributed Redis.
      cattr_accessor :redis_connector, default: ->(config) do
        ::Redis.new(config.slice(:url, :host, :port, :db, :password))
      end

On Rails 6 it looks like config.except(...) rather than config.slice is used, so that all keys in the config hash are included by default, which seems like a good solution to me.

Let me know if I can provide any more details, thanks! I’m happy to make a PR for this if what I suggested sounds reasonable.

Murphydbuffalo on Jul 23, 2021

@Murphydbuffalo It’s not possible to backport this patch to Rails 5 now, I would suggest upgrading your application to Rails 6 if that fixes the issue. 🙇

zzak on Aug 8, 2021

@ryanstrickler yeah we just tested it on our staging instance with updated vars and then swapped it back. We didn’t have any issues on prod. We’re on Rails 6. When I was testing locally trying to connect to our staging Redis instance, it seemed like there was a pretty particular syntax we needed to make ssl_params work. In cable.yml, we have

production:
  adapter: redis
  url: <%= ENV.fetch("REDIS_URL") %>
  ssl_params: {
    verify_mode: <%= OpenSSL::SSL::VERIFY_NONE %>
  }

Hope something in there helps!

annienero on Jul 8, 2021

@Murphydbuffalo Sorry I didn’t see this earlier. Looks like you got farther on this than we did. Heroku support couldn’t find any problems with our configuration and suggested we start with a fresh Rails 5.2 install and try to reproduce the issue. We decided it would be better to put that time into upgrading to Rails 6, so that’s our plan, but if we try your fix and it works, I’ll report back.

ryanstrickler on Aug 10, 2021

Having a similar issue but the above solution hasn’t helped. Let me know if I should open a separate issue/an issue with graphql instead.

For us, the app doesn’t crash but our graphql subscriptions aren’t working and we’re getting OpenSSL::SSL::SSLError errors in our logs as well.

I updated our graphql schema definition to have use(GraphQL::Subscriptions::ActionCableSubscriptions, redis: Redis.new(url: ENV['REDIS_URL'], ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE })) and added

production:
  adapter: redis
  url: <%= ENV.fetch("REDIS_URL") { "redis://localhost:6379/1" } %>
  ssl_params:
    verify_mode: <%= OpenSSL::SSL::VERIFY_NONE %>

to our cable.yml

I also tried setting config.cache_store = :redis_cache_store, { url: ENV['REDIS_URL'], ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } } in config/environments/production.rb but haven’t had any luck with any combination of the above.

Not sure if it’s relevant because it seemed like a separate, resolved issue but we also happen to use redis with sidekiq and have updated our config/initializers/sidekiq.rb to contain

Sidekiq.configure_server do |config|
  config.redis = { ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } }
end

Sidekiq.configure_client do |config|
  config.redis = { ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } }
end

We’re on rails 6.0.3.7 and ruby 2.7.1. Thanks in advance.

annienero on May 17, 2021

@ashiksp No I did not have a chance to work on it yet. I’m still happy to do so eventually, but not sure when I have time. If you want to make a PR for it, then that would be great as well!

kreintjes on May 5, 2021