rails: ActionCable Redis adapter crashes with Heroku Redis 6
Steps to reproduce
Any Rails project with ActionCable configured using the Redis adapter and a Redis url pointing to a Heroku Redis instance. Then upgrade (or already use) Redis 6 for this instance. Heroku requires TLS/SSL connections for Premium Redis 6 instances and I think that causes the problem. After rebooting the application and visiting any page that uses ActionCable, then the application crashes.
Expected behavior
I would expect the application (page) to not crash and ActionCable to function correctly (delivering the push notifications).
Actual behavior
The page crashes with an OpenSSL::SSL::SSLError exception:
2021-04-19T10:09:51.134544+00:00 app[web.1]: #<Thread:0x000055f0144a9e58@/app/vendor/bundle/ruby/2.6.0/gems/actioncable-6.0.3.6/lib/action_cable/subscription_adapter/redis.rb:151 run> terminated with exception (report_on_exception is true):
2021-04-19T10:09:51.134580+00:00 app[web.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:260:in `connect_nonblock': SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) (OpenSSL::SSL::SSLError)
2021-04-19T10:09:51.134584+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:260:in `connect'
2021-04-19T10:09:51.134586+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:302:in `connect'
2021-04-19T10:09:51.134587+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:354:in `establish_connection'
2021-04-19T10:09:51.134622+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:112:in `block in connect'
2021-04-19T10:09:51.134623+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:313:in `with_reconnect'
2021-04-19T10:09:51.134623+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:111:in `connect'
2021-04-19T10:09:51.134624+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:294:in `with_socket_timeout'
2021-04-19T10:09:51.134624+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:144:in `call_loop'
2021-04-19T10:09:51.134624+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/subscribe.rb:44:in `subscription'
2021-04-19T10:09:51.134627+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/subscribe.rb:14:in `subscribe'
2021-04-19T10:09:51.134627+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:3507:in `_subscription'
2021-04-19T10:09:51.134628+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:2326:in `block in subscribe'
2021-04-19T10:09:51.134630+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:69:in `block in synchronize'
2021-04-19T10:09:51.134632+00:00 app[web.1]: from /app/vendor/ruby-2.6.6/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
2021-04-19T10:09:51.134635+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:69:in `synchronize'
2021-04-19T10:09:51.134675+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:2325:in `subscribe'
2021-04-19T10:09:51.134678+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/actioncable-6.0.3.6/lib/action_cable/subscription_adapter/redis.rb:84:in `block in listen'
2021-04-19T10:09:51.134678+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:313:in `with_reconnect'
2021-04-19T10:09:51.134681+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:75:in `block in with_reconnect'
2021-04-19T10:09:51.134683+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:69:in `block in synchronize'
2021-04-19T10:09:51.134730+00:00 app[web.1]: from /app/vendor/ruby-2.6.6/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
2021-04-19T10:09:51.134731+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:69:in `synchronize'
2021-04-19T10:09:51.134731+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:74:in `with_reconnect'
2021-04-19T10:09:51.134732+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:81:in `without_reconnect'
2021-04-19T10:09:51.134735+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/actioncable-6.0.3.6/lib/action_cable/subscription_adapter/redis.rb:81:in `listen'
2021-04-19T10:09:51.134736+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.6.0/gems/actioncable-6.0.3.6/lib/action_cable/subscription_adapter/redis.rb:155:in `block in ensure_listener_running'
2021-04-19T10:09:51.134910+00:00 app[web.1]: Exiting
System configuration
Rails version: 6.0.3.6
Ruby version: 2.6.6
Notes
I think this problem is caused because Heroku uses self-signed certificates as described here. The issue can be resolved by setting verify_mode: OpenSSL::SSL::VERIFY_NONE
when initializing the Redis connection, but it is unclear to me how to do this conveniently for ActionCable, since the adapter initializes the connection instead of doing it manually. Setting VERIFY_NONE did work however when setting-up a custom connection through Redis.new
.
Not sure if this is an actual bug, might be that I just missed a configuration option or a piece of documentation. I’m wondering if anybody else ran into this issue and knows how to fix it (my searches turned up empty). Or if some configuration option to ActionCable should be added.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 20 (8 by maintainers)
I found the solution. You can disable SSL verify in
cable.yml
using the following config:However it seems that the ability to specify SSL configuration for the Redis connection for ActionCable is poorly documented (or at least I couldn’t find anything about it). Might it be worth to add this to the ActionCable guide? If so, then I will create a PR. If not, then this issue can be closed.
In the mean time, I hope this issue helps anybody running into this issue as well.
In case anyone has the same problem, it was indeed a Heroku issue! Because RACK_ENV and RAILS_ENV were set to staging (we wanted to test before upgrading on prod), Heroku ignores what you have in
cable.yml
for some reason. Thanks again @zzak@ryanstrickler @pixeltrix hi all, I’m running into this issue on Rails 5.2.6 despite setting:
in my
cable.yml
config file.I believe this is because in
actioncable-5.2.6/lib/action_cable/subscription_adapter/redis.rb
thessl_params
key does not get passed through to redis (see below where only certain keys from the config are sliced):On Rails 6 it looks like
config.except(...)
rather thanconfig.slice
is used, so that all keys in the config hash are included by default, which seems like a good solution to me.Let me know if I can provide any more details, thanks! I’m happy to make a PR for this if what I suggested sounds reasonable.
@Murphydbuffalo It’s not possible to backport this patch to Rails 5 now, I would suggest upgrading your application to Rails 6 if that fixes the issue. 🙇
@ryanstrickler yeah we just tested it on our staging instance with updated vars and then swapped it back. We didn’t have any issues on prod. We’re on Rails 6. When I was testing locally trying to connect to our staging Redis instance, it seemed like there was a pretty particular syntax we needed to make
ssl_params
work. Incable.yml
, we haveHope something in there helps!
@Murphydbuffalo Sorry I didn’t see this earlier. Looks like you got farther on this than we did. Heroku support couldn’t find any problems with our configuration and suggested we start with a fresh Rails 5.2 install and try to reproduce the issue. We decided it would be better to put that time into upgrading to Rails 6, so that’s our plan, but if we try your fix and it works, I’ll report back.
Having a similar issue but the above solution hasn’t helped. Let me know if I should open a separate issue/an issue with graphql instead.
For us, the app doesn’t crash but our graphql subscriptions aren’t working and we’re getting
OpenSSL::SSL::SSLError
errors in our logs as well.I updated our graphql schema definition to have
use(GraphQL::Subscriptions::ActionCableSubscriptions, redis: Redis.new(url: ENV['REDIS_URL'], ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE }))
and addedto our cable.yml
I also tried setting
config.cache_store = :redis_cache_store, { url: ENV['REDIS_URL'], ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } }
inconfig/environments/production.rb
but haven’t had any luck with any combination of the above.Not sure if it’s relevant because it seemed like a separate, resolved issue but we also happen to use redis with sidekiq and have updated our
config/initializers/sidekiq.rb
to containWe’re on rails 6.0.3.7 and ruby 2.7.1. Thanks in advance.
@ashiksp No I did not have a chance to work on it yet. I’m still happy to do so eventually, but not sure when I have time. If you want to make a PR for it, then that would be great as well!