redis-rb: disconnect on certain types of connection errors
When certain kinds of connection errors happen, disconnect to allow reconnection on next operation.
We ran into an issue where AWS ElastiCache switched the current master into a slave but did not actually go down hard. Existing connections were left intact but all write operations would fail:
Redis::CommandError
READONLY You can't write against a read only slave.
If the connection were to automatically disconnect upon this type of error, the subsequent connection would reconnect to the correct node.
One, potentially flexible solution would be to allow a callback upon certain kinds of errors so that the client code (in this case it is Sidekiq, but it could also have been our main app code) could gracefully handle the problem.
This also might be specific to how AWS has implemented clustering with ElastiCache – I don’t know enough about their implementation to comment on that.
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Reactions: 9
- Comments: 18 (2 by maintainers)
Commits related to this issue
- Support ElastiCache master failover https://github.com/redis/redis-rb/issues/543 — committed to DianthuDia/redis_master_slave by deleted user 8 years ago
- Support ElastiCache master failover reconnect at once by Redis::ConnectionError https://github.com/redis/redis-rb/blob/master/lib/redis/client.rb#L355-L362 (reconnect_attempts = 1) Ref: https://gith... — committed to DianthuDia/redis_master_slave by deleted user 8 years ago
- Support ElastiCache master failover reconnect at once by Redis::ConnectionError https://github.com/redis/redis-rb/blob/master/lib/redis/client.rb#L355-L362 (reconnect_attempts = 1) Ref: https://gith... — committed to DianthuDia/redis_master_slave by deleted user 8 years ago
- Support ElastiCache master failover reconnect at once by Redis::ConnectionError https://github.com/redis/redis-rb/blob/master/lib/redis/client.rb#L355-L362 (reconnect_attempts = 1) Redis::CommandErr... — committed to DianthuDia/redis_master_slave by deleted user 8 years ago
- Reconnect on READONLY errors Fix: https://github.com/redis/redis-rb/issues/543 Fix: https://github.com/redis-rb/redis-client/issues/64 READONLY errors generally happens during failover scenarios and... — committed to redis-rb/redis-client by byroot 2 years ago
- Reconnect on READONLY errors Fix: https://github.com/redis/redis-rb/issues/543 Fix: https://github.com/redis-rb/redis-client/issues/64 READONLY errors generally happens during failover scenarios and... — committed to redis-rb/redis-client by byroot 2 years ago
Is the following parameter related? It seems that we can handle the issue if we use version
2.8.23or later.https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/ParameterGroups.Redis.html
Also, the above parameter is renamed
close-on-replica-writesince5.0.0.https://github.com/redis/redis-rb#reconnections
https://github.com/redis/redis-rb/blob/9fd381b18da20248a111cd9551354907c772a00a/lib/redis/client.rb#L365-L401
https://github.com/redis/redis-rb/blob/9fd381b18da20248a111cd9551354907c772a00a/lib/redis/errors.rb#L23-L41
Hi,
Had an issue with my redis instances that I had a failover and I was connected to replica and got “READONLY” errors (since master become slave) and needed to reconnect to the new master.
We had the same issue with our nodejs client and we did the exact-thing wrote here: https://github.com/luin/ioredis#reconnect-on-error - when we get “READONLY” error we do reconnect.
Is it possible to consider such feature for redis-rb?
This gem implements a patch for elasticache https://github.com/craigmcnamara/redis-elasticache