redisson: Can't Find Node in Slaves in Version 3.16.5
Expected behavior Have 1 master and 1 slave (with AWS ElastiCache). When testing failover by through modifying of the node type of the redis instance, expect after failover that there are no errors when querying the new master/slave.
Actual behavior
Getting the following error message:
Can't find replicaAddr/<host> in slaves! Available slaves: [[addr=master1Addr], [addr=master1Addr], [addr=master1Addr], [addr=replicaAddr]]
Trace:
at java.lang.Thread.getStackTrace(Thread.java:1596)
at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
at ch.qos.logback.classic.Logger.filterAndLog_2(Logger.java:414)
at ch.qos.logback.classic.Logger.error(Logger.java:530)
at org.redisson.connection.balancer.LoadBalancerManager.unfreeze(LoadBalancerManager.java:127)
at org.redisson.connection.MasterSlaveEntry.slaveUp(MasterSlaveEntry.java:424)
at org.redisson.connection.ReplicatedConnectionManager.slaveUp(ReplicatedConnectionManager.java:206)
at org.redisson.connection.ReplicatedConnectionManager.lambda$checkNode$5(ReplicatedConnectionManager.java:195)
at org.redisson.misc.RedissonPromise.lambda$onComplete$0(RedissonPromise.java:187)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)
at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
at org.redisson.misc.RedissonPromise.trySuccess(RedissonPromise.java:82)
at org.redisson.client.handler.CommandDecoder.completeResponse(CommandDecoder.java:460)
at org.redisson.client.handler.CommandDecoder.handleResult(CommandDecoder.java:454)
at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:385)
at org.redisson.client.handler.CommandDecoder.decodeCommand(CommandDecoder.java:198)
at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:137)
at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:113)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:831)
Steps to reproduce or test case
- Deploy Service
- Modify node type of redis instance
- View application logs and see errors
Redis version 6.0.5
Redisson version 3.16.5
Redisson configuration
redissonConfig
.useReplicatedServers()
.setTimeout(5000)
.setNodeAddresses(addressList);
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 23 (11 by maintainers)
Commits related to this issue
- Fixed - "can't find node" error arise in replicated mode. #4005 — committed to redisson/redisson by deleted user 3 years ago
@mrniko thanks! Is it possible to get it is out sooner? We are planning a full migration over to redisson but need fail overs to be handled before doing so. If not, no worries. Thanks again.
Seems to be fixed. Thanks @mrniko!
@mrniko the failover itself happens without downtime. Seems like the existing connection is dead but isn’t cleaned up by redisson
@mrniko , just in case, can I test an updated jar with the fix to verify on our side. Thanks!