camellia: [Camellia-redis-proxy] Redis cluster: CROSSSLOT Keys in request don't hash to the same slot

Bug: [Camellia-redis-proxy] Redis cluster: CROSSSLOT Keys in request don’t hash to the same slot

1. Issue

• I try to use the Redisson to connect to the camellia redis proxy and test LockExamples, but always get an error:

[main] INFO org.redisson.Version - Redisson 3.10.6
[redisson-netty-1-26] INFO org.redisson.connection.pool.MasterPubSubConnectionPool - 1 connections initialized for 10.50.69.8/10.50.69.8:6381
[redisson-netty-1-3] INFO org.redisson.connection.pool.MasterConnectionPool - 32 connections initialized for 10.50.69.8/10.50.69.8:6381
>>>>>> start thread
Exception in thread "Thread-0" org.redisson.client.RedisException: CROSSSLOT Keys in request don't hash to the same slot. channel: [id: 0x0559429c, L:/10.79.xx.xx:51149 - R:10.50.xx.xx/10.50.xx.xx:6381] command: (EVAL), params: [if (redis.call('hexists', KEYS[1], ARGV[3]) == 0) then return nil;end; local counter = redis.call('h..., 2, lock, redisson_lock__channel:{lock}, 0, 30000, f46f747b-a31b-4d38-bb7a-9acb9b96a4e4:46]
	at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:351)
	at org.redisson.client.handler.CommandDecoder.decodeCommand(CommandDecoder.java:199)
	at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:139)
	at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:114)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
	at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:677)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)

2. Setup/Configurations

• Camellia version 1.1.9
• Redis resource is a cluster redis
• (*) Using converterPlugin with implementation is DefaultTenancyNamespaceKeyConverter converter.key.className=com.netease.nim.camellia.redis.proxy.plugin.converter.DefaultTenancyNamespaceKeyConverter
• Use this example: redisson/LockExamples.java
• My test class:

public class LockExamples {

    public static void main(String[] args) throws InterruptedException {
        Config config = new Config();

        // rediss - defines to use SSL for Redis connection
        config.useSingleServer()
                .setAddress("redis://127.0.0.1:6380") // this is camellia-redis-proxy
                .setClientName("camellia_13_tass") // tenant to a cluster redis
                .setTimeout(100000);
        RedissonClient redisson = Redisson.create(config);

        RLock lock = redisson.getLock("lock");
        lock.lock(2, TimeUnit.SECONDS);

        Thread t = new Thread(() -> {
            System.out.println(">>>>>> start thread");
            RLock lock1 = redisson.getLock("lock");
            lock1.lock();
            lock1.unlock();
            System.out.println(">>>>>> done");
        });

        t.start();
        t.join();

        redisson.shutdown();
    }

}

3. Trace/Debug

• The redisson will put 2 keys into redis: lock and redisson_lock__channel:{lock}
• If correct, both of those keys should be in the same slot, but in reality, they are not. Leads to exception: org.redisson.client.RedisException: CROSSSLOT Keys in request don't hash to the same slot.
• I debug the camellia-redis-proxy and discover that:
  • after the convert key plugin: 2 keys above will be:
    • lock -> 13|tass|lock
    • and redisson_lock__channel:{lock} -> 13|tass|redisson_lock__channel:{lock}
  • this leads to the above 2 keys no longer are the same slot
• At this line, after converting key, 2 keys are not the same slot -> the result of int slot = RedisClusterCRC16Utils.checkSlot(keys); is -1 -> throw exception here: https://github.com/netease-im/camellia/blob/814316b0ca1c7f34d240e258e8f0ae2d92f4b46a/camellia-redis-proxy/src/main/java/com/netease/nim/camellia/redis/proxy/upstream/cluster/AsyncCamelliaRedisClusterClient.java#L212-L218

4. Solutions

• I’m not sure, but maybe the order of plugins is the problem pls help me check this issue, @caojiajun

thank you.