lettuce-core: RedisCommandTimeoutException: Command timed out after 1 second(s)

Bug Report

Current Behavior

Stack trace
2020-07-23 22:29:50,920 [http-nio2-8080-exec-128] ERROR com.paytm.pgplus.redis.RedisBridgeWorker.get():94 {REQUEST_ID=1595523588463} - Exception while reading from redis cluster: 
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 second(s)
	at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
	at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
	at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
	at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
	at com.sun.proxy.$Proxy211.get(Unknown Source)

Input Code

Input Code
// Code to configure redis client and establish a connection with redis:

private final SerializedObjectCodec serializedObjectCodec = new SerializedObjectCodec();
    private static Long defaultExpireTimeInSecond;
    private RedisClusterClient clusterClient = null;
    private StatefulRedisClusterConnection<String, Object> clusterConnection;

@PostConstruct
    private void connectToCluster() {

        RedisURI[] nodes = redisClusterClientUtil.getRedisURINodes(addresses);
        clusterClient = RedisClusterClient.create(Arrays.asList(nodes));
        setClusterClientOptions(clusterClient);

        clusterConnection = clusterClient.connect(serializedObjectCodec);
    }

    @PreDestroy
    public void closeConnection() {
        LOGGER.info("closing cluster connection");
        clusterConnection.close();
        clusterClient.shutdown();
    }

// Code to fetch connection at the time of operation calling(get, set,etc):

private StatefulRedisClusterConnection<String, Object> getConnection() {
        try {
            if (clusterConnection == null) {
                synchronized (RedisClusterClientLettuceService.class) {
                    if (clusterConnection == null) {
                        LOGGER.info("creating connection");
                        clusterConnection = clusterClient.connect(serializedObjectCodec);
                    }
                }
            }
            return clusterConnection;
        } catch (Exception e) {
            LOGGER.error("Failed to get connection!");
        }
        throw new RuntimeException("Failed to get connection");
    }

Properties used to configure lettuce:

redis.cluster.topologyRefreshOptions.enable=true redis.cluster.cancelCommandsOnReconnectFailure=true redis.cluster.enablePeriodicRefresh=true redis.cluster.periodicRefreshPeriod.seconds=30(in seconds) redis.cluster.enableAllAdaptiveRefreshTriggers=true redis.cluster.timeout.millis=1000(in ms) redis.cluster.refreshTriggerReconnectAttempts=5

Expected behavior/code

Environment

  • Lettuce version(s): 5.2.2.RELEASE
  • Redis version: 5.0.7
  • We have a config of 5 Master- 5 slave configuration

Possible Solution

Additional context

  • This issue is not consistent across all the application servers, the issue is intermittent and due to this a CPU spike is also observed.

  • As soon as we boot up the server in production and the high load traffic hits the server, we start getting this issue

  • Also, we have recently started facing this issue and the system was up from last 3 months, and never faced this issue

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 20 (6 by maintainers)

Most upvoted comments

Thanks @kiteplayer your suggestion about the switching of the spring-cloud dependencies didn’t work. I was on Hoxton.SR8 and I switched to SR6. However, your other suggestion of switching to jedis worked out well for me. In which case I did this

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-data-redis</artifactId>
      <exclusions>
        <exclusion>
          <groupId>io.lettuce</groupId>
          <artifactId>lettuce-core</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>redis.clients</groupId>
      <artifactId>jedis</artifactId>
      <scope>runtime</scope>
    </dependency>

I am still wondering why on one project the default which is lettuce worked but on another it didn’t. I don’t see anything particularly different between the two projects in terms of redis. Also I am using Redis in a container environment and it is isolated so it’s an empty redis on both when it starts.

Has anyone reached a solution? Or just switch to jedis? I tried with the latest versions, spring-boot-starter-data-redis-reactive 2.5.4 and I get the same exception

edit: I found the problem for my issue I was using redisTemplate.opsForList().rightPopAndLeftPush() but there wasn’t any item in Redis at my source and the request resulted in timeout.

I got this problem on our Aliyun Redis (might be same on AWS)

The CORE CAUSE is that the connection is reset by sever, which WITHOUT sending FIN to lettuce side. The connection will be recover after received FIN by linux (TCP_RETRIES2, default 15min)

My solution is add IdleStateHandler(from netty) to monitor read-idles and force reconnection

code example:

NettyCustomizer nettyCustomizer = new NettyCustomizer() {
        
        @Override
        public void afterChannelInitialized(Channel channel) {
            channel.pipeline().addLast(new IdleStateHandler(readerIdleTime, writerIdleTime, allIdleTime));
            channel.pipeline().addLast(new ChannelDuplexHandler() {
                @Override
                public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception {
                    if (evt instanceof IdleStateEvent) {
                        ctx.disconnect();
                    }
                }
            });
        }        
    };

return ClientResources.builder().nettyCustomizer(nettyCustomizer ).build();

adding an error counter or Circuitbreaker to stats command timeouts and trigger reconnect is also a good idea

if use spring-boot.2.3.x:

  lettuce:
      cluster:
        refresh:
          adaptive: true
          #20秒自动刷新一次
          period: 20

Dear friend,     I am kiteplayer,Very glad to hear from you. I am not sure whether this problem is  solved completely,because I use the SpringBoot framework integrate springcloud and springcloud Aliabba, when I change the version of springcloud and springcloud Aliabba,this issue disappeared。But i still don’t know the root cause,I am sorry。 This follows are my program version,FYI. or you also try to use jedis replace lettuce. <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.3.0.RELEASE</version> <relativePath/> </parent>

<dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-dependencies</artifactId> <version>Hoxton.SR6</version> <type>pom</type> <scope>import</scope> </dependency>

    <dependency&gt;
        <groupId&gt;com.alibaba.cloud</groupId&gt;
        <artifactId&gt;spring-cloud-alibaba-dependencies</artifactId&gt;
        <version&gt;2.1.0.RELEASE</version&gt;
        <type&gt;pom</type&gt;
        <scope&gt;import</scope&gt;
    </dependency&gt;
</dependencies&gt;

</dependencyManagement>

------------------ 原始邮件 ------------------ 发件人: “lettuce-io/lettuce-core” <notifications@github.com>; 发送时间: 2020年9月26日(星期六) 下午4:47 收件人: “lettuce-io/lettuce-core”<lettuce-core@noreply.github.com>; 抄送: “青山流水”<598353271@qq.com>;“Mention”<mention@noreply.github.com>; 主题: Re: [lettuce-io/lettuce-core] RedisCommandTimeoutException: Command timed out after 1 second(s) (#1362)

hello @Srthak100 May I ask How did you solve this problem in the end? I get the same situation in the env Lettuce version(s): 5.2.2.RELEASE Redis version: 5.0.7

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.