lettuce-core: RedisCommandTimeoutException: Command timed out after 1 second(s)
Bug Report
Current Behavior
Stack trace
2020-07-23 22:29:50,920 [http-nio2-8080-exec-128] ERROR com.paytm.pgplus.redis.RedisBridgeWorker.get():94 {REQUEST_ID=1595523588463} - Exception while reading from redis cluster:
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 second(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy211.get(Unknown Source)
Input Code
Input Code
// Code to configure redis client and establish a connection with redis:
private final SerializedObjectCodec serializedObjectCodec = new SerializedObjectCodec();
private static Long defaultExpireTimeInSecond;
private RedisClusterClient clusterClient = null;
private StatefulRedisClusterConnection<String, Object> clusterConnection;
@PostConstruct
private void connectToCluster() {
RedisURI[] nodes = redisClusterClientUtil.getRedisURINodes(addresses);
clusterClient = RedisClusterClient.create(Arrays.asList(nodes));
setClusterClientOptions(clusterClient);
clusterConnection = clusterClient.connect(serializedObjectCodec);
}
@PreDestroy
public void closeConnection() {
LOGGER.info("closing cluster connection");
clusterConnection.close();
clusterClient.shutdown();
}
// Code to fetch connection at the time of operation calling(get, set,etc):
private StatefulRedisClusterConnection<String, Object> getConnection() {
try {
if (clusterConnection == null) {
synchronized (RedisClusterClientLettuceService.class) {
if (clusterConnection == null) {
LOGGER.info("creating connection");
clusterConnection = clusterClient.connect(serializedObjectCodec);
}
}
}
return clusterConnection;
} catch (Exception e) {
LOGGER.error("Failed to get connection!");
}
throw new RuntimeException("Failed to get connection");
}
Properties used to configure lettuce:
redis.cluster.topologyRefreshOptions.enable=true redis.cluster.cancelCommandsOnReconnectFailure=true redis.cluster.enablePeriodicRefresh=true redis.cluster.periodicRefreshPeriod.seconds=30(in seconds) redis.cluster.enableAllAdaptiveRefreshTriggers=true redis.cluster.timeout.millis=1000(in ms) redis.cluster.refreshTriggerReconnectAttempts=5
Expected behavior/code
Environment
- Lettuce version(s): 5.2.2.RELEASE
- Redis version: 5.0.7
- We have a config of 5 Master- 5 slave configuration
Possible Solution
Additional context
-
This issue is not consistent across all the application servers, the issue is intermittent and due to this a CPU spike is also observed.
-
As soon as we boot up the server in production and the high load traffic hits the server, we start getting this issue
-
Also, we have recently started facing this issue and the system was up from last 3 months, and never faced this issue
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 20 (6 by maintainers)
Thanks @kiteplayer your suggestion about the switching of the spring-cloud dependencies didn’t work. I was on Hoxton.SR8 and I switched to SR6. However, your other suggestion of switching to
jedis
worked out well for me. In which case I did thisI am still wondering why on one project the default which is
lettuce
worked but on another it didn’t. I don’t see anything particularly different between the two projects in terms of redis. Also I am using Redis in a container environment and it is isolated so it’s an empty redis on both when it starts.I got this problem on our Aliyun Redis (might be same on AWS)
The CORE CAUSE is that the connection is reset by sever, which WITHOUT sending FIN to lettuce side. The connection will be recover after received FIN by linux (TCP_RETRIES2, default 15min)
My solution is add IdleStateHandler(from netty) to monitor read-idles and force reconnection
code example:
adding an error counter or Circuitbreaker to stats command timeouts and trigger reconnect is also a good idea
if use spring-boot.2.3.x:
Dear friend, I am kiteplayer,Very glad to hear from you. I am not sure whether this problem is solved completely,because I use the SpringBoot framework integrate springcloud and springcloud Aliabba, when I change the version of springcloud and springcloud Aliabba,this issue disappeared。But i still don’t know the root cause,I am sorry。 This follows are my program version,FYI. or you also try to use jedis replace lettuce. <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.3.0.RELEASE</version> <relativePath/> </parent>
<dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-dependencies</artifactId> <version>Hoxton.SR6</version> <type>pom</type> <scope>import</scope> </dependency>
</dependencyManagement>
------------------ 原始邮件 ------------------ 发件人: “lettuce-io/lettuce-core” <notifications@github.com>; 发送时间: 2020年9月26日(星期六) 下午4:47 收件人: “lettuce-io/lettuce-core”<lettuce-core@noreply.github.com>; 抄送: “青山流水”<598353271@qq.com>;“Mention”<mention@noreply.github.com>; 主题: Re: [lettuce-io/lettuce-core] RedisCommandTimeoutException: Command timed out after 1 second(s) (#1362)
hello @Srthak100 May I ask How did you solve this problem in the end? I get the same situation in the env Lettuce version(s): 5.2.2.RELEASE Redis version: 5.0.7
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.