redisson: Unable to acquire subscription lock
During relatively high load (~150-250 requests per minute), I often see this error
Caused by: org.redisson.client.RedisTimeoutException: Unable to acquire subscription lock after 3000ms. Increase 'subscriptionsPerConnection' and/or 'subscriptionConnectionPoolSize' parameters.
at org.redisson.pubsub.PublishSubscribe.lambda$subscribe$2(PublishSubscribe.java:69)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:715)
at io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:703)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:790)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:503)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
I doubled the subscriptionsPerConnection
from 5 to 10, the subscriptionConnectionPoolSize
from 50 to 100, and the connectionPoolSize
from 64 to 128, but it doesn’t seem to have helped.
How can I go about diagnosing/debugging this? Do I need to just keep increasing these parameters till it works?
Redis version
6.x
Redisson version
3.16.8
Redisson configuration
config.useSingleServer().apply {
path.toIntOrNull()?.also {
database = it
} ?: run {
logger.warn { "Unable to parse database number: $path from redis configuration" }
}
subscriptionsPerConnection = 10
subscriptionConnectionPoolSize = 100
connectionPoolSize = 128
timeout = 6000
}
config.nettyThreads = 64
I have also tried using Redisson versions 12.0, 13.0, 14.0, 15.0, 16.0 and 17.0 without luck.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 35 (17 by maintainers)
Commits related to this issue
- Fixed - RedissonLock.tryLock method throws CancellationException. #4210 Fixed - subscription timeout calculation. #4216 — committed to redisson/redisson by deleted user 2 years ago
I am experiencing this issue. I’m on version
3.17.7
We have been experiencing this issue for years. I had raised a similar thread a while ago. It started happening to us again recently so I decided to search again and found this thread.
For us the indication that something was very obviously broken on the client side is that the operation would timeout immediately despite claiming that the timeout was many seconds.
We had very similar issues where seemingly only some of our kubernetes pods would experience this issue. It would remain throughout reboots.
So I took this opportunity to try upgrading to 3.17.5 and it appears to have solved the problem!