spring-cloud-stream-binder-aws-kinesis: consumer stops consuming events and throws KinesisMessageDrivenChannelAdapter : The lock for key 'xxxxxxxx:shardId-00000000000X' was not renewed in time
Hi,
I’ve got interesting use case and it’s very simple to replicate. I am using the latest binder version.
My setup:
- One Kinesis stream with two shards.
- One consumer app with 3 instances let’s call them instance1, instance2, instance3.
- Latest binder version.
To replicate:
-
Start the consumer: usually one instance gets all both shards as owner. Let’s say instance1 gets all the shards. So far so good. Instance consuming events.
-
Restart the active consuming instance e.g. instance1. Instance2 will get now shard-0 and instance3 gets shard-1 as owner. (This step might take couple of attempts as sometimes instance2 or instance3 gets all the shards. Keep trying until you get one shards per instance)
-
Once you got instance1 - doing nothing; instance2 owner of shard-0 and instance-3 owner of shard-1, that’s when the instances stop consuming any events and both of them have this in the logs every 30 seconds:
.i.k.KinesisMessageDrivenChannelAdapter : The lock for key 'xxxxxxxx:shardId-000000000002' was not renewed in time java.util.concurrent.TimeoutException: null at
java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960) ~[na:na]
org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ShardConsumer.renewLockIfAny(Kin
esisMessageDrivenChannelAdapter.java:1030) ~[spring-integration-aws-2.4.0.jar:na]
org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ShardConsumer.execute(KinesisMes
sageDrivenChannelAdapter.java:946) ~[spring-integration-aws-2.4.0.jar:na]
org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ConsumerDispatcher.run(KinesisMessageDrivenChannelAdapter.java:856) ~[spring-integration-aws-2.4.0.jar:na
The only way to stop this is to restart one of the consuming instances so all the shards are back to one instance.
The binder version 2.0.1.RELEASE doesn’t have this issue. Pretty sure it’s not even the binder issue - it’s probably integrations-aws lib issue.
Thanks.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 43 (26 by maintainers)
Commits related to this issue
- Fix DynamoDbLockRegistry for additionalTimeToWait Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis/issues/186 The `DynamoDbLockClient` waits extra `leaseDuration` time in... — committed to spring-projects/spring-integration-aws by artembilan 4 years ago
- Fix DynamoDbLockRegistry for additionalTimeToWait Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis/issues/186 The `DynamoDbLockClient` waits extra `leaseDuration` time in... — committed to Griffin1989106/SpringWithAWS by Griffin1989106 4 years ago
- Fix DynamoDbLockRegistry for additionalTimeToWait Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis/issues/186 The `DynamoDbLockClient` waits extra `leaseDuration` time in... — committed to guygriffin1989106/SpringWithAWS by guygriffin1989106 4 years ago
So, yeah… My assumption and investigation is correct. The
tryLock()contraction must be honored in theDynamoDbLockRegistry. Otherwise theAmazonDynamoDBLockClientsteps into a busy-wait loop and never fulfill aFuturefor the lock.I’m going to push the fix today, but if you need a release for Spring Integration AWS
2.5.2, then I can do that tomorrow.I’m moving this to Spring Integration AWS project though…
Hi @malvine !
Thank you for the sample. I definitely could reproduce it locally with my Docker. It’s impossible to debug it, but at least I see where we are heading. So, I guess we can explain the problem like this: As long as consumer doesn’t take all the shards ownership from the stream, it doesn’t consume even from a shard it owns, neither update the lock for the shard in DynamoDb. I mean it looks like lock manager is broken for this partial situation somehow…
Will look into code today.
Please, try now with
-Ufor Maven or--refresh-dependenciesfor Gradle: the fresh Spring Integration AWS SNAPSHOT is already there.Thank you for assisting in testing!
I guess you have to remove your
SpringIntegrationLockRegistrytable from DynamoDB and let the framework to create a fresh one for you. As I said before: there is no thatsortKeyany more involved in a newDynamoDbLockRepositorymodel. The Spring Integration AWS must be3.0.0-SNAPSHOTor just managed transitively by thespring-cloud-stream-binder-kinesisdependency.So, here is a Kinesis Binder
4.0.0-SNAPSHOTfor your consideration: https://repo.spring.io/snapshot/org/springframework/cloud/spring-cloud-stream-binder-kinesis/4.0.0-SNAPSHOT. Note: if you use custom values forpartitionKey,sortKeyName,sortKeyandheartbeatPeriod, they are not valid any more. SeeDynamoDbLockRepositoryJavadocs for a new table structure.Yeah… i think we are facing this one: https://github.com/awslabs/amazon-dynamodb-lock-client/pull/79. Kinda there is no info in the lock DynamoDB record that it was not used for a while.
It’s probably worth to try with much shorter
leaseDuration.@RomanAbakumov ,
https://repo.spring.io/snapshot
Essentially here: https://repo.spring.io/snapshot/org/springframework/integration/spring-integration-aws/2.5.5-SNAPSHOT/
Found the problem. When we have several instances competing for shards we stuck on the
if (lock.tryLock()) {exactly forleaseDurationon the lock which does not belong to us. This is not OK and must be fixed this or that way sincelock.tryLock()contract must return immediately if no way to lock at that moment. I think there is something wrong in theAmazonDynamoDBLockClientby itself, but I’ll see how that can be mitigated on our side.Thank you for the patience!
@postalservice14 ,
any chances that you can upgrade your solution (not that sample app) to the latest Spring Boot and
org.springframework.cloud:spring-cloud-stream-binder-kinesis:3.0.0? There is no@EnableBindingand friends any more in Spring Cloud Stream: everything has to be mapped viajava.util.functionprimitives.I’ll try to reproduce over weekend in a simple unit test with Testcontainers. I also don’t see a reason in a producer side for that sample: according to the issue description we got a problem just with shards assignments - no need in any data!
Well, I’d prefer to have a new GH issue with fresh, actual and relevant info instead of trying to resurrect this issue against out-dated versions. Thanks for understanding!