spring-cloud-stream-binder-aws-kinesis: consumer stops consuming events and throws KinesisMessageDrivenChannelAdapter : The lock for key 'xxxxxxxx:shardId-00000000000X' was not renewed in time

Hi,

I’ve got interesting use case and it’s very simple to replicate. I am using the latest binder version.

My setup:

  • One Kinesis stream with two shards.
  • One consumer app with 3 instances let’s call them instance1, instance2, instance3.
  • Latest binder version.

To replicate:

  1. Start the consumer: usually one instance gets all both shards as owner. Let’s say instance1 gets all the shards. So far so good. Instance consuming events.

  2. Restart the active consuming instance e.g. instance1. Instance2 will get now shard-0 and instance3 gets shard-1 as owner. (This step might take couple of attempts as sometimes instance2 or instance3 gets all the shards. Keep trying until you get one shards per instance)

  3. Once you got instance1 - doing nothing; instance2 owner of shard-0 and instance-3 owner of shard-1, that’s when the instances stop consuming any events and both of them have this in the logs every 30 seconds:


.i.k.KinesisMessageDrivenChannelAdapter : The lock for key 'xxxxxxxx:shardId-000000000002' was not renewed in time java.util.concurrent.TimeoutException: null 	at 
java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960) ~[na:na]
org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ShardConsumer.renewLockIfAny(Kin
esisMessageDrivenChannelAdapter.java:1030) ~[spring-integration-aws-2.4.0.jar:na] 
org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ShardConsumer.execute(KinesisMes
sageDrivenChannelAdapter.java:946) ~[spring-integration-aws-2.4.0.jar:na] 
org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ConsumerDispatcher.run(KinesisMessageDrivenChannelAdapter.java:856) ~[spring-integration-aws-2.4.0.jar:na 

The only way to stop this is to restart one of the consuming instances so all the shards are back to one instance.

The binder version 2.0.1.RELEASE doesn’t have this issue. Pretty sure it’s not even the binder issue - it’s probably integrations-aws lib issue.

Thanks.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 43 (26 by maintainers)

Commits related to this issue

Most upvoted comments

So, yeah… My assumption and investigation is correct. The tryLock() contraction must be honored in the DynamoDbLockRegistry. Otherwise the AmazonDynamoDBLockClient steps into a busy-wait loop and never fulfill a Future for the lock.

I’m going to push the fix today, but if you need a release for Spring Integration AWS 2.5.2, then I can do that tomorrow.

I’m moving this to Spring Integration AWS project though…

Hi @malvine !

Thank you for the sample. I definitely could reproduce it locally with my Docker. It’s impossible to debug it, but at least I see where we are heading. So, I guess we can explain the problem like this: As long as consumer doesn’t take all the shards ownership from the stream, it doesn’t consume even from a shard it owns, neither update the lock for the shard in DynamoDb. I mean it looks like lock manager is broken for this partial situation somehow…

Will look into code today.

Please, try now with -U for Maven or --refresh-dependencies for Gradle: the fresh Spring Integration AWS SNAPSHOT is already there.

Thank you for assisting in testing!

I guess you have to remove your SpringIntegrationLockRegistry table from DynamoDB and let the framework to create a fresh one for you. As I said before: there is no that sortKey any more involved in a new DynamoDbLockRepository model. The Spring Integration AWS must be 3.0.0-SNAPSHOT or just managed transitively by the spring-cloud-stream-binder-kinesis dependency.

So, here is a Kinesis Binder 4.0.0-SNAPSHOT for your consideration: https://repo.spring.io/snapshot/org/springframework/cloud/spring-cloud-stream-binder-kinesis/4.0.0-SNAPSHOT. Note: if you use custom values for partitionKey, sortKeyName, sortKey and heartbeatPeriod, they are not valid any more. See DynamoDbLockRepository Javadocs for a new table structure.

Yeah… i think we are facing this one: https://github.com/awslabs/amazon-dynamodb-lock-client/pull/79. Kinda there is no info in the lock DynamoDB record that it was not used for a while.

It’s probably worth to try with much shorter leaseDuration.

Found the problem. When we have several instances competing for shards we stuck on the if (lock.tryLock()) { exactly for leaseDuration on the lock which does not belong to us. This is not OK and must be fixed this or that way since lock.tryLock() contract must return immediately if no way to lock at that moment. I think there is something wrong in the AmazonDynamoDBLockClient by itself, but I’ll see how that can be mitigated on our side.

Thank you for the patience!

@postalservice14 ,

any chances that you can upgrade your solution (not that sample app) to the latest Spring Boot and org.springframework.cloud:spring-cloud-stream-binder-kinesis:3.0.0 ? There is no @EnableBinding and friends any more in Spring Cloud Stream: everything has to be mapped via java.util.function primitives.

I’ll try to reproduce over weekend in a simple unit test with Testcontainers. I also don’t see a reason in a producer side for that sample: according to the issue description we got a problem just with shards assignments - no need in any data!

Well, I’d prefer to have a new GH issue with fresh, actual and relevant info instead of trying to resurrect this issue against out-dated versions. Thanks for understanding!