spring-cloud-stream-binder-aws-kinesis: Kinesis binder locks timed out and messages not delivered
We use Spring Cloud stream with Kinesis, sometimes we see that messages are not delivering. After investigation, we found that the locks are not processing correctly, and we have messages about it in the logs.
To Reproduce Steps to reproduce the behavior:
- Clone the repository https://github.com/RomanAbakumov/kinesis-lock-issues-demo
- run localstack using startLocalstack.cmd/startLocalstack.sh
- start application using mvnw spring-boot:run
- Wait for app to create a streams and tables and start producing and receiving messages, wait for about 5 minutes you should see messages like: Message sent: Message: 3 Received message: Message: 3 Message sent: Message: 4
- Start 5 other instances using mvnw spring-boot:run At this point only first instance (started on step 3) receiving all the messages and allocate the locks
- Stop application started on step 3 by using kill -TERM or closing the window.
Version of the framework org.springframework.cloud:spring-cloud-dependencies:2022.0.1 spring-cloud-stream-binder-kinesis: 3.0.0 Expected behavior Locks allocated by running nodes and messages are received Observed behavior Messages lost, delayed. Timeouts in the logs: `2023-04-27T12:25:37.735-04:00 INFO 31356 — [is-dispatcher-1] a.i.k.KinesisMessageDrivenChannelAdapter : The lock for key ‘event-group:stream-v0:shardId-000000000009’ was not renewed in time
java.util.concurrent.TimeoutException: null at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960) ~[na:na] at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095) ~[na:na] at org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ShardConsumer.renewLockIfAny(KinesisMessageDrivenChannelAdapter.java:1035) ~[spring-integration-aws-2.5.4.jar:na] at org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ShardConsumer.execute(KinesisMessageDrivenChannelAdapter.java:947) ~[spring-integration-aws-2.5.4.jar:na] at org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ConsumerDispatcher.run(KinesisMessageDrivenChannelAdapter.java:857) ~[spring-integration-aws-2.5.4.jar:na] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na] ` Additional context The issue is likely happening when shard locks is distributed between multiple nodes, it is not observed when all locks are allocated by one instance. We running 2 pods listening stream with 4 shards, sometimes messages are not delivered, and it may be remediated by clearing the locks table. When locks are allocated to different pods timeout messages like the above appear in the logs.
The behavior is not consistent, sometimes it works as it should, and sometimes message delivery is stuck, but using a linked project it is reproducible almost all the time.
You may check lock distribution by configuring test credentials in .aws/config like: aws_access_key_id=test aws_secret_access_key=test and executing getLockTableContent.sh/getLockTableContent.cmd
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 17 (6 by maintainers)
Got it! Please, raise a new GH issue in the https://github.com/spring-projects/spring-integration-aws project with more info.
It looks like we we don’t catch this kind of errors in the
KinesisMessageDrivenChannelAdapter.While it works as it should, with no errors, and shard consumers able to allocate unused nodes I faced another issue that may need attention. My laptop runs localstack and 3 app instances, I put it to sleep and when I wake it up I see that one app instance reports errors, and it can’t recover of it for more than 30 minutes.
2023-05-01T14:54:10.459-04:00 INFO 16112 — [esis-consumer-1] a.i.k.KinesisMessageDrivenChannelAdapter : Got an exception java.util.concurrent.CompletionException: software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException: The shard iterator has expired. Shard iterators are only valid for 300 seconds (Service: Kinesis, Status Code: 400, Request ID: AKSSPO8S0D2O3C4WWIF4QKMEL8VJSL2NANQN52N899PIFJFO7I09) during [ShardConsumer{shardOffset=KinesisShardOffset{iteratorType=AFTER_SEQUENCE_NUMBER, sequenceNumber=‘49640352981598798030864787941543021567332992185863766098’, timestamp=null, stream=‘stream-v0’, shard=‘shardId-000000000005’, reset=false}, state=CONSUME}] task invocation. Process will be retried on the next iteration.
In May. Or better to say when Spring Cloud AWS 3.0 is GA.