spring-cloud-stream-binder-aws-kinesis: not able to bind consumer group with particular shard of kinesis stream
I am defining consumer group and shard id like below in application.yml
spring:
cloud:
stream:
bindings:
ordersIn:
destination: testhk
group: shipping-group
shardId: shardId-000000000001
content-type: application/json
But when I am starting consumer it’s showing both shard ids in shard offset. below are the startup logs
2018-02-14 18:12:35.547 INFO 10024 --- [ main] a.i.k.KinesisMessageDrivenChannelAdapter : started KinesisMessageDrivenChannelAdapter{shardOffsets=[KinesisShardOffset{iteratorType=TRIM_HORIZON, sequenceNumber='null', timestamp=null, stream='testhk', shard='shardId-000000000000', reset=false}, KinesisShardOffset{iteratorType=TRIM_HORIZON, sequenceNumber='null', timestamp=null, stream='testhk', shard='shardId-000000000001', reset=false}], consumerGroup='shipping-group'}
2018-02-14 18:13:04.508 WARN 10024 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=46s178ms985µs770ns).
2018-02-14 18:13:24.911 INFO 10024 --- [esis-consumer-1] a.i.k.KinesisMessageDrivenChannelAdapter : The [ShardConsumer{shardOffset=KinesisShardOffset{iteratorType=TRIM_HORIZON, sequenceNumber='null', timestamp=null, stream='testhk', shard='shardId-000000000000', reset=false}, state=NEW}] has been started.
2018-02-14 18:13:25.022 INFO 10024 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 64399 (http) with context path ''
2018-02-14 18:13:25.027 INFO 10024 --- [ main] demo.KinesisApplication : Started KinesisApplication in 83.839 seconds (JVM running for 85.396)
2018-02-14 18:13:27.057 INFO 10024 --- [esis-consumer-1] a.i.k.KinesisMessageDrivenChannelAdapter : The [ShardConsumer{shardOffset=KinesisShardOffset{iteratorType=TRIM_HORIZON, sequenceNumber='null', timestamp=null, stream='testhk', shard='shardId-000000000001', reset=false}, state=NEW}] has been started.
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Comments: 15 (12 by maintainers)
Commits related to this issue
- GH-35: Bind consumer with a specific shard Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis/issues/35 * Add docs * Add missing javadoc to pass checkstyle violation * Clea... — committed to spring-cloud/spring-cloud-stream-binder-aws-kinesis by dmytrodanilenkov 5 years ago
@artembilan created PR
I don’t think there is something relevant to the provisioner in this case. What we really need is just an option to let end-user to configure a specific shard and use exactly that shard when we create a consumer instance.
See
KinesisMessageChannelBinderand itscreateConsumerEndpoint()impl:and its
createKinesisConsumerEndpoint():So, we do here distribute shards according instance index of your consumer application, but we don’t consult here with some extra property if specific shard is provided. So, my thought is if we would have something like
shardconfiguration property for theKinesisConsumerProperties, we would be able to propagate it down to thatKinesisMessageDrivenChannelAdapterinstance.NOTE: KCL does not expose any option to deal with a specific shard. Therefore a feature we would could implement here would not be compatible with a KCL mode.
The provisioner must do nothing in this case probably. Or could consult with that new property for the fact that requested shard really exists in the stream. As far as I know we cannot assign specific id for the shard we can create in the stream.
Thanks for your interest in contribution!
Sorry, we didn’t have a chance to look into this.
Contribution is welcome though!