beam: [Bug]: KinesisIO source on FlinkRunner initializes the same splits twice
What happened?
Bug description
Setup details:
- FlinkRunner (Flink 1.15.4) (also replicated with Flink 1.18.2)
- KinesisIO (from
beam-sdks-java-io-amazon-web-services2) - Beam version 2.56.0
- Pipeline attached mode (false)
Bug details:
- When restoring from snapshot on Flink, the
org.apache.beam.sdk.io.aws2.kinesis.KinesisReaderis assigned the same splits twice, once with snapshot state, and once without. This leads to duplicate data being processed.
Replication steps:
- Start a Flink job with KinesisIO source.
- Stop the Flink job with a savepoint.
- Start the same Flink job from savepoint.
Logs:
- From Flink Taskmanager (worker node) log dump below, we can see that splits for
shardId-000000000000toshardId-000000000003are first initialized with checkpoint stateAFTER_SEQUENCE_NUMBER(correct). - Following that, we see that they are initialized without checkpoint state with
AT_TIMESTAMP(not correct).
2024-05-16 12:29:43,263 INFO org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.FlinkSourceReaderBase [] - Adding splits [FlinkSourceSplit{splitIndex=0, beamSource=org.apache.beam.sdk.io.aws2.kinesis.KinesisSource@269f1286, splitState.isNull=false, checkpointMark=null}]
2024-05-16 12:29:43,264 WARN org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.unbounded.FlinkUnboundedSourceReader [] - AutoWatermarkInterval is not set, watermarks will be emitted at a default interval of 200 ms
2024-05-16 12:29:43,264 INFO org.apache.flink.runtime.taskmanager.Task [] - Source: KDS Source/Read(KinesisSource) -> Flat Map -> ParDo(Anonymous)/ParMultiDo(Anonymous) -> KDS Sink/ParDo(Anonymous)/ParMultiDo(Anonymous) (1/1)#0 (9e813ffa491b6a4d44e7860742f1576b) switched from INITIALIZING to RUNNING.
2024-05-16 12:29:43,265 WARN org.apache.beam.sdk.coders.SerializableCoder [] - Can't verify serialized elements of type KinesisReaderCheckpoint have well defined equals method. This may produce incorrect results on some PipelineRunner implementations2024-05-16 12:29:43,265 INFO org.apache.beam.sdk.io.aws2.kinesis.KinesisSource [] - Got checkpoint mark [Checkpoint AFTER_SEQUENCE_NUMBER for stream ExampleInputStream,shard shardId-000000000000: 49651326501327272958725198009666143092350196858539212802 0, Checkpoint AFTER_SEQUENCE_NUMBER for stream ExampleInputStream, shard shardId-000000000001: 49651326500635949857570748692274909048439253764329701394 0, Checkpoint AFTER_SEQUENCE_NUMBER for stream ExampleInputStream, shard shardId-000000000002: 49651326503155934065004709107267236287428903985330782242 0, Checkpoint AFTER_SEQUENCE_NUMBER for stream ExampleInputStream, shard shardId-000000000003: 49651326499966927501614829993237864477127027156192329778 0]2024-05-16 12:29:43,266 INFO org.apache.beam.sdk.io.aws2.kinesis.KinesisSource [] - Creating new reader using [Checkpoint AFTER_SEQUENCE_NUMBER for stream ExampleInputStream, shard shardId-000000000000: 49651326501327272958725198009666143092350196858539212802 0, Checkpoint AFTER_SEQUENCE_NUMBER for stream ExampleInputStream, shard shardId-000000000001: 49651326500635949857570748692274909048439253764329701394 0, Checkpoint AFTER_SEQUENCE_NUMBER for stream ExampleInputStream, shard shardId-000000000002: 49651326503155934065004709107267236287428903985330782242 0, Checkpoint AFTER_SEQUENCE_NUMBER for stream ExampleInputStream, shard shardId-000000000003: 49651326499966927501614829993237864477127027156192329778 0]
2024-05-16 12:29:43,268 INFO org.apache.beam.sdk.io.aws2.kinesis.KinesisReader [] - Starting reader using [Checkpoint AFTER_SEQUENCE_NUMBER for stream ExampleInputStream, shard shardId-000000000000: 49651326501327272958725198009666143092350196858539212802 0, Checkpoint AFTER_SEQUENCE_NUMBER for stream ExampleInputStream, shard shardId-000000000001: 49651326500635949857570748692274909048439253764329701394 0, Checkpoint AFTER_SEQUENCE_NUMBER for stream ExampleInputStream, shard shardId-000000000002: 49651326503155934065004709107267236287428903985330782242 0, Checkpoint AFTER_SEQUENCE_NUMBER for stream ExampleInputStream, shard shardId-000000000003: 496513264999669275016148299932378644771270271561923297780]
2024-05-16 12:29:43,272 INFO org.apache.beam.sdk.io.aws2.kinesis.SimplifiedKinesisClient [] - Starting getIterator request GetShardIteratorRequest(StreamName=ExampleInputStream, ShardId=shardId-000000000000, ShardIteratorType=AT_SEQUENCE_NUMBER, StartingSequenceNumber=49651326501327272958725198009666143092350196858539212802)
2024-05-16 12:29:43,795 INFO org.apache.beam.sdk.io.aws2.kinesis.SimplifiedKinesisClient [] - Starting getIterator request GetShardIteratorRequest(StreamName=ExampleInputStream, ShardId=shardId-000000000001, ShardIteratorType=AT_SEQUENCE_NUMBER, StartingSequenceNumber=49651326500635949857570748692274909048439253764329701394)
2024-05-16 12:29:43,891 INFO org.apache.beam.sdk.io.aws2.kinesis.SimplifiedKinesisClient [] - Starting getIterator request GetShardIteratorRequest(StreamName=ExampleInputStream, ShardId=shardId-000000000002, ShardIteratorType=AT_SEQUENCE_NUMBER, StartingSequenceNumber=49651326503155934065004709107267236287428903985330782242)
2024-05-16 12:29:43,983 INFO org.apache.beam.sdk.io.aws2.kinesis.SimplifiedKinesisClient [] - Starting getIterator request GetShardIteratorRequest(StreamName=ExampleInputStream, ShardId=shardId-000000000003, ShardIteratorType=AT_SEQUENCE_NUMBER, StartingSequenceNumber=49651326499966927501614829993237864477127027156192329778)
2024-05-16 12:29:44,080 INFO org.apache.beam.sdk.io.aws2.kinesis.ShardReadersPool [] - Starting to read ExampleInputStream stream from [shardId-000000000000, shardId-000000000001, shardId-000000000002, shardId-000000000003] shards
2024-05-16 12:29:44,266 INFO org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.FlinkSourceReaderBase [] - Adding splits [FlinkSourceSplit{splitIndex=0, beamSource=org.apache.beam.sdk.io.aws2.kinesis.KinesisSource@2a249887, splitState.isNull=true, checkpointMark=null}]
2024-05-16 12:29:44,266 INFO org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.FlinkSourceReaderBase [] - Received NoMoreSplits signal from enumerator.
2024-05-16 12:29:44,898 INFO org.apache.beam.sdk.io.aws2.kinesis.KinesisSource [] - No checkpointMark specified, fall back to initial [Checkpoint AT_TIMESTAMP for stream ExampleInputStream, shard shardId-000000000000: null null, Checkpoint AT_TIMESTAMP for stream ExampleInputStream, shard shardId-000000000001: null null, Checkpoint AT_TIMESTAMP for stream ExampleInputStream, shard shardId-000000000002: null null, Checkpoint AT_TIMESTAMP for stream ExampleInputStream, shard shardId-000000000003: null null]
2024-05-16 12:29:44,898 INFO org.apache.beam.sdk.io.aws2.kinesis.KinesisSource [] - Creating new reader using [Checkpoint AT_TIMESTAMP for stream ExampleInputStream, shard shardId-000000000000: null null, Checkpoint AT_TIMESTAMP for stream ExampleInputStream, shard shardId-000000000001: null null, Checkpoint AT_TIMESTAMP for stream ExampleInputStream, shard shardId-000000000002: null null, Checkpoint AT_TIMESTAMP for stream ExampleInputStream, shard shardId-000000000003: null null]
2024-05-16 12:29:44,899 INFO org.apache.beam.sdk.io.aws2.kinesis.KinesisReader [] - Starting reader using [Checkpoint AT_TIMESTAMP for stream ExampleInputStream, shard shardId-000000000000: null null, Checkpoint AT_TIMESTAMP for stream ExampleInputStream, shard shardId-000000000001: null null, Checkpoint AT_TIMESTAMP for stream ExampleInputStream,shard shardId-000000000002: null null, Checkpoint AT_TIMESTAMP for stream ExampleInputStream, shard shardId-000000000003: null null]
2024-05-16 12:29:44,899 INFO org.apache.beam.sdk.io.aws2.kinesis.SimplifiedKinesisClient [] - Starting getIterator request GetShardIteratorRequest(StreamName=ExampleInputStream, ShardId=shardId-000000000000, ShardIteratorType=AT_TIMESTAMP, Timestamp=2024-05-16T10:59:38.912Z)
2024-05-16 12:29:45,311 INFO org.apache.beam.sdk.io.aws2.kinesis.SimplifiedKinesisClient [] - Starting getIterator request GetShardIteratorRequest(StreamName=ExampleInputStream, ShardId=shardId-000000000001, ShardIteratorType=AT_TIMESTAMP, Timestamp=2024-05-16T10:59:38.912Z)
2024-05-16 12:29:45,412 INFO org.apache.beam.sdk.io.aws2.kinesis.SimplifiedKinesisClient [] - Starting getIterator request GetShardIteratorRequest(StreamName=ExampleInputStream, ShardId=shardId-000000000002, ShardIteratorType=AT_TIMESTAMP, Timestamp=2024-05-16T10:59:38.912Z)
2024-05-16 12:29:45,514 INFO org.apache.beam.sdk.io.aws2.kinesis.SimplifiedKinesisClient [] - Starting getIterator request GetShardIteratorRequest(StreamName=ExampleInputStream, ShardId=shardId-000000000003, ShardIteratorType=AT_TIMESTAMP, Timestamp=2024-05-16T10:59:38.912Z)
2024-05-16 12:29:45,620 INFO org.apache.beam.sdk.io.aws2.kinesis.ShardReadersPool [] - Starting to read ExampleInputStream stream from [shardId-000000000000, shardId-000000000001, shardId-000000000002, shardId-000000000003] shards
Issue Priority
Priority: 3 (minor)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam YAML
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Spark Runner
- Component: Flink Runner
- Component: Samza Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 3
- Comments: 28 (12 by maintainers)
Commits related to this issue
- [flink] FlinkRunner initializes the same split twice (#31313) — committed to je-ik/beam by je-ik 2 years ago
- [flink] Redistribute splits after rescaling (#31313) — committed to je-ik/beam by je-ik 2 years ago
- [flink] Redistribute splits after rescaling (#31313) — committed to je-ik/beam by je-ik 2 years ago
- [flink] FlinkRunner initializes the same split twice (#31313) (#33606) * [flink] FlinkRunner initializes the same split twice (#31313) — committed to apache/beam by je-ik 2 years ago
- [flink] FlinkRunner initializes the same split twice (#31313) (#33606) * [flink] FlinkRunner initializes the same split twice (#31313) — committed to VardhanThigle/beam by je-ik 2 years ago
- [flink] FlinkRunner initializes the same split twice (#31313) (#33606) * [flink] FlinkRunner initializes the same split twice (#31313) — committed to JozoVilcek/beam by je-ik 2 years ago
Hi, we are seeing the same behavior on our pipeline.
In our case we are using:
beam-sdks-java-io-kinesis)Hi @je-ik , was just able to reproduce the issue by manually running the jar file.
Started the job by running
java -cp <jar> <class> --runner=flink --flinkMaster=...and then usedflink stopwith savepoint to take a savepoint. afterwards, I reran,java -cp <jar> <class> --runner=flink --flinkMaster=... --savepointPath=...with the snapshot I just took. After observing the task manager logs, I see:this was a few seconds after the job was submitted. I trimmed the output, but these two logs were there for all of my shards.
Yes.
Got it, double confirm on the suggested solution at the moment
Good catch! 👍 The reason is that the splits are statically assigned to the workers after the initial split. Seems the implementation of
addSplitsBackis wrong, I’ll look into that.Thanks for the suggestions, will give them a try. I believe the first comment of the ticket provides a simple pipeline that exhibits this behavior on the flink runner but if that doesn’t work, happy to provide another. The example also submits the job in detached mode which may be related, although have seen similar behavior without it. Appreciate your help looking into this, if there’s anything I can assist with, please let me know