beam: [Bug]: BigQuery Write step timeout since version 2.54.0
What happened?
I have a Batch pipeline that reads data from Firestore and writes it to BigQuery
pipeline
.apply(Create.of("myCollection"))
.apply("QueryRequest", ParDo.of(new QueryRequestFn(projectId, firestoreDatabaseId))) // Creates a RunQueryRequest
.apply("ReadFromFirestore", FirestoreIO.v1().read()
.runQuery()
.withRpcQosOptions(rpcQosOptions)
.build())
.apply("WriteToBigQuery", writeToBigQuery(projectId, firestoreStateDataset));
The pipeline was working with Apache Beam Version 2.52.0.
When I updated to 2.55.1 and it failed, I tested with 2.55.0 and 2.54.0 and it also failed, it only worked back with 2.53.0.
I fails when reading from Firestore, this PTransform internally uses Reshuffle.viaRandomKey() and it’s where it fails.
The error message is not very informative, it contains only timeout errors:
Workflow failed. Causes: S24:ReadPromFirestore/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+ReadFromFirestore/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+ReadFromFirestore/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)+ReadFromFirestore/Reshuffle.ViaRandomKey/Reshuffle/RestoreMetadata/ParDo(Anonymous)/ParMultiDo(Anonymous)+ReadFromFirestore/Reshuffle.ViaRandomKey/Values/Values/Map/ParMultiDo(Anonymous)+WriteToBigQuery/WriteProfilesToBigQuery/PrepareWrite/ParDo(Anonymous)/ParMultiDo(Anonymous)+WriteoBigQuery/WriteProfilesToBigQuery/BatchLoads/rewindowIntoGlobal/Window.Assign+WriteToBigQuery/WriteToBigQuery/BatchLoads/WriteBundlesToFiles+WriteToBigQuery/WriteToBigQuery/BatchLoads/ReifyResults/View.AsIterable/MapElements/Map/ParMultiDo(Anonymous)+WriteToBigQuery/WriteToBigQuery/BatchLoads/GroupByDestination/Write failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. If the logs only contain generic timeout errors related to accessing external resources, such as MongoDB, verify that the worker service account has permission to access the resource's subnetwork. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. The work item was attempted on these workers:
Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact.
Worker ID: builder-tes-04240840-lbhk-harness-kshq,
Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact.
Worker ID: builder-tes-04240840-lbhk-harness-m8ht,
Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact.
Worker ID: builder-tes-04240840-lbhk-harness-r37n,
Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact.
Worker ID: builder-tes-04240840-lbhk-harness-ht76
I noticed the implementation of the Reshuffle changed
Graph using version 2.54.0
Graph using version 2.53.0
This is the commit that changed the Reshuffle implementation from @kennknowles
Could someone investigate or give me some ideas of what might be the problem?
Thanks
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam YAML
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Spark Runner
- Component: Flink Runner
- Component: Samza Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 30 (17 by maintainers)
So to summarize, we could check these configurations:
--experiments=use_runner_v2--experiments=disable_runner_v2If the problem is with “new reshuffle” then we in fact have a flag that will allow you to use the old expansion while still upgrading to Beam 2.54.0:
--updateCompatibilityVersion=2.53.0. This is not specifically meant for this purpose (it is aimed at streaming update compatibility) and can impact other transforms as well. Also note that the reason for the change was that the old reshuffle yielded incorrect results in pipelines that use triggers; not an issue for batch.Is it possible for you to run your job on 2.53.0 with Runner V2 enabled?
Thanks for reporting this. Those four error messages are from the service’s point of view (after 4 it fails a batch job). I wonder if we see a crash loop or some such thing on the worker.
@kennknowles FYI.
For the support ticket, please check https://cloud.google.com/dataflow/docs/support/getting-support
For 2.54.0, can you try to disable Runner V2 (
--experiments=disable_runner_v2)?another change is that since Beam 2.54.0 batch job is using Dataflow runner v2 by default. Was your 2.53.0 job show “runner v2: disabled” and 2.54.0 “runner v2: enabled” in the Dataflow UI?