beam: [Bug]: grpcio==1.48.0 causes Java expansion service to hang on startup in Python XLang pipelines.
What happened?
Symptoms
22:32:02 apache_beam/transforms/sql_test.py::SqlTransformTest::test_filter
22:32:02 -------------------------------- live log call ---------------------------------
22:32:02 INFO apache_beam.utils.subprocess_server:subprocess_server.py:239 Using pre-built snapshot at /home/jenkins/jenkins-slave/workspace/beam_PostCommit_XVR_Direct/src/sdks/java/extensions/sql/expansion-service/build/libs/beam-sdks-java-extensions-sql-expansion-service-2.42.0-SNAPSHOT.jar
22:32:02 INFO root:external.py:773 Starting a JAR-based expansion service from JAR /home/jenkins/jenkins-slave/workspace/beam_PostCommit_XVR_Direct/src/sdks/java/extensions/sql/expansion-service/build/libs/beam-sdks-java-extensions-sql-expansion-service-2.42.0-SNAPSHOT.jar
22:32:02 INFO apache_beam.utils.subprocess_server:subprocess_server.py:116 Starting service with ['java' '-jar' '/home/jenkins/jenkins-slave/workspace/beam_PostCommit_XVR_Direct/src/sdks/java/extensions/sql/expansion-service/build/libs/beam-sdks-java-extensions-sql-expansion-service-2.42.0-SNAPSHOT.jar' '56083' '--filesToStage=/home/jenkins/jenkins-slave/workspace/beam_PostCommit_XVR_Direct/src/sdks/java/extensions/sql/expansion-service/build/libs/beam-sdks-java-extensions-sql-expansion-service-2.42.0-SNAPSHOT.jar']
22:32:02 Build timed out (after 100 minutes). Marking the build as aborted.
23:50:24 Build was aborted
Workaround:
pip install grpcio==1.47.0
Affected test suites
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/2949/ https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/5697/ https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow/715/ https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/4014/ https://ci-beam.apache.org/job/beam_PostCommit_Python37/5528/ https://ci-beam.apache.org/job/beam_PostCommit_Python38/2939/ https://ci-beam.apache.org/job/beam_PostCommit_Python39/661/ https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/5931/
Issue Priority
Priority: 1
Issue Component
Component: cross-language
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 42 (34 by maintainers)
@gnossen and @drfloob yes, that seems to have fixed it. The test doesn’t get stuck on pull/30572/head but gets stuck on gRPC master. Thanks a lot for investigating.
@aaltay and @kileys seems like the issue is in the gRPC side. So no need to block on this. I’ll resolve this Beam side issue.
@chamikaramj Thanks for the feedback. We’ll keep digging and come back to you when we have another candidate to test.
I’ll also keep on trying to create our own in-repo reproduction.