arrow: [CI][Integration] Spark jobs are failing with problem on org.apache.arrow.flatbuf
Describe the bug, including details regarding any error messages, version, and platform.
The spark jobs: test-conda-python-3.10-spark-v3.5.0 test-conda-python-3.11-spark-master test-conda-python-3.8-spark-v3.5.0
Are currently failing with:
Error: ] /spark/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala:26: object flatbuf is not a member of package org.apache.arrow
Error: ] /spark/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala:456: Class org.apache.arrow.flatbuf.Message not found - continuing with a stub.
Error: [ERROR] two errors found
Error: Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.8.0:compile (scala-compile-first) on project spark-sql_2.12: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.8.0:compile failed: org.apache.commons.exec.ExecuteException: Process exited with an error: 255 (Exit value: 255) -> [Help 1]
Error:
Error: To see the full stack trace of the errors, re-run Maven with the -e switch.
Error: Re-run Maven using the -X switch to enable full debug logging.
Error:
Error: For more information about the errors and possible solutions, please read the following articles:
Error: [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
Error:
Error: After correcting the problems, you can resume the build with the command
Error: mvn <args> -rf :spark-sql_2.12
Component(s)
Continuous Integration, Integration
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (15 by maintainers)
Commits related to this issue
- GH-41571: [Java] Revert GH-41307 (#41309) This reverts commit 9090e679da91e0544171c2da9f2b9ce8ba23d389. — committed to lidavidm/arrow by lidavidm 3 years ago
- GH-41571: [Java] Revert GH-41307 (#41309) This reverts commit 9090e679da91e0544171c2da9f2b9ce8ba23d389. — committed to lidavidm/arrow by lidavidm 3 years ago
- GH-41571: [Java] Revert GH-41307 (#41309) (#41628) ### Rationale for this change The commit in question caused a lot of CI issues ### Are these changes tested? N/A ### Are there any user-facing c... — committed to apache/arrow by lidavidm 3 years ago
- GH-41571: [Java] Revert GH-41307 (#41309) (#41628) ### Rationale for this change The commit in question caused a lot of CI issues ### Are these changes tested? N/A ### Are there any user-facing c... — committed to vibhatha/arrow by lidavidm 3 years ago
This is also reproducible for 16.1.0 but I don’t think this is related to any change on our side and doesn’t seem to be a blocker. @danepitkin @vibhatha @kiszk any thoughts?
The issue seems to be because
maven-shade-plugingenerates and attach a new pom file containing the following dependency:which is exactly what the plugin is supposed to be doing
I have the same resulting pom by using
maven-shade-plugin3.3.0, and going through changes, I found this change which is a bugfix to the plugin for this issueAs to how to address this within the Arrow project, I’d need some perspective. The main issue is that a Maven module has a unique
pom.xmldefinition for all the artifacts attached to it. But when attaching shaded artifacts the list of dependencies is not quite the same, and this is definitively conflicting. I have no idea which project is actually using the shaded version ofarrow-vector, but Spark isn’t and so if we agree that the pom.xml should reflect the dependencies for the main artifact, then we should tellmaven-shade-pluginto not generate a newpom.xmlfile. If we need both dependencies, then we are quite in a predicament I’d guess (and same would apply to the other plugins usingmaven-shade-plugin)It would be good to know why maven-shade-plugin is causing issues, though
We can revert. Sorry about that @kiszk