hudi: [SUPPORT] Hive Sync tool fails to sync Hoodi table written using Flink 1.16 to HMS
Tips before filing an issue
- Have you gone through our FAQs?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an issue directly.
Describe the problem you faced Hive sync tool fails with the following stacktrace
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/calcite/plan/RelOptRule
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at org.apache.hudi.hive.util.IMetaStoreClientUtil.getMSC(IMetaStoreClientUtil.java:40)
at org.apache.hudi.hive.HoodieHiveSyncClient.<init>(HoodieHiveSyncClient.java:88)
at org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:122)
at org.apache.hudi.hive.HiveSyncTool.<init>(HiveSyncTool.java:116)
at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:482)
Caused by: java.lang.ClassNotFoundException: org.apache.calcite.plan.RelOptRule
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.C
To Reproduce
Write hudi tables using Flink 1.16, run Hive Sync tool on the local box to sync metadata to HMS specify sync-mode as hms and provide required parameters
Expected behavior
This should sync hoodie table to HMS
Environment Description
-
Hudi version : 0.13.0
-
Spark version :
-
Hive version : 3.1.2
-
Hadoop version : 2.10.2
-
Storage (HDFS/S3/GCS…) : ABFS ( Azure )
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 17 (8 by maintainers)
If you specify the hive profile, the hive-exec scope would switch to
compile:-Drat.skip=true -Pflink-bundle-shade-hive3I believe that only Hive3 needs the calcire related jars, when you make changes to the
run_sync_toolscript, keep caution for that.