hudi: [SUPPORT] Hive Sync tool fails to sync Hoodi table written using Flink 1.16 to HMS

Tips before filing an issue

  • Have you gone through our FAQs?
  • Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
  • If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced Hive sync tool fails with the following stacktrace

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/calcite/plan/RelOptRule
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
        at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
        at java.lang.Class.getMethod0(Class.java:3018)
        at java.lang.Class.getMethod(Class.java:1784)
        at org.apache.hudi.hive.util.IMetaStoreClientUtil.getMSC(IMetaStoreClientUtil.java:40)
        at org.apache.hudi.hive.HoodieHiveSyncClient.<init>(HoodieHiveSyncClient.java:88)
        at org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:122)
        at org.apache.hudi.hive.HiveSyncTool.<init>(HiveSyncTool.java:116)
        at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:482)
Caused by: java.lang.ClassNotFoundException: org.apache.calcite.plan.RelOptRule
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.C

To Reproduce Write hudi tables using Flink 1.16, run Hive Sync tool on the local box to sync metadata to HMS specify sync-mode as hms and provide required parameters

Expected behavior

This should sync hoodie table to HMS

Environment Description

  • Hudi version : 0.13.0

  • Spark version :

  • Hive version : 3.1.2

  • Hadoop version : 2.10.2

  • Storage (HDFS/S3/GCS…) : ABFS ( Azure )

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Comments: 17 (8 by maintainers)

Most upvoted comments

If you specify the hive profile, the hive-exec scope would switch to compile: -Drat.skip=true -Pflink-bundle-shade-hive3

I believe that only Hive3 needs the calcire related jars, when you make changes to the run_sync_tool script, keep caution for that.