iceberg: Spark 3.2.0 java.lang.IncompatibleClassChangeError when using IcebergSparkSessionExtensions

Iceberg version: 0.12.0 Spark version: 3.2.0

from pyspark.sql import SparkSession

spark = (SparkSession.builder
            .appName("spark.tests.iceberg")
            .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
            .getOrCreate())


spark.sql("select 'hello'").show()

…yields

: java.lang.IncompatibleClassChangeError: class org.apache.spark.sql.catalyst.plans.logical.DynamicFileFilterWithCardinalityCheck has interface org.apache.spark.sql.catalyst.plans.logical.BinaryNode as super class
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions.$anonfun$apply$8(IcebergSparkSessionExtensions.scala:50)
	at org.apache.spark.sql.SparkSessionExtensions.$anonfun$buildOptimizerRules$1(SparkSessionExtensions.scala:201)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 3
Comments: 31 (16 by maintainers)

Most upvoted comments

As of Iceberg 0.13.0, the iceberg-spark-runtime jars have changed to reflect the Spark (and Scala) versions:

You can test the 0.13.0-rc1, fetching it from the staging maven repository, with the following command line flags for Spark 3.2: --packages 'org.apache.iceberg:iceberg-spark-runtime-3.2_2.12:0.13.0' --repositories https://repository.apache.org/content/repositories/orgapacheiceberg-1079/

For other Spark versions than 3.2, use the artifactIds below (in place of iceberg-spark-runtime-3.2_2.12 above).

Iceberg 0.13.0 spark-runtime jar names Spark 3.0: iceberg-spark3-runtime:0.13.0 Spark 3.1: iceberg-spark-runtime-3.1_2.12 Spark 3.2: iceberg-spark-runtime-3.2_2.12

The complete package name for any depends on your spark version now. iceberg-spark3-runtime should only be used for Spark 3.0.

kbendick on Jan 25, 2022

https://github.com/apache/iceberg#compatibility

KnightChess on Jan 11, 2022

Thank you so much @nreich for working with us to debug this. This is a very important part of the release process and it really helps to have community members testing things out.

I’m going to close this issue in a bit if there are no more comments. Anybody please feel free to open a new issue referencing this one if need be!

TLDR - Be sure to have the correct Iceberg artifact based on the Spark version, as well as ensuring there aren’t extra iceberg jars in your Spark /jars folder (I’ve definitely done that more than once).

kbendick on Jan 26, 2022

That was the issue @rdblue: I had an errant copy of the old iceberg-spark3-runtime-0.12.1.jar on the classpath. After removing it, my original installation of spark 3.2.0 now functions as expected with iceberg 0.13.0.

nreich on Jan 26, 2022

@rdblue @kbendick I had tried that jar first, actually: had the same exact result, so I looked to the docs in master for getting started and found they still referred to the old jar (I must have missed the correct location to look for the changed getting started instructions?).

I cleared dependency caches (just in case), and ran again with:

bin/spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.2_2.12:0.13.0 \
    --repositories https://repository.apache.org/content/repositories/orgapacheiceberg-1079/ \
    ...

but still got the exact same stacktrace as before. Just as a sanity check, I switched over to spark 3.1.2 and the org.apache.iceberg:iceberg-spark-runtime-3.1_2.12:0.13.0 jar and that was able to create the table successfully.

nreich on Jan 26, 2022

Using spark 3.2.0 and iceberg-spark-runtime-3.2_2.12 (0.13.0-SNAPSHOT, built locally from master today), I still see this issue (did try adding .config(“spark.sql.sources.partitionOverwriteMode”, “dynamic”) with no luck). Is this issue still present in current master or is there more configuration required to resolve the issue or possibly an issue with my local build?

nreich on Jan 24, 2022

There is no version support now, but had supported in master branch #3335

KnightChess on Nov 20, 2021