unitycatalog: Spark Structured Streaming is not supported - DELTA_UNSUPPORTED_SCHEMA_DURING_READ
Describe the bug
It seems that using Spark Structured Streaming from a managed table (that should support tead) to an external table (that should support read/write) - is not working. It throws DELTA_UNSUPPORTED_SCHEMA_DURING_READ.
To Reproduce
Steps to reproduce the behavior:
- Download and tar fresh copy of Spark 3.5.3
- Start UC server
- Start Spark with UC:
bin/spark-shell --name "local-uc-test" \
--master "local[*]" \
--packages "io.delta:delta-spark_2.12:3.2.1,io.unitycatalog:unitycatalog-spark_2.12:0.2.0" \
--conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \
--conf "spark.sql.catalog.spark_catalog=io.unitycatalog.spark.UCSingleCatalog" \
--conf "spark.sql.catalog.unity=io.unitycatalog.spark.UCSingleCatalog" \
--conf "spark.sql.catalog.unity.uri=http://localhost:8080" \
--conf "spark.sql.catalog.unity.token=" \
--conf "spark.sql.defaultCatalog=unity"
- Create the external output table as follows:
spark.sql("create table unity.default.output using delta location '/Users/username/Downloads/TestTable' as select * from unity.default.marksheet limit 1") - Run a stream as follows:
spark.readStream.format("delta").table("unity.default.marksheet").writeStream.format("delta").outputMode("append").option("checkpointLocation", "/Users/username/Downloads/TestTable").toTable("unity.default.output").awaitTermination
We get an exception:
org.apache.spark.sql.delta.DeltaAnalysisException: [DELTA_UNSUPPORTED_SCHEMA_DURING_READ] Delta does not support specifying the schema at read time.
at org.apache.spark.sql.delta.DeltaErrorsBase.specifySchemaAtReadTimeException(DeltaErrors.scala:1034)
at org.apache.spark.sql.delta.DeltaErrorsBase.specifySchemaAtReadTimeException$(DeltaErrors.scala:1033)
at org.apache.spark.sql.delta.DeltaErrors$.specifySchemaAtReadTimeException(DeltaErrors.scala:3382)
at org.apache.spark.sql.delta.sources.DeltaDataSource.sourceSchema(DeltaDataSource.scala:78)
at org.apache.spark.sql.execution.datasources.DataSource.sourceSchema(DataSource.scala:233)
at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo$lzycompute(DataSource.scala:118)
at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo(DataSource.scala:118)
at org.apache.spark.sql.execution.streaming.StreamingRelation$.apply(StreamingRelation.scala:36)
at org.apache.spark.sql.execution.datasources.FindDataSourceTable.org$apache$spark$sql$execution$datasources$FindDataSourceTable$$getStreamingRelation(DataSourceStrategy.scala:274)
at org.apache.spark.sql.execution.datasources.FindDataSourceTable$$anonfun$apply$2.applyOrElse(DataSourceStrategy.scala:307)
at org.apache.spark.sql.execution.datasources.FindDataSourceTable$$anonfun$apply$2.applyOrElse(DataSourceStrategy.scala:278)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDownWithPruning$2(AnalysisHelper.scala:170)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:76)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDownWithPruning$1(AnalysisHelper.scala:170)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:323)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDownWithPruning(AnalysisHelper.scala:168)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDownWithPruning$(AnalysisHelper.scala:164)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsDownWithPruning(LogicalPlan.scala:32)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDownWithPruning$4(AnalysisHelper.scala:175)
at org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren(TreeNode.scala:1216)
at org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren$(TreeNode.scala:1215)
at org.apache.spark.sql.catalyst.plans.logical.SubqueryAlias.mapChildren(basicLogicalOperators.scala:1676)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDownWithPruning$1(AnalysisHelper.scala:175)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:323)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDownWithPruning(AnalysisHelper.scala:168)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDownWithPruning$(AnalysisHelper.scala:164)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsDownWithPruning(LogicalPlan.scala:32)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsWithPruning(AnalysisHelper.scala:99)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsWithPruning$(AnalysisHelper.scala:96)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsWithPruning(LogicalPlan.scala:32)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperators(AnalysisHelper.scala:76)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperators$(AnalysisHelper.scala:75)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:32)
at org.apache.spark.sql.execution.datasources.FindDataSourceTable.apply(DataSourceStrategy.scala:278)
at org.apache.spark.sql.execution.datasources.FindDataSourceTable.apply(DataSourceStrategy.scala:241)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:222)
at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
at scala.collection.immutable.List.foldLeft(List.scala:91)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:219)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:211)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:211)
at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:240)
at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$execute$1(Analyzer.scala:236)
at org.apache.spark.sql.catalyst.analysis.AnalysisContext$.withNewAnalysisContext(Analyzer.scala:187)
at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:236)
at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:202)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:182)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:89)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:182)
at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:223)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:330)
at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:222)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:77)
at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:138)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$2(QueryExecution.scala:219)
at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:546)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:219)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900)
at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:218)
at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:77)
at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:74)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:66)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$1(Dataset.scala:91)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:89)
at org.apache.spark.sql.streaming.DataStreamReader.table(DataStreamReader.scala:316)
... 47 elided
Expected behavior The strram should work đ
System [please complete the following information]:
- OS: Mac OS Sonoma [14.7.1 (23H222)]
Additional context AFAIK this should be working from 0.2.0 - as we support read from managed tables and read/write of external tables - and it seems that write to external tables using structured streaming is not workingâŚ
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 22 (10 by maintainers)
Commits related to this issue
- [SPARK] Fix test illustrating streaming issues with Unity Catalog * The existing test for Spark Structured Streaming did not fully cover the issues described in #715. * I've updated the test to inclu... — committed to arturowczarek/unitycatalog by arturowczarek 2 years ago
- [SPARK] Fix test illustrating streaming issues with Unity Catalog * The existing test for Spark Structured Streaming did not fully cover the issues described in #715. * I've updated the test to inclu... — committed to arturowczarek/unitycatalog by arturowczarek 2 years ago
- [SPARK] Fix test illustrating streaming issues with Unity Catalog * The existing test for Spark Structured Streaming did not fully cover the issues described in #715. * I've updated the test to inclu... — committed to arturowczarek/unitycatalog by arturowczarek 2 years ago
- [SPARK] Fix test illustrating streaming issues with Unity Catalog * The existing test for Spark Structured Streaming did not fully cover the issues described in #715. * I've updated the test to inclu... — committed to arturowczarek/unitycatalog by arturowczarek 2 years ago
- [SPARK] Fix test illustrating streaming issues with Unity Catalog * The existing test for Spark Structured Streaming did not fully cover the issues described in #715. * I've updated the test to inclu... — committed to arturowczarek/unitycatalog by arturowczarek 2 years ago
- [SPARK] Fix test illustrating streaming issues with Unity Catalog * The existing test for Spark Structured Streaming did not fully cover the issues described in #715. * I've updated the test to inclu... — committed to arturowczarek/unitycatalog by arturowczarek 2 years ago
- [SPARK] Fix test illustrating streaming issues with Unity Catalog * The existing test for Spark Structured Streaming did not fully cover the issues described in #715. * I've updated the test to inclu... — committed to arturowczarek/unitycatalog by arturowczarek 2 years ago
- [SPARK] Fix test illustrating streaming issues with Unity Catalog * The existing test for Spark Structured Streaming did not fully cover the issues described in #715. * I've updated the test to inclu... — committed to arturowczarek/unitycatalog by arturowczarek 2 years ago
OK, Iâve figured it out now. This is a long-standing issue in Delta Lake, that the Delta table must have an empty schema in the catalog.
Delta Lake always sets the table schema to empty before saving the table into the catalog. However, the default table in UC does not keep the schema empty, and then we hit this issue. Databricks is fine as its Spark fork has a workaround for it.
If you donât use the default table in UC but create tables via Spark manually, you wonât hit an issue, see https://github.com/unitycatalog/unitycatalog/pull/769
I think the right fix is to relax the check in Delta Lake here: https://github.com/delta-io/delta/blob/4d2c5cfd45119d78a86560de12e7b946228d77f2/spark/src/main/scala/org/apache/spark/sql/delta/sources/DeltaDataSource.scala#L78 Itâs fine to have the specified schema, as it may come from the table metadata, as long as the schema is the same as the actual delta table schema.
cc @tdas @jaceklaskowski
@tdas Youâre correct. âthe schema was specified for the source, but the query you provided does not specify the schema.â is the issue that I could reproduce myself and am digging deeper now.