iceberg: Parquet 1.11.1 update causes regressions while reading iceberg data written with v1.11.0

As part of https://github.com/apache/iceberg/issues/1441 Parquet version was updated from 1.11.0 to 1.11.1. After rebasing our internal version with latest changes from master we found that certain fields written with iceberg using Parquet v1.11.0 are not readable with iceberg built against Parquet v1.11.1

Problem: Data written using iceberg 0.9.0 that contains certain complex map types ( Map[string, Map[string, struct]] in this case) fails when reading with iceberg 0.10.0

Error:

java.lang.IllegalArgumentException: [segmentMembership, map, key] required binary key (STRING) = 9 is not in the store: [[identityMap, map, key] required binary key (STRING) = 3, [identityMap, map, value, list, element, id] optional binary id (STRING) = 7, [identityMap, map, value, list, element, authenticatedState] optional binary authenticatedState (STRING) = 6, [identityMap, map, value, list, element, primary] optional boolean primary = 8] 4
	at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ColumnChunkPageReadStore.getPageReader(ColumnChunkPageReadStore.java:231)
	at org.apache.iceberg.parquet.ParquetValueReaders$PrimitiveReader.setPageSource(ParquetValueReaders.java:185)
	at org.apache.iceberg.parquet.ParquetValueReaders$RepeatedKeyValueReader.setPageSource(ParquetValueReaders.java:529)
	at org.apache.iceberg.parquet.ParquetValueReaders$StructReader.setPageSource(ParquetValueReaders.java:685)
	at org.apache.iceberg.parquet.ParquetReader$FileIterator.advance(ParquetReader.java:142)
	at org.apache.iceberg.parquet.ParquetReader$FileIterator.next(ParquetReader.java:112)
	at org.apache.iceberg.io.FilterIterator.advance(FilterIterator.java:66)
	at org.apache.iceberg.io.FilterIterator.hasNext(FilterIterator.java:50)
	at org.apache.iceberg.spark.source.BaseDataReader.next(BaseDataReader.java:87)
	at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.hasNext(DataSourceRDD.scala:49)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:640)
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$$anonfun$run$3.apply(WriteToDataSourceV2Exec.scala:117)
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$$anonfun$run$3.apply(WriteToDataSourceV2Exec.scala:116)
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1560)
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:146)
	at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:67)
	at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:66)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.doRunTask(Task.scala:139)
	at org.apache.spark.scheduler.Task.run(Task.scala:112)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$13.apply(Executor.scala:497)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1526)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:503)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Field in question is of this form:

 |-- segmentMembership: map (nullable = true)
 |    |-- key: string
 |    |-- value: map (valueContainsNull = true)
 |    |    |-- key: string
 |    |    |-- value: struct (valueContainsNull = true)
 |    |    |    |-- payload: struct (nullable = true)
 |    |    |    |    |-- boolA: boolean (nullable = true)
 |    |    |    |    |-- doubleValueA: double (nullable = true)
 |    |    |    |    |-- doubleValueB: double (nullable = true)
 |    |    |    |    |-- stringValueA: string (nullable = true)
 |    |    |    |    |-- stringValueB: string (nullable = true)
 |    |    |    |-- status: string (nullable = true)
 |    |    |    |-- lastQualificationTime: timestamp (nullable = true)

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 16 (15 by maintainers)

Most upvoted comments

Thanks to @kbendick providing a table that reproduces the problem, I’ve been able to debug to find out what’s going on here. The issue is in column pruning. When Iceberg prunes the fields in the file schema using the expected Iceberg schema, it can return a Parquet schema that doesn’t match the file anymore because of the field rename.

Iceberg uses Parquet’s Types.map builder to produce a new map type in PruneColumns. But that was changed in https://github.com/apache/parquet-mr/pull/798 to produce a key/value records named key_value instead of map. So rebuilding the type using Parquet’s helpers actually produces a type with different names. Then Iceberg passes the new schema into Parquet as the projection and that causes the map column to be dropped because there is no mapCol.key_value structure, instead there is a mapCol.map.

The reason why this sometimes works in 0.12.0 is that the value check changed to equals instead of identity (==), so if you project the whole value the original map is returned. You can reproduce the issue in 0.12.0 by selecting a projection of the map value rather than *. For example:

SELECT mapCol.value.str FROM repro_table

The solution is to rebuild the map structure to exactly match the incoming file schema rather than relying on Parquet to produce the same thing across versions. I’ll open a PR.

rdblue on Oct 18, 2021

I just merged #3309 to fix this. We’ll also get this out in a 0.12.1 release shortly. Thanks for the bug reports and for helping us track this down, everyone. @kbendick, nice work on getting the repro case that could be debugged.

rdblue on Oct 19, 2021