trino: Incorrect results on parquet files written by parquet-cpp-arrow version 5.0.0
After upgrading Trino version from 451 to 476, we discovered a regression where Hive tables created on the location of existing Parquet files in S3 return zero rows when queried.
This pattern with “preloaded-data” worked correctly in Trino version 451 and also works in Apache Spark, but in Trino version 476, the same setup results in an empty table scan — even though the data files are present and readable.
This regression is particularly impactful for our users, many of whom follow a pattern where:
- They preprocess data using tools such as Pandas
- Upload the resulting Parquet files directly to S3
- Create Hive tables in that S3 location to query that data with Trino
Reproduction steps
- Upload valid Parquet files to an S3 directory.
- Create a Hive table in anywhere(Spark, Trino v451, or Trino v476) pointing to that location, like:
CREATE TABLE hive.default.preloaded_parquet_test ( id BIGINT, name VARCHAR ) WITH ( external_location = 's3://my-bucket/test-data/preloaded/', format = 'PARQUET' ); - Query the table:
SELECT * FROM hive.default.preloaded_parquet_test;
→ Returns zero rows
However…the data is read correctly and returned as expected with Spark or Trino v451.
We use AWS Glue Catalog as the metastore. so, I initially suspected the issue might be related to Trino’s switch to the AWS Glue Data Catalog V2 API, but we confirmed that even Trino 451 works correctly when configured to use Glue V2 API.
Expected behavior
Trino should discover and read the existing Parquet files in the specified external location and return the data, as it did in version 451 (and as Spark still does).
Additional context
(additional tests on partitioned tables with “preloaded-data” file location)
Even for partitioned tables, the issue persists:
- Running
MSCK REPAIR TABLEfrom Spark to register partitions, or - Executing Trino’s
CALL system.sync_partition_metadata(...)procedure
…does not resolve the issue in Trino v476. The table still returns zero rows. (the data is read correctly and returned as expected with Spark or Trino v451.)
We hope this regression can be addressed in an upcoming patch release.
Let us know if any additional diagnostics or test files are needed to reproduce the issue…!
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 17 (17 by maintainers)
The relevant parquet file was shared privately with me. The version is from output of
parquet metaThe performance overhead of decoding column metadata for 1 column per row-group is minimal. All the relevant metadata is already in memory here. The original scenario which triggered this optimization was of parquet files with lots of columns and very small row groups due to some other writer bug. Having to decode column metadata for 1 column per row-group should still be far better than the original implementation which was decoding all and discarding most of column metadata because of split boundaries. This code was never a bottleneck for typical parquet files. We have a
created_byfield in parquet footer, and this field has been used to avoid other known writer bugs like some writer producing invalid column stats. I’d rather keep things simple in this case because we are not giving up much and there is no guarantee that there aren’t other writers with similar bug. There are too many parquet writers out there for an exhaustive writer + version search to be viable.The parquet reader code is correct, but the metadata in the parquet file (written by
parquet-cpp-arrow version 5.0.0) that we rely on is incorrect in this case. The file offset produced byorg.apache.parquet.format.RowGroup#getFile_offsetis not matchingorg.apache.parquet.format.ColumnMetaData#getDictionary_page_offsetororg.apache.parquet.format.ColumnMetaData#getData_page_offsetThe file offset in the RowGroup seems to be wrongly populated by the writer here. https://github.com/trinodb/trino/pull/26064 will fix this by changing the code to not rely on file offset anymore.Are we able to tell whether a parquet file was written by pyarrow (e.g., is there a writer/version recorded in the files)? If so, the reader could decide what approach to use based on that information, and potentially emit a warning to let the user know about the corruption.
Can you find the first bad version with bisect? What happens if you remove the following
/fromexternal_locationtable property?