trino: Incorrect results on parquet files written by parquet-cpp-arrow version 5.0.0

After upgrading Trino version from 451 to 476, we discovered a regression where Hive tables created on the location of existing Parquet files in S3 return zero rows when queried.

This pattern with “preloaded-data” worked correctly in Trino version 451 and also works in Apache Spark, but in Trino version 476, the same setup results in an empty table scan — even though the data files are present and readable.

This regression is particularly impactful for our users, many of whom follow a pattern where:

  • They preprocess data using tools such as Pandas
  • Upload the resulting Parquet files directly to S3
  • Create Hive tables in that S3 location to query that data with Trino

Reproduction steps

  1. Upload valid Parquet files to an S3 directory.
  2. Create a Hive table in anywhere(Spark, Trino v451, or Trino v476) pointing to that location, like:
    CREATE TABLE hive.default.preloaded_parquet_test (
      id BIGINT,
      name VARCHAR
    )
    WITH (
      external_location = 's3://my-bucket/test-data/preloaded/',
      format = 'PARQUET'
    );
    
  3. Query the table:
SELECT * FROM hive.default.preloaded_parquet_test;

→ Returns zero rows

However…the data is read correctly and returned as expected with Spark or Trino v451.

We use AWS Glue Catalog as the metastore. so, I initially suspected the issue might be related to Trino’s switch to the AWS Glue Data Catalog V2 API, but we confirmed that even Trino 451 works correctly when configured to use Glue V2 API.

Expected behavior

Trino should discover and read the existing Parquet files in the specified external location and return the data, as it did in version 451 (and as Spark still does).

Additional context

(additional tests on partitioned tables with “preloaded-data” file location)

Even for partitioned tables, the issue persists:

  • Running MSCK REPAIR TABLE from Spark to register partitions, or
  • Executing Trino’s CALL system.sync_partition_metadata(...) procedure

…does not resolve the issue in Trino v476. The table still returns zero rows. (the data is read correctly and returned as expected with Spark or Trino v451.)

We hope this regression can be addressed in an upcoming patch release.

Let us know if any additional diagnostics or test files are needed to reproduce the issue…!

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 17 (17 by maintainers)

Most upvoted comments

@raunaqmorarka Where is the metadata Parquet file located, and how do you check its version information?

The relevant parquet file was shared privately with me. The version is from output of parquet meta

parquet meta ~/test_data/test_bug/rider_first_pickup.parquet

File path:  /Users/raunaq.morarka/test_data/test_bug/rider_first_pickup.parquet
Created by: parquet-cpp-arrow version 5.0.0
Properties:
        pandas: {"index_columns": [], "column_indexes": [], "columns": [{"name": "brms_rider_id", "field_name": "brms_rider_id", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "bros_rider_id", "field_name": "bros_rider_id", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "first_business_day", "field_name": "first_business_day", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "first_pickup_kst_fl", "field_name": "first_pickup_kst_fl", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "first_business_day_fl", "field_name": "first_business_day_fl", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "first_dlvry_sn_fl", "field_name": "first_dlvry_sn_fl", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "first_dlvry_method_fl", "field_name": "first_dlvry_method_fl", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "first_pickup_rgn3_cd_fl", "field_name": "first_pickup_rgn3_cd_fl", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "first_pickup_kst_3pl", "field_name": "first_pickup_kst_3pl", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "first_business_day_3pl", "field_name": "first_business_day_3pl", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "first_dlvry_sn_3pl", "field_name": "first_dlvry_sn_3pl", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "first_dlvry_method_3pl", "field_name": "first_dlvry_method_3pl", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "first_pickup_rgn3_cd_3pl", "field_name": "first_pickup_rgn3_cd_3pl", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}], "creator": {"library": "pyarrow", "version": "5.0.0"}, "pandas_version": "1.2.0"}
  ARROW:schema: /////wALAAAQAAAAAAAKAA4ABgAFAAgACgAAAAABBAAQAAAAAAAKAAwAAAAEAAgACgAAAMwHAAAEAAAAAQAAAAwAAAAIAAwABAAIAAgAAAAIAAAAEAAAAAYAAABwYW5kYXMAAJQHAAB7ImluZGV4X2NvbHVtbnMiOiBbXSwgImNvbHVtbl9pbmRleGVzIjogW10sICJjb2x1bW5zIjogW3sibmFtZSI6ICJicm1zX3JpZGVyX2lkIiwgImZpZWxkX25hbWUiOiAiYnJtc19yaWRlcl9pZCIsICJwYW5kYXNfdHlwZSI6ICJ1bmljb2RlIiwgIm51bXB5X3R5cGUiOiAib2JqZWN0IiwgIm1ldGFkYXRhIjogbnVsbH0sIHsibmFtZSI6ICJicm9zX3JpZGVyX2lkIiwgImZpZWxkX25hbWUiOiAiYnJvc19yaWRlcl9pZCIsICJwYW5kYXNfdHlwZSI6ICJ1bmljb2RlIiwgIm51bXB5X3R5cGUiOiAib2JqZWN0IiwgIm1ldGFkYXRhIjogbnVsbH0sIHsibmFtZSI6ICJmaXJzdF9idXNpbmVzc19kYXkiLCAiZmllbGRfbmFtZSI6ICJmaXJzdF9idXNpbmVzc19kYXkiLCAicGFuZGFzX3R5cGUiOiAidW5pY29kZSIsICJudW1weV90eXBlIjogIm9iamVjdCIsICJtZXRhZGF0YSI6IG51bGx9LCB7Im5hbWUiOiAiZmlyc3RfcGlja3VwX2tzdF9mbCIsICJmaWVsZF9uYW1lIjogImZpcnN0X3BpY2t1cF9rc3RfZmwiLCAicGFuZGFzX3R5cGUiOiAidW5pY29kZSIsICJudW1weV90eXBlIjogIm9iamVjdCIsICJtZXRhZGF0YSI6IG51bGx9LCB7Im5hbWUiOiAiZmlyc3RfYnVzaW5lc3NfZGF5X2ZsIiwgImZpZWxkX25hbWUiOiAiZmlyc3RfYnVzaW5lc3NfZGF5X2ZsIiwgInBhbmRhc190eXBlIjogInVuaWNvZGUiLCAibnVtcHlfdHlwZSI6ICJvYmplY3QiLCAibWV0YWRhdGEiOiBudWxsfSwgeyJuYW1lIjogImZpcnN0X2RsdnJ5X3NuX2ZsIiwgImZpZWxkX25hbWUiOiAiZmlyc3RfZGx2cnlfc25fZmwiLCAicGFuZGFzX3R5cGUiOiAidW5pY29kZSIsICJudW1weV90eXBlIjogIm9iamVjdCIsICJtZXRhZGF0YSI6IG51bGx9LCB7Im5hbWUiOiAiZmlyc3RfZGx2cnlfbWV0aG9kX2ZsIiwgImZpZWxkX25hbWUiOiAiZmlyc3RfZGx2cnlfbWV0aG9kX2ZsIiwgInBhbmRhc190eXBlIjogInVuaWNvZGUiLCAibnVtcHlfdHlwZSI6ICJvYmplY3QiLCAibWV0YWRhdGEiOiBudWxsfSwgeyJuYW1lIjogImZpcnN0X3BpY2t1cF9yZ24zX2NkX2ZsIiwgImZpZWxkX25hbWUiOiAiZmlyc3RfcGlja3VwX3JnbjNfY2RfZmwiLCAicGFuZGFzX3R5cGUiOiAidW5pY29kZSIsICJudW1weV90eXBlIjogIm9iamVjdCIsICJtZXRhZGF0YSI6IG51bGx9LCB7Im5hbWUiOiAiZmlyc3RfcGlja3VwX2tzdF8zcGwiLCAiZmllbGRfbmFtZSI6ICJmaXJzdF9waWNrdXBfa3N0XzNwbCIsICJwYW5kYXNfdHlwZSI6ICJ1bmljb2RlIiwgIm51bXB5X3R5cGUiOiAib2JqZWN0IiwgIm1ldGFkYXRhIjogbnVsbH0sIHsibmFtZSI6ICJmaXJzdF9idXNpbmVzc19kYXlfM3BsIiwgImZpZWxkX25hbWUiOiAiZmlyc3RfYnVzaW5lc3NfZGF5XzNwbCIsICJwYW5kYXNfdHlwZSI6ICJ1bmljb2RlIiwgIm51bXB5X3R5cGUiOiAib2JqZWN0IiwgIm1ldGFkYXRhIjogbnVsbH0sIHsibmFtZSI6ICJmaXJzdF9kbHZyeV9zbl8zcGwiLCAiZmllbGRfbmFtZSI6ICJmaXJzdF9kbHZyeV9zbl8zcGwiLCAicGFuZGFzX3R5cGUiOiAidW5pY29kZSIsICJudW1weV90eXBlIjogIm9iamVjdCIsICJtZXRhZGF0YSI6IG51bGx9LCB7Im5hbWUiOiAiZmlyc3RfZGx2cnlfbWV0aG9kXzNwbCIsICJmaWVsZF9uYW1lIjogImZpcnN0X2RsdnJ5X21ldGhvZF8zcGwiLCAicGFuZGFzX3R5cGUiOiAidW5pY29kZSIsICJudW1weV90eXBlIjogIm9iamVjdCIsICJtZXRhZGF0YSI6IG51bGx9LCB7Im5hbWUiOiAiZmlyc3RfcGlja3VwX3JnbjNfY2RfM3BsIiwgImZpZWxkX25hbWUiOiAiZmlyc3RfcGlja3VwX3JnbjNfY2RfM3BsIiwgInBhbmRhc190eXBlIjogInVuaWNvZGUiLCAibnVtcHlfdHlwZSI6ICJvYmplY3QiLCAibWV0YWRhdGEiOiBudWxsfV0sICJjcmVhdG9yIjogeyJsaWJyYXJ5IjogInB5YXJyb3ciLCAidmVyc2lvbiI6ICI1LjAuMCJ9LCAicGFuZGFzX3ZlcnNpb24iOiAiMS4yLjAifQAAAAANAAAA0AIAAIwCAABUAgAAHAIAAOABAACoAQAAbAEAADABAAD0AAAAuAAAAIAAAABEAAAABAAAAHT9//8AAAEFEAAAACwAAAAEAAAAAAAAABgAAABmaXJzdF9waWNrdXBfcmduM19jZF8zcGwAAAAAcP3//7D9//8AAAEFEAAAACgAAAAEAAAAAAAAABYAAABmaXJzdF9kbHZyeV9tZXRob2RfM3BsAACo/f//6P3//wAAAQUQAAAAJAAAAAQAAAAAAAAAEgAAAGZpcnN0X2RsdnJ5X3NuXzNwbAAA3P3//xz+//8AAAEFEAAAACgAAAAEAAAAAAAAABYAAABmaXJzdF9idXNpbmVzc19kYXlfM3BsAAAU/v//VP7//wAAAQUQAAAAKAAAAAQAAAAAAAAAFAAAAGZpcnN0X3BpY2t1cF9rc3RfM3BsAAAAAEz+//+M/v//AAABBRAAAAAoAAAABAAAAAAAAAAXAAAAZmlyc3RfcGlja3VwX3JnbjNfY2RfZmwAhP7//8T+//8AAAEFEAAAACgAAAAEAAAAAAAAABUAAABmaXJzdF9kbHZyeV9tZXRob2RfZmwAAAC8/v///P7//wAAAQUQAAAAJAAAAAQAAAAAAAAAEQAAAGZpcnN0X2RsdnJ5X3NuX2ZsAAAA8P7//zD///8AAAEFEAAAACgAAAAEAAAAAAAAABUAAABmaXJzdF9idXNpbmVzc19kYXlfZmwAAAAo////aP///wAAAQUQAAAAJAAAAAQAAAAAAAAAEwAAAGZpcnN0X3BpY2t1cF9rc3RfZmwAXP///5z///8AAAEFEAAAACQAAAAEAAAAAAAAABIAAABmaXJzdF9idXNpbmVzc19kYXkAAJD////Q////AAABBRAAAAAgAAAABAAAAAAAAAANAAAAYnJvc19yaWRlcl9pZAAAAMD///8QABQACAAGAAcADAAAABAAEAAAAAAAAQUQAAAAJAAAAAQAAAAAAAAADQAAAGJybXNfcmlkZXJfaWQAAAAEAAQABAAAAA==

The performance overhead of decoding column metadata for 1 column per row-group is minimal. All the relevant metadata is already in memory here. The original scenario which triggered this optimization was of parquet files with lots of columns and very small row groups due to some other writer bug. Having to decode column metadata for 1 column per row-group should still be far better than the original implementation which was decoding all and discarding most of column metadata because of split boundaries. This code was never a bottleneck for typical parquet files. We have a created_by field in parquet footer, and this field has been used to avoid other known writer bugs like some writer producing invalid column stats. I’d rather keep things simple in this case because we are not giving up much and there is no guarantee that there aren’t other writers with similar bug. There are too many parquet writers out there for an exhaustive writer + version search to be viable.

The parquet reader code is correct, but the metadata in the parquet file (written by parquet-cpp-arrow version 5.0.0) that we rely on is incorrect in this case. The file offset produced by org.apache.parquet.format.RowGroup#getFile_offset is not matching org.apache.parquet.format.ColumnMetaData#getDictionary_page_offset or org.apache.parquet.format.ColumnMetaData#getData_page_offset The file offset in the RowGroup seems to be wrongly populated by the writer here. https://github.com/trinodb/trino/pull/26064 will fix this by changing the code to not rely on file offset anymore.

Are we able to tell whether a parquet file was written by pyarrow (e.g., is there a writer/version recorded in the files)? If so, the reader could decide what approach to use based on that information, and potentially emit a warning to let the user know about the corruption.

Can you find the first bad version with bisect? What happens if you remove the following / from external_location table property?