arrow-rs: Allow Parquet reader to read incorrectly written (negative) uint8, uint16 values for compatibility

Describe the bug The parquet spec says a uint8 or uint16 value must be an int32 annotated by INT(8, false), INT(16, false). A file with such values gets read into a int32 vector and the value read may be negative. When casting these values to the unsigned value, the cast method checks if the value is outside the range of valid values for an unsigned value. Since a negative value is outside the range the cast method will either return null or throw an error (depending on the specified cast option).

To Reproduce I modified parquet/examples/read_parquet.rs to read columns _9, and _10 from the attached file.

The file schema and contents as dumped by the parquet cli -

Schema

File path:  ./alltypes_extended_plain.parquet
Created by: parquet-mr version 1.13.1 (build db4183109d5b734ec5930d870cdae161e408ddba)
Properties:
  writer.model.name: example
Schema:
message root {
  optional boolean _1;
  optional int32 _2 (INTEGER(8,true));
  optional int32 _3 (INTEGER(16,true));
  optional int32 _4;
  optional int64 _5;
  optional float _6;
  optional double _7;
  optional binary _8 (STRING);
  optional int32 _9 (INTEGER(8,false));
  optional int32 _10 (INTEGER(16,false));
  optional int32 _11 (INTEGER(32,false));
  optional int64 _12 (INTEGER(64,false));
  optional binary _13 (ENUM);
  optional fixed_len_byte_array(3) _14;
  optional int32 _15 (DECIMAL(5,2));
  optional int64 _16 (DECIMAL(18,10));
  optional fixed_len_byte_array(16) _17 (DECIMAL(38,37));
  optional int64 _18 (TIMESTAMP(MILLIS,true));
  optional int64 _19 (TIMESTAMP(MICROS,true));
  optional int32 _20 (DATE);
}

Values -

{"_1": null, "_2": null, "_3": null, "_4": null, "_5": null, "_6": null, "_7": null, "_8": null, "_9": null, "_10": null, "_11": null, "_12": null, "_13": null, "_14": null, "_15": null, "_16": null, "_17": null, "_18": null, "_19": null, "_20": null}
{"_1": null, "_2": null, "_3": null, "_4": null, "_5": null, "_6": null, "_7": null, "_8": null, "_9": null, "_10": null, "_11": null, "_12": null, "_13": null, "_14": null, "_15": null, "_16": null, "_17": null, "_18": null, "_19": null, "_20": null}
{"_1": true, "_2": 18, "_3": 10002, "_4": 10002, "_5": 10002, "_6": 10002.0, "_7": 10002.0, "_8": "100021000210002100021000210002100021000210002100021000210002100021000210002100021000210002100021000210002100021000210002100021000210002100021000210002100021000210002100021000210002100021000210002100021000210002100021000210002100021000210002", "_9": -18, "_10": -10002, "_11": -10002, "_12": -10002, "_13": "10002", "_14": [50, 50, 50], "_15": 10002, "_16": 10002, "_17": [50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50], "_18": 10002, "_19": 10002, "_20": 10002}
{"_1": null, "_2": null, "_3": null, "_4": null, "_5": null, "_6": null, "_7": null, "_8": null, "_9": null, "_10": null, "_11": null, "_12": null, "_13": null, "_14": null, "_15": null, "_16": null, "_17": null, "_18": null, "_19": null, "_20": null}
{"_1": true, "_2": 20, "_3": 10004, "_4": 10004, "_5": 10004, "_6": 10004.0, "_7": 10004.0, "_8": "100041000410004100041000410004100041000410004100041000410004100041000410004100041000410004100041000410004100041000410004100041000410004100041000410004100041000410004100041000410004100041000410004100041000410004100041000410004100041000410004", "_9": -20, "_10": -10004, "_11": -10004, "_12": -10004, "_13": "10004", "_14": [52, 52, 52], "_15": 10004, "_16": 10004, "_17": [52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52], "_18": 10004, "_19": 10004, "_20": 10004}
{"_1": null, "_2": null, "_3": null, "_4": null, "_5": null, "_6": null, "_7": null, "_8": null, "_9": null, "_10": null, "_11": null, "_12": null, "_13": null, "_14": null, "_15": null, "_16": null, "_17": null, "_18": null, "_19": null, "_20": null}
{"_1": null, "_2": null, "_3": null, "_4": null, "_5": null, "_6": null, "_7": null, "_8": null, "_9": null, "_10": null, "_11": null, "_12": null, "_13": null, "_14": null, "_15": null, "_16": null, "_17": null, "_18": null, "_19": null, "_20": null}
{"_1": null, "_2": null, "_3": null, "_4": null, "_5": null, "_6": null, "_7": null, "_8": null, "_9": null, "_10": null, "_11": null, "_12": null, "_13": null, "_14": null, "_15": null, "_16": null, "_17": null, "_18": null, "_19": null, "_20": null}
{"_1": true, "_2": 24, "_3": 10008, "_4": 10008, "_5": 10008, "_6": 10008.0, "_7": 10008.0, "_8": "100081000810008100081000810008100081000810008100081000810008100081000810008100081000810008100081000810008100081000810008100081000810008100081000810008100081000810008100081000810008100081000810008100081000810008100081000810008100081000810008", "_9": -24, "_10": -10008, "_11": -10008, "_12": -10008, "_13": "10008", "_14": [56, 56, 56], "_15": 10008, "_16": 10008, "_17": [56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56], "_18": 10008, "_19": 10008, "_20": 10008}
{"_1": null, "_2": null, "_3": null, "_4": null, "_5": null, "_6": null, "_7": null, "_8": null, "_9": null, "_10": null, "_11": null, "_12": null, "_13": null, "_14": null, "_15": null, "_16": null, "_17": null, "_18": null, "_19": null, "_20": null}

Results

+----+-----+
| _9 | _10 |
+----+-----+
|    |     |
|    |     |
|    |     |
|    |     |
|    |     |
|    |     |
|    |     |
|    |     |
|    |     |
|    |     |
+----+-----+

**Expected behavior**
Expect non-null values to be returned.

**Additional context**
Parquet file generated by Spark:  

[alltypes_extended_plain.parquet.zip](https://github.com/user-attachments/files/18577979/alltypes_extended_plain.parquet.zip)

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 39 (38 by maintainers)

Most upvoted comments

Thanks @etseidl. The issue is in the ExampleParquetWriter (from parquet-java) used here in Spark unit tests : https://github.com/apache/spark/blob/ece14704cc083f17689d2e0b9ab8e31cf71a7a2d/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala#L871 (Spark itself writes only signed integers but will read the file written by the ExampleParquetWriter successfully). I’ll log an issue in Parquet-java and we can follow up there.

I’ve logged an issue in parquet-java already (https://github.com/apache/parquet-java/issues/3142). I logged it only for the ExampleParquetWriter but the issue is not in the example - it is just a thin wrapper around the group ParquetWriter apis.

I guess I got off on a bit of a tangent here. I was asking not so much about handling the bad data, but rather the serendipitous finding that the existing code is quite slow doing casts from 32 bits down to 8 or 16. That can be raised in a separate issue.

Yes 100% . Filed this one to start the conversation

Now I’m conflicted. On the one hand, sentiment in the Parquet community seems to be heading towards returning an error for data that is malformed as in this issue. On the other, I’ve done some benchmarking of the fix vs the current code and have found that the fallible cast currently used is up to 2X slower than using the infallible casting as in #7055. This is also true for signed INT8 and INT16, but not for UINT32.

@tustvold and @alamb, what do you think of special casing casts to truncated integer types? Is the speedup worth the extra code? And how do we then handle detecting malformed data and returning an error? Would we add a validation step for 8 and 16 bit integers?

At least for C++ the writes, do not appear to actually number of bit written but instead do a cast:

https://github.com/apache/arrow/blob/f8723722341df31c0091c91ec451319ded58c214/cpp/src/parquet/column_writer.cc#L1854

Read side fo C++ just does the oppositve operation, so it isn’t masking explicitly just relying on the down-cast behavior

Fair enough. I will track down where the issue originates and log an issue with the appropriate project. Also, updating the issue title to reflect this is not a bug but a compatibility issue.

I would suggest filling a bug against whatever produced the file, as I do think it is incorrect to produce such a file. I’d also be curious if the statistics are also incorrect.

Once we have some consensus on what is correct, I’m fine with working around the bug by masking, but I think it sensible to get feedback from the broader community first. Perhaps there is something we are missing