arrow-rs: Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow`

Hello,

internally, we wrote an own library that wraps arrow-rs to make it useable from Python. Such a thing also exists publicly available through arro3 which I used here for some minimal reproducible example:

import pyarrow.parquet
import arro3.io

data = [8388855, 8388924, 8388853, 8388880, 8388876, 8388879]

schema = pyarrow.schema([pyarrow.field("html", pyarrow.binary())])
data = [{"html": b"0" * d} for d in data]

t = pyarrow.Table.from_pylist(data, schema=schema)

path = "/tmp/foo.parquet"
with open(path, "wb") as file:
    for b in t.to_batches():
        arro3.io.write_parquet(b, file, max_row_group_size=len(data) - 3)

reader = pyarrow.parquet.ParquetFile(path)
for i in range(2):
    print(len(reader.read_row_group(i)))

This code writes a bit of dummy binary data through arrow-rs. Reading that with pyarrow results in

  File "pyarrow/_parquet.pyx", line 1655, in pyarrow._parquet.ParquetReader.read_row_group
  File "pyarrow/_parquet.pyx", line 1691, in pyarrow._parquet.ParquetReader.read_row_groups
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
OSError: Couldn't deserialize thrift: No more data to read.
Deserializing page header failed.

Observations

  • Reading in the same file through arro3 or own internal library wrapping arrow-rs works just fine
  • Reading in the same file through duckdb also works just fine
  • Reducing the amount of binary data per row slightly leads to the error disappearing (8_388_855 per row or 25_166_565 per row group or more seems to be the problematic amount)
  • Issue is reproducible with pyarrow version 18.1.0, 19.0.1 and 20.0.0

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (16 by maintainers)

Commits related to this issue

Most upvoted comments

@etseidl @alamb it seems like the issue is fixed now! πŸ˜ƒ Now that https://github.com/apache/arrow-rs/pull/7555 is merged, I checked out arrow and parquet with the recent git version:

@etseidl does great work πŸ˜ƒ

The file sizes seem like there really should be a statistics_truncate_length default somewhere below or equal to 1024 or so.

Yes I agree – here is a PR that proposes defaulting it to 64 (to match page stats). Any thoughts?

@alamb a slightly offtopic question: Comparing EnabledStatistics CHUNK and PAGE (with statistics_truncate_length == None) it seems that in PAGE mode compared to CHUNK mode, the file is 3 times the size?

Yes, it is currently 3 times the size. As @etseidl mentions, we have a plan to fix this and it is described in more detail at

Yes I agree – here is a PR that proposes defaulting it to 64 (to match page stats). Any thoughts?

I don’t have an opinion on what should be the default πŸ˜„ Whatever other tools do I guess πŸ˜…

I think this issue could be closed now, right? πŸ˜ƒ The only remaining issue is purely on the pyarrow side, but also not super relevant, since files with so huge, untruncated statistics header are overblown massively anyways.

Sounds good – let’s call this closed by

Thanks again for all the help reporting and testing this @jonded94 – I think everyone will benefit from the result

Yes I agree – here is a PR that proposes defaulting it to 64 (to match page stats). Any thoughts?

I don’t have an opinion on what should be the default πŸ˜„ Whatever other tools do I guess πŸ˜…

I think this issue could be closed now, right? πŸ˜ƒ The only remaining issue is purely on the pyarrow side, but also not super relevant, since files with so huge, untruncated statistics header are overblown massively anyways.

Setting PAGE also enables writing of the page index, so there’s the 3X.

I have a PR (#7594) that will update the EnabledStatistics::Page documentation to explicitly state this.

@etseidl @alamb it seems like the issue is fixed now! πŸ˜ƒ Now that #7555 is merged, I checked out arrow and parquet with the recent git version:

arrow = { git = "https://github.com/apache/arrow-rs.git", rev = "0ae9f66d10141c8d5054fd77f73168c7a2ea2819", features = ["pyarrow"] }
parquet = { git = "https://github.com/apache/arrow-rs.git", rev = "0ae9f66d10141c8d5054fd77f73168c7a2ea2819", features = ["async"] }

In the previously linked pytest test, I added cases where I truncate statistics to 7Mi (slightly below the threshold after which pyarrow crashes on reads, at least with EnabledStatistics == PAGE) and also 8Mi (leading to pyarrow crashes).

Not only are file sizes now actually changing, but the pyarrow crashes are exactly as expected (i.e. only for page level statistics longer than 8MiB per value)! 😁

Here is an overview over file sizes & pyarrow crashes for generated test files:

EnabledStatistics statistics_truncate_length size [bytes] fails with pyarrow
None None 787344 No
Chunk 1024 789415 No
Chunk 7Mi 15467435 No
Chunk 8Mi 17564587 No
Chunk None 34341803 No
Page 1024 793552 No
Page 7Mi 44827620 No
Page 8Mi 51119076 Yes
Page None 101450724 Yes

The file sizes seem like there really should be a statistics_truncate_length default somewhere below or equal to 1024 or so.

@jonded94 could you try your test with the branch in #7555? I’m hoping that will at least allow the workaround to work. Then we can see about truncating statistics by default.

I think I found it…you say pyarrow is failing because the page headers are too large. So this implicates the page stats rather than chunk or column index stats. I’m looking at the page writing code and it seems the page stats do not perform truncation (but I may be missing something). https://github.com/apache/arrow-rs/blob/72c72941e2d6caf9349399882f0a175ffdbc3b5d/parquet/src/column/writer/mod.rs#L963-L980

I’ll try writing some tests for this.