arrow-rs: Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow`
Hello,
internally, we wrote an own library that wraps arrow-rs to make it useable from Python.
Such a thing also exists publicly available through arro3 which I used here for some minimal reproducible example:
import pyarrow.parquet
import arro3.io
data = [8388855, 8388924, 8388853, 8388880, 8388876, 8388879]
schema = pyarrow.schema([pyarrow.field("html", pyarrow.binary())])
data = [{"html": b"0" * d} for d in data]
t = pyarrow.Table.from_pylist(data, schema=schema)
path = "/tmp/foo.parquet"
with open(path, "wb") as file:
for b in t.to_batches():
arro3.io.write_parquet(b, file, max_row_group_size=len(data) - 3)
reader = pyarrow.parquet.ParquetFile(path)
for i in range(2):
print(len(reader.read_row_group(i)))
This code writes a bit of dummy binary data through arrow-rs. Reading that with pyarrow results in
File "pyarrow/_parquet.pyx", line 1655, in pyarrow._parquet.ParquetReader.read_row_group
File "pyarrow/_parquet.pyx", line 1691, in pyarrow._parquet.ParquetReader.read_row_groups
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
OSError: Couldn't deserialize thrift: No more data to read.
Deserializing page header failed.
Observations
- Reading in the same file through
arro3or own internal library wrappingarrow-rsworks just fine - Reading in the same file through
duckdbalso works just fine - Reducing the amount of binary data per row slightly leads to the error disappearing (8_388_855 per row or 25_166_565 per row group or more seems to be the problematic amount)
- Issue is reproducible with
pyarrowversion18.1.0,19.0.1and20.0.0
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (16 by maintainers)
@etseidl does great work π
Yes I agree β here is a PR that proposes defaulting it to 64 (to match page stats). Any thoughts?
Yes, it is currently 3 times the size. As @etseidl mentions, we have a plan to fix this and it is described in more detail at
Sounds good β letβs call this closed by
Thanks again for all the help reporting and testing this @jonded94 β I think everyone will benefit from the result
I donβt have an opinion on what should be the default π Whatever other tools do I guess π
I think this issue could be closed now, right? π The only remaining issue is purely on the
pyarrowside, but also not super relevant, since files with so huge, untruncated statistics header are overblown massively anyways.Setting
PAGEalso enables writing of the page index, so thereβs the 3X.I have a PR (#7594) that will update the
EnabledStatistics::Pagedocumentation to explicitly state this.@etseidl @alamb it seems like the issue is fixed now! π Now that #7555 is merged, I checked out
arrowandparquetwith the recent git version:In the previously linked
pytesttest, I added cases where I truncate statistics to 7Mi (slightly below the threshold after whichpyarrowcrashes on reads, at least withEnabledStatistics==PAGE) and also 8Mi (leading topyarrowcrashes).Not only are file sizes now actually changing, but the
pyarrowcrashes are exactly as expected (i.e. only for page level statistics longer than 8MiB per value)! πHere is an overview over file sizes &
pyarrowcrashes for generated test files:pyarrowThe file sizes seem like there really should be a
statistics_truncate_lengthdefault somewhere below or equal to 1024 or so.@jonded94 could you try your test with the branch in #7555? Iβm hoping that will at least allow the workaround to work. Then we can see about truncating statistics by default.
I think I found itβ¦you say pyarrow is failing because the page headers are too large. So this implicates the page stats rather than chunk or column index stats. Iβm looking at the page writing code and it seems the page stats do not perform truncation (but I may be missing something). https://github.com/apache/arrow-rs/blob/72c72941e2d6caf9349399882f0a175ffdbc3b5d/parquet/src/column/writer/mod.rs#L963-L980
Iβll try writing some tests for this.