arrow: [Python] Reading partial data/first block hangs on some cloud filesystems
Describe the bug, including details regarding any error messages, version, and platform.
Take the following example using a publicly available dataset:
import gcsfs
from pyarrow.dataset import dataset
# without fsspec filesystem, get segmentation fault
fs = None
# with fsspec filesystem, hangs and never finishes
# fs = gcsfs.GCSFileSystem()
uri = "gs://datachain-demo/laion-aesthetics-csv/laion_aesthetics_1024_33M_1.csv"
ds = dataset(uri, format="csv", filesystem=fs)
print(ds.head(5))
As noted in the comments, depending on which filesystem is passed, it will either hang indefinitely or hit a segmentation fault. Strangely, s3 paths work (don’t hang or fail) with the pyarrow filesystem but hang with the fsspec s3fs filesystem. Local paths work with either the pyarrow or fsspec filesystem.
Other findings:
- Similar operations like
ds.take()andnext(ds.to_batches())have the same behavior asds.head() ds.head(use_threads=False)completes successfully with any filesystem but takes much longer
Component(s)
Python
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 1
- Comments: 17 (11 by maintainers)
I do not exactly get a hang on Python 3.12 and PyArrow 20.0, though it crashes at the end:
And here is the given thread’s backtrace: