arrow: [Python] Reading partial data/first block hangs on some cloud filesystems

Describe the bug, including details regarding any error messages, version, and platform.

Take the following example using a publicly available dataset:

import gcsfs
from pyarrow.dataset import dataset

# without fsspec filesystem, get segmentation fault
fs = None 
# with fsspec filesystem, hangs and never finishes
# fs = gcsfs.GCSFileSystem()

uri = "gs://datachain-demo/laion-aesthetics-csv/laion_aesthetics_1024_33M_1.csv"
ds = dataset(uri, format="csv", filesystem=fs)
print(ds.head(5))

As noted in the comments, depending on which filesystem is passed, it will either hang indefinitely or hit a segmentation fault. Strangely, s3 paths work (don’t hang or fail) with the pyarrow filesystem but hang with the fsspec s3fs filesystem. Local paths work with either the pyarrow or fsspec filesystem.

Other findings:

  • Similar operations like ds.take() and next(ds.to_batches()) have the same behavior as ds.head()
  • ds.head(use_threads=False) completes successfully with any filesystem but takes much longer

Component(s)

Python

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 1
  • Comments: 17 (11 by maintainers)

Most upvoted comments

I do not exactly get a hang on Python 3.12 and PyArrow 20.0, though it crashes at the end:

$ python ~/arrow/dev/issue_43497.py 


Schema from dataset:
URL: string
TEXT: string
WIDTH: double
HEIGHT: double
similarity: double
punsafe: double
pwatermark: double
AESTHETIC_SCORE: double
hash: int64

Data from dataset:
                                                 URL                                               TEXT   WIDTH  ...  pwatermark  AESTHETIC_SCORE                 hash
0  https://endscan.com/media/36373/fb0bf7b2abe7ac...  View 47 photos of this 3 bed, 4 bath, and 2,49...  2080.0  ...    0.100449         5.040063   929872200875109155
1  https://static0.colliderimages.com/wordpress/w...                         john barrowman - photo #12  2011.0  ...    0.732896         5.544570  8338800302313723098
2  https://images.squarespace-cdn.com/content/v1/...  A black and white limited edition portraits of...  1655.0  ...    0.081598         5.969655  7578604913656441916

[3 rows x 9 columns]
Fatal Python error: PyGILState_Release: auto-releasing thread-state, but no thread-state for this thread
Python runtime state: finalizing (tstate=0x0000000000ba6ac8)

Abandon

And here is the given thread’s backtrace:

#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x0000771a9f84527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x0000771a9f8288ff in __GI_abort () at ./stdlib/abort.c:79
#5  0x00000000004b0f23 in ?? ()
#6  0x00000000004b25d9 in _Py_FatalErrorFunc ()
#7  0x00000000004b2aff in ?? ()
#8  0x0000771a9cc9f5db in arrow::py::PyAcquireGIL::release() [clone .part.0] () from /home/antoine/t/venv-3.12/lib/python3.12/site-packages/pyarrow/libarrow_python.so.2000
#9  0x0000771a9cc9f974 in decltype ({parm#1}()) arrow::py::SafeCallIntoPython<arrow::py::PyReadableFile::Read(long, void*)::{lambda()#1}>(arrow::py::PyReadableFile::Read(long, void*)::{lambda()#1}&&) [clone .cold] () from /home/antoine/t/venv-3.12/lib/python3.12/site-packages/pyarrow/libarrow_python.so.2000
#10 0x0000771a9cd18301 in arrow::py::PyReadableFile::Read(long, void*) () from /home/antoine/t/venv-3.12/lib/python3.12/site-packages/pyarrow/libarrow_python.so.2000
#11 0x0000771a99c34cbd in arrow::io::BufferedInputStream::Impl::Read(long, void*) () from /home/antoine/t/venv-3.12/lib/python3.12/site-packages/pyarrow/libarrow.so.2000
#12 0x0000771a99c358be in arrow::io::BufferedInputStream::Impl::Read(long) () from /home/antoine/t/venv-3.12/lib/python3.12/site-packages/pyarrow/libarrow.so.2000
#13 0x0000771a99c35a9d in arrow::io::internal::InputStreamConcurrencyWrapper<arrow::io::BufferedInputStream>::Read(long) ()
   from /home/antoine/t/venv-3.12/lib/python3.12/site-packages/pyarrow/libarrow.so.2000
#14 0x0000771a99c21e3d in arrow::Result<std::shared_ptr<arrow::Buffer> > arrow::Iterator<std::shared_ptr<arrow::Buffer> >::Next<arrow::io::(anonymous namespace)::InputStreamBlockIterator>(void*) () from /home/antoine/t/venv-3.12/lib/python3.12/site-packages/pyarrow/libarrow.so.2000
#15 0x0000771a99b23946 in arrow::BackgroundGenerator<std::shared_ptr<arrow::Buffer> >::WorkerTask(std::shared_ptr<arrow::BackgroundGenerator<std::shared_ptr<arrow::Buffer> >::State>) () from /home/antoine/t/venv-3.12/lib/python3.12/site-packages/pyarrow/libarrow.so.2000
#16 0x0000771a99b23f6d in arrow::internal::FnOnce<void ()>::FnImpl<arrow::BackgroundGenerator<std::shared_ptr<arrow::Buffer> >::State::DoRestartTask(std::shared_ptr<arrow::BackgroundGenerator<std::shared_ptr<arrow::Buffer> >::State>, arrow::util::Mutex::Guard)::{lambda()#1}>::invoke() ()
   from /home/antoine/t/venv-3.12/lib/python3.12/site-packages/pyarrow/libarrow.so.2000
#17 0x0000771a99d3a70c in std::thread::_State_impl<std::thread::_Invoker<std::tuple<arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::{lambda()#1}> > >::_M_run() ()
   from /home/antoine/t/venv-3.12/lib/python3.12/site-packages/pyarrow/libarrow.so.2000
#18 0x0000771a9dcecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#19 0x0000771a9f89caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#20 0x0000771a9f929c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78