arrow: [Python] Arrow to Python list (to_pylist) conversion is slow

It seems that we are 20x slower than Numpy for converting the exact same data to a Python list.

With integers:


>>> arr = np.arange(0,1000, dtype=np.int64)
>>> %timeit arr.tolist()
8.24 µs ± 3.46 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
>>> parr = pa.array(arr)
>>> %timeit parr.to_pylist()
218 µs ± 2.39 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

With floats:


>>> arr = np.arange(0,1000, dtype=np.float64)
>>> %timeit arr.tolist()
10.2 µs ± 25.5 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
>>> parr = pa.array(arr)
>>> %timeit parr.to_pylist()
199 µs ± 1.04 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Reporter: Antoine Pitrou / @pitrou

Related issues:

Note: This issue was originally created as ARROW-12976. Please see the migration documentation for further details.

About this issue

Most upvoted comments

Just started using PyArrow. Are there any updates/recommended practices on this issue, this would significantly speed up “iterating through rows” for me.