arrow: [Python] OSError: List index overflow.

Hello,

I am storing pandas dataframe as .parquet with pd.to_parquet and then try to load them back with pd.read_parquet. I am experiencing some error for which I do not find solution and would kindly ask for help to solve this please …

Here is the trace:

  File "/home/gnlzm/miniconda3/envs/antidoto/lib/python3.9/site-packages/pandas/io/parquet.py", line 493, in read_parquet
    return impl.read(
  File "/home/gnlzm/miniconda3/envs/antidoto/lib/python3.9/site-packages/pandas/io/parquet.py", line 240, in read
    result = self.api.parquet.read_table(
  File "/home/gnlzm/miniconda3/envs/antidoto/lib/python3.9/site-packages/pyarrow/parquet/__init__.py", line 2827, in read_table
    return dataset.read(columns=columns, use_threads=use_threads,
  File "/home/gnlzm/miniconda3/envs/antidoto/lib/python3.9/site-packages/pyarrow/parquet/__init__.py", line 2473, in read
    table = self._dataset.to_table(
  File "pyarrow/_dataset.pyx", line 331, in pyarrow._dataset.Dataset.to_table
  File "pyarrow/_dataset.pyx", line 2577, in pyarrow._dataset.Scanner.to_table
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: List index overflow.

If I store a small dataframe, I do not face this error. If I store a larger dataframe with e.g. 295.912.999 rows then I get this error.

However before saving it, I print the index range and it is bound in 0 295912998. Whether I save the .parquet with index=True or False gives the same error but I do not understand why there is an overflow on the bounded index …

Any hints are much appreciated, thanks !

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 3
  • Comments: 25 (14 by maintainers)

Most upvoted comments

Problem

Same here. I’m trying to load a 6GB parquet file with 3 cols two string cols and one with embeddings (arrays just like @aschmu described) in pandas with

 df = pd.read_parquet("test.parquet")
File size in bytes: 6207538015 bytes
File size in kilobytes: 6062048.84 KB

Tried with Python 3.11 and pandas 2.0.3 and latest 2.1.3 on Windows (32Gb RAM) and Ubuntu (128Gb RAM):

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
File <timed exec>:4

File ~/anaconda3/lib/python3.11/site-packages/pandas/io/parquet.py:509, in read_parquet(path, engine, columns, storage_options, use_nullable_dtypes, dtype_backend, **kwargs)
    506     use_nullable_dtypes = False
    507 check_dtype_backend(dtype_backend)
--> 509 return impl.read(
    510     path,
    511     columns=columns,
    512     storage_options=storage_options,
    513     use_nullable_dtypes=use_nullable_dtypes,
    514     dtype_backend=dtype_backend,
    515     **kwargs,
    516 )

File ~/anaconda3/lib/python3.11/site-packages/pandas/io/parquet.py:227, in PyArrowImpl.read(self, path, columns, use_nullable_dtypes, dtype_backend, storage_options, **kwargs)
    220 path_or_handle, handles, kwargs["filesystem"] = _get_path_or_handle(
    221     path,
    222     kwargs.pop("filesystem", None),
    223     storage_options=storage_options,
    224     mode="rb",
    225 )
    226 try:
--> 227     pa_table = self.api.parquet.read_table(
    228         path_or_handle, columns=columns, **kwargs
    229     )
    230     result = pa_table.to_pandas(**to_pandas_kwargs)
    232     if manager == "array":

File ~/anaconda3/lib/python3.11/site-packages/pyarrow/parquet/core.py:2973, in read_table(source, columns, use_threads, metadata, schema, use_pandas_metadata, read_dictionary, memory_map, buffer_size, partitioning, filesystem, filters, use_legacy_dataset, ignore_prefixes, pre_buffer, coerce_int96_timestamp_unit, decryption_properties, thrift_string_size_limit, thrift_container_size_limit)
   2962         # TODO test that source is not a directory or a list
   2963         dataset = ParquetFile(
   2964             source, metadata=metadata, read_dictionary=read_dictionary,
   2965             memory_map=memory_map, buffer_size=buffer_size,
   (...)
   2970             thrift_container_size_limit=thrift_container_size_limit,
   2971         )
-> 2973     return dataset.read(columns=columns, use_threads=use_threads,
   2974                         use_pandas_metadata=use_pandas_metadata)
   2976 warnings.warn(
   2977     "Passing 'use_legacy_dataset=True' to get the legacy behaviour is "
   2978     "deprecated as of pyarrow 8.0.0, and the legacy implementation will "
   2979     "be removed in a future version.",
   2980     FutureWarning, stacklevel=2)
   2982 if ignore_prefixes is not None:

File ~/anaconda3/lib/python3.11/site-packages/pyarrow/parquet/core.py:2601, in _ParquetDatasetV2.read(self, columns, use_threads, use_pandas_metadata)
   2593         index_columns = [
   2594             col for col in _get_pandas_index_columns(metadata)
   2595             if not isinstance(col, dict)
   2596         ]
   2597         columns = (
   2598             list(columns) + list(set(index_columns) - set(columns))
   2599         )
-> 2601 table = self._dataset.to_table(
   2602     columns=columns, filter=self._filter_expression,
   2603     use_threads=use_threads
   2604 )
   2606 # if use_pandas_metadata, restore the pandas metadata (which gets
   2607 # lost if doing a specific `columns` selection in to_table)
   2608 if use_pandas_metadata:

File ~/anaconda3/lib/python3.11/site-packages/pyarrow/_dataset.pyx:369, in pyarrow._dataset.Dataset.to_table()

File ~/anaconda3/lib/python3.11/site-packages/pyarrow/_dataset.pyx:2818, in pyarrow._dataset.Scanner.to_table()

File ~/anaconda3/lib/python3.11/site-packages/pyarrow/error.pxi:144, in pyarrow.lib.pyarrow_internal_check_status()

File ~/anaconda3/lib/python3.11/site-packages/pyarrow/error.pxi:115, in pyarrow.lib.check_status()

OSError: List index overflow.

The weird thing is that I processed 20 of these files with different file sizes and even bigger ones than this one (with 7GB) worked.

Workaround

UPDATE: Thanks a lot @aschmu for pointing in the right direction. I used polars too and convert to pandas so I can just continue with my normal workflow 😃

This worked like a charm for my big files!

# pip install polars 
import polars as pl
df = pl.read_parquet("test.parquet")
df = df.to_pandas()
# del df["__index_level_0__"] # if needed delete the polars specific column

FWIW I’ve had success using polars to read large parquet containing columns of lists of float64s and converting them to pandas afterwards when pd.read_parquet caused an error.

Same issue, fix is needed

Same issue , any updates?

Managed to reproduced this error from a dataset with a single column containing a list of integers.

  • to generate the dataset
import numpy as np
import pandas as pd

# total rows < max(int32)
n_rows = 108000000

# dataframe has only one column containing a list of 200 integers
# 200 * n_rows > max(int32)
data = [np.zeros(200, dtype='int8')] * n_rows

print('generating...')
df = pd.DataFrame()
# only one column
df['a'] = data

print('saving ...')
df.to_parquet('/tmp/pq')
print('done')
  • to load the dataset
import pandas as pd

print('loading...')
df = pd.read_parquet('/tmp/pq', use_threads=False)
print('size = {}'.format(df.shape))

Tested with pyarrow-9.0.0 and pandas-1.5. Loading dataset failed with OSError: List index overflow..

NOTE: loading the dataset leads to “out of memory kill” on a machine with 128G RAM. I have to test it on a 256G RAM machine.

Would iter_batches() as a workaround ok?

i found that iter_batches() actually works.

i’ve met the same issue while reading 800k vectors with dimension 2560. using pyarrow.parquet read_parquet will trigger same issue: OSError: List index overflow.

And i found using iter_batches to read parquet file while i was trying to take the subset of the data, it works.

here is corresponding code:

for batch in parquet_file.iter_batches(batch_size=batch_size):
    if total_rows >= rows_to_read:
        break
    table = pa.Table.from_batches([batch])
    result_tables.append(table.slice(0, min(batch_size, rows_to_read - total_rows)))
    total_rows += batch.num_rows