arrow: `pyarrow.parquet.DataDataset` memory leak when reading and exporting Pandas

Hi, I’m trying to process a Pandas DataFrame (stored as a Parquet dataset -multiple parts, snappy compression), but I can’t -> Python is being killed due to reaching memory limit.

Steps to reproduce

Sample data:

$ du -ach  *
0       _SUCCESS
11M     part-00000-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
10M     part-00001-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.8M    part-00002-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.8M    part-00003-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.8M    part-00004-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.8M    part-00005-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.8M    part-00006-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.7M    part-00007-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.7M    part-00008-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.7M    part-00009-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M    part-00010-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M    part-00011-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M    part-00012-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M    part-00013-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M    part-00014-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M    part-00015-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M    part-00016-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M    part-00017-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M    part-00018-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M    part-00019-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.5M    part-00020-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.4M    part-00021-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.4M    part-00022-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.4M    part-00023-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.4M    part-00024-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
7.7M    part-00025-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
6.4M    part-00026-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
6.3M    part-00027-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
261M    total

Python code:

>>> import pyarrow.parquet as pq
>>> dataset = pq.ParquetDataset('.')
>>> dataset.read_pandas()

Now the python process memory allocation reaches (in kilobytes):

$ ps -eo size,command | grep python
9064128 python

When the read is chained with an export to pandas, it results in reaching a memory limit and killing Python.

>>> import pyarrow.parquet as pq
>>> dataset = pq.ParquetDataset('.')
>>> dataset.read_pandas().to_pandas()
[1]    610 killed     python

Last known memory allocation size for Python before being killed is:

$ while sleep 1; do ps -eo size,command | grep " python$"; done
...
20018716 python

So, that means that pyarrow consumes 20GB worth of memory to process a 261M dataset.

In comparison, fastparquet is able to process the same dataset with Python comsuming about 3,6GB of memory:

>>> from glob import glob
>>> from fastparquet import ParquetFile
>>> paths = glob('*.parquet)
>>> pf = ParquetFile(paths)
>>> dataframe = pf.to_pandas()
$ ps -eo size,command | grep " python$"
3795128 python

Version info

>>> import sys, pyarrow; print(pyarrow.__version__, sys.version)
0.10.0 3.6.6 (default, Sep 13 2018, 16:29:18)
[GCC 8.2.1 20180831]

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Reactions: 5
  • Comments: 27 (15 by maintainers)

Most upvoted comments

Hello! Just want to mention. Today the whole day I was struggling with similar issue - Out of memory - during pyarrow.parquet.read_table() of Parquet file written just before this. And version of pyarrow was 0.14.0. The problem appeared only when I tried to read string column which is ID in fact. One million rows was ok, but for 2+ mln - was “Out of memory” At the end I found solution - just to update pyarrow to 0.15.0 and this problem gone 😃

Just bisected and the regression is between 0.14.1 pyarrow and 0.15.0 on Ubuntu 18.04.2 LTS but not on MacOS (so far).

I am reproducing with a script similar to this:

import pandas as pd
import os
import gi

path = ... # some data 70 cols, mixed strings and floats with one date. about 15k rows.

for i in range(1000):
    print(i)
    df = pd.read_parquet(path)

And I’m tracking memory usage with memory-profiler: mprof run script.py and then viewing the output with mprof plot mprof*.dat

I might open an issue for this …

I’m experiencing a very similar problem. I was looking to use parquet format to save 20gb pandas dataframe. It’s 55 millions rows and 187 columns with mostly very low cardinality (most have less than 10 000 categories) and lot’s of N/A.

In attempting to gets somme insight, I tested to write the pandas dataset to parquet via a table

table = pa.Table.from_pandas(df)
pq.write_table(table, "data.parquet", use_dictionary=True,compression='none')

After that, I tried to just load it in a table like this :

table2 = pq.read_table("data.parquet")

Even then, I almost reach the limit of memor, with arround 140Gb of memory, 46 Gb resident memory (I’m working on a server with 60 Gb of RAM).

Memory use before and after loading the file (I restart the kernel after the write to have a clean slate)

ps -eo size,command | grep python
543724
ps -eo size,command | grep python
139282440

The parquet file, even without compression is 2.5 Gb and with compresison it fall to 1.2 Gb, which is pretty impressive.

Has this issue been resolved?

I have an instance that has about 250 GB Ram. The instance crashed when I tried to import a 2.5 GB parquet file into my jupyter notebook from s3. I am not sure why it used up this much memory.

path_file = '...'
start = timeit.default_timer()
s3 = s3fs.S3FileSystem()
data_orig = pq.ParquetDataset(path_file, filesystem = s3).read_pandas().to_pandas()
stop = timeit.default_timer()
print('> Run Time - Load Data: ', stop - start, '\n')

I can say that I encountered Memory error when reading my parquet file in through pandas.read_parquet. The file was able to be saved just fine, but the next day when I came to work, I couldn’t read it in. The solution was to upgrade pyarrow from 0.14 to 0.16

pyarrow 0.12.0 uses far less memory for converting from Arrow to pandas in common cases now. Please re-run your workloads and let us know how it goes

We’re implementing further memory optimizations by direct reads to categorical which should hopefully appear in pyarrow 0.13.0