arrow: `pyarrow.parquet.DataDataset` memory leak when reading and exporting Pandas
Hi, I’m trying to process a Pandas DataFrame (stored as a Parquet dataset -multiple parts, snappy compression), but I can’t -> Python is being killed due to reaching memory limit.
Steps to reproduce
Sample data:
$ du -ach *
0 _SUCCESS
11M part-00000-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
10M part-00001-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.8M part-00002-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.8M part-00003-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.8M part-00004-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.8M part-00005-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.8M part-00006-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.7M part-00007-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.7M part-00008-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.7M part-00009-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M part-00010-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M part-00011-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M part-00012-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M part-00013-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M part-00014-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M part-00015-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M part-00016-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M part-00017-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M part-00018-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.6M part-00019-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.5M part-00020-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.4M part-00021-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.4M part-00022-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.4M part-00023-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
9.4M part-00024-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
7.7M part-00025-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
6.4M part-00026-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
6.3M part-00027-90655d30-aacc-469c-9c77-ba991a964c68-c000.snappy.parquet
261M total
Python code:
>>> import pyarrow.parquet as pq
>>> dataset = pq.ParquetDataset('.')
>>> dataset.read_pandas()
Now the python process memory allocation reaches (in kilobytes):
$ ps -eo size,command | grep python
9064128 python
When the read is chained with an export to pandas, it results in reaching a memory limit and killing Python.
>>> import pyarrow.parquet as pq
>>> dataset = pq.ParquetDataset('.')
>>> dataset.read_pandas().to_pandas()
[1] 610 killed python
Last known memory allocation size for Python before being killed is:
$ while sleep 1; do ps -eo size,command | grep " python$"; done
...
20018716 python
So, that means that pyarrow consumes 20GB worth of memory to process a 261M dataset.
In comparison, fastparquet is able to process the same dataset with Python comsuming about 3,6GB of memory:
>>> from glob import glob
>>> from fastparquet import ParquetFile
>>> paths = glob('*.parquet)
>>> pf = ParquetFile(paths)
>>> dataframe = pf.to_pandas()
$ ps -eo size,command | grep " python$"
3795128 python
Version info
>>> import sys, pyarrow; print(pyarrow.__version__, sys.version)
0.10.0 3.6.6 (default, Sep 13 2018, 16:29:18)
[GCC 8.2.1 20180831]
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Reactions: 5
- Comments: 27 (15 by maintainers)
Hello! Just want to mention. Today the whole day I was struggling with similar issue - Out of memory - during
pyarrow.parquet.read_table()of Parquet file written just before this. And version of pyarrow was 0.14.0. The problem appeared only when I tried to read string column which is ID in fact. One million rows was ok, but for 2+ mln - was “Out of memory” At the end I found solution - just to updatepyarrowto 0.15.0 and this problem gone 😃Just bisected and the regression is between 0.14.1 pyarrow and 0.15.0 on Ubuntu 18.04.2 LTS but not on MacOS (so far).
I am reproducing with a script similar to this:
And I’m tracking memory usage with memory-profiler: mprof run script.py and then viewing the output with mprof plot mprof*.dat
I might open an issue for this …
I’m experiencing a very similar problem. I was looking to use parquet format to save 20gb pandas dataframe. It’s 55 millions rows and 187 columns with mostly very low cardinality (most have less than 10 000 categories) and lot’s of N/A.
In attempting to gets somme insight, I tested to write the pandas dataset to parquet via a table
After that, I tried to just load it in a table like this :
Even then, I almost reach the limit of memor, with arround 140Gb of memory, 46 Gb resident memory (I’m working on a server with 60 Gb of RAM).
Memory use before and after loading the file (I restart the kernel after the write to have a clean slate)
The parquet file, even without compression is 2.5 Gb and with compresison it fall to 1.2 Gb, which is pretty impressive.
Has this issue been resolved?
I have an instance that has about 250 GB Ram. The instance crashed when I tried to import a 2.5 GB parquet file into my jupyter notebook from s3. I am not sure why it used up this much memory.
I can say that I encountered Memory error when reading my parquet file in through
pandas.read_parquet. The file was able to be saved just fine, but the next day when I came to work, I couldn’t read it in. The solution was to upgrade pyarrow from 0.14 to 0.16pyarrow 0.12.0 uses far less memory for converting from Arrow to pandas in common cases now. Please re-run your workloads and let us know how it goes
We’re implementing further memory optimizations by direct reads to categorical which should hopefully appear in pyarrow 0.13.0