pandas: BUG: to_parquet produces ValueError exception

Pandas version checks

I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
d = {"col1": [1, 2], "col2": [3, 4]}
df = pd.DataFrame(data=d)
bytes_data = df.to_parquet(engine="fastparquet")

Issue Description

From the documentation for to_parquet (https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html) we can read this behavior for the path argument (default value is None):

If None, the result is returned as bytes.

When I run the above I get this exception:

Traceback (most recent call last):
  File "c:\git\useful-snippets\cryptography\tmp.py", line 4, in <module>
    bytes_data = df.to_parquet(engine="fastparquet")
  File "C:\Users\serexj\Miniconda3\envs\cryptography\lib\site-packages\pandas\util\_decorators.py", line 211, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\serexj\Miniconda3\envs\cryptography\lib\site-packages\pandas\core\frame.py", line 2976, in to_parquet
    return to_parquet(
  File "C:\Users\serexj\Miniconda3\envs\cryptography\lib\site-packages\pandas\io\parquet.py", line 442, in to_parquet
    return path_or_buf.getvalue()
ValueError: I/O operation on closed file.

Expected Behavior

Instead of an exception, return the result as bytes, as the documentation states.

Installed Versions

INSTALLED VERSIONS

commit : 2e218d10984e9919f0296931d92ea851c6a6faf5 python : 3.10.9.final.0 python-bits : 64 OS : Windows OS-release : 10 Version : 10.0.19044 machine : AMD64 processor : Intel64 Family 6 Model 141 Stepping 1, GenuineIntel byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : Swedish_Sweden.1252

pandas : 1.5.3 numpy : 1.24.1 pytz : 2022.7.1 dateutil : 2.8.2 setuptools : 65.6.3 pip : 22.3.1 Cython : None pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : None IPython : None pandas_datareader: None bs4 : None bottleneck : None brotli : None fastparquet : 2023.1.0 fsspec : 2023.1.0 gcsfs : None matplotlib : None numba : None numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : None pyreadstat : None pyxlsb : None s3fs : None scipy : None snappy : None sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None xlwt : None zstandard : None tzdata : None None

About this issue

Original URL
State: closed
Created a year ago
Comments: 16 (3 by maintainers)

Most upvoted comments

@MSfinnet and @msSarriman yes, the error is exclusive to fastparquet. Pyarrow works.

bytes_data= df.to_parquet(engine="pyarrow") # works
bytes_data= df.to_parquet(engine="fastparquet") # fails

I’ll use pyarrow as a workaround.

johanrex on Mar 2, 2023

@msSarriman

Is there any workaround until this is resolved?

It’s kinda late response, but late is better then never 😃 so, here is my version of workaround:

def parquet(df: pd.DataFrame) -> BytesIO:
    data: BytesIO = BytesIO()
    
    orig_close = data.close
    data.close = lambda: None
    try:
        df.to_parquet(data, engine="fastparquet", index=False)
    finally:
        data.close = orig_close

    return data

dmkulazhenko on May 4, 2023

@MSfinnet

Note that using “memory://”, you are actually creating a filesystem in memory, so you aren’t utilizing actual files. The operations will not reach out of the memory heap of your application.

Still though, this is a workaround to the original issue.

msSarriman on Feb 28, 2023

@MSfinnet I am looking in memory operations, yes.

I have attempted something like the following, but didn’t test extensively:

import fsspec

df.to_parquet("memory://temp.parquet")
with fsspec.open("memory://temp.parquet", "rb") as f:
   response = f.read()

return response

msSarriman on Feb 28, 2023