pandas: error in to_pickle

Hi, I think there is a bug in to_pickle

>>> data = pd.read_hdf(DG.load_path + 'time_series/PSI_TS_after_alignment_and_reshaping.h5', 'table')
>>> data.shape
(1006, 288095)
>>> data.to_pickle(DG.load_path + 'time_series/PSI_TS_after_alignment_and_reshaping2.pkl')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/donbeo/MyApps/phd_python/lib/python3.4/site-packages/pandas/core/generic.py", line 994, in to_pickle
    return to_pickle(self, path)
  File "/Users/donbeo/MyApps/phd_python/lib/python3.4/site-packages/pandas/io/pickle.py", line 14, in to_pickle
    pkl.dump(obj, f, protocol=pkl.HIGHEST_PROTOCOL)
OSError: [Errno 22] Invalid argument
>>> pd.version.version
'0.16.2'
>>>

About this issue

Original URL
State: closed
Created 9 years ago
Comments: 16 (11 by maintainers)

Most upvoted comments

I had the same problem, the same error for pickling a large file. I solved the issue using this code:

import joblib
joblib.dump(df,'df_path.pkl')

So maybe a pandas’ issue?

shadzic on Oct 26, 2017

probably not a pandas bug. check this out http://bugs.python.org/issue24658

kawochen on Jan 28, 2016

so I guess what is happening is the file is too big and raises an unhelpful exception

jreback on Jan 28, 2016