pandas: BUG:to_pickle() raises TypeError when compressing large dataframe

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample

import pandas as pd

df = pd.DataFrame(range(100000))
df.to_pickle("df.pkl.xz", protocol=5)

Problem description

the above code raises TypeError

TypeError                                 Traceback (most recent call last)
<ipython-input-220-8c4b6e9cecfc> in <module>
      2 
      3 df = pd.DataFrame(range(100000))
----> 4 df.to_pickle("df.pkl.xz", protocol=5)

/python3.9/site-packages/pandas/core/generic.py in to_pickle(self, path, compression, protocol, storage_options)
   2859         from pandas.io.pickle import to_pickle
   2860 
-> 2861         to_pickle(
   2862             self,
   2863             path,

/python3.9/site-packages/pandas/io/pickle.py in to_pickle(obj, filepath_or_buffer, compression, protocol, storage_options)
     95         storage_options=storage_options,
     96     ) as handles:
---> 97         pickle.dump(obj, handles.handle, protocol=protocol)  # type: ignore[arg-type]
     98 
     99 

/python3.9/lzma.py in write(self, data)
    232         compressed = self._compressor.compress(data)
    233         self._fp.write(compressed)
--> 234         self._pos += len(data)
    235         return len(data)
    236 

TypeError: object of type 'pickle.PickleBuffer' has no len()

Note that in order to reproduce the bug, it have to:

• the data frame is large enough
• it is using compressing (“.xz" or ".zip”)
• it is using latest pickle protocol (default). set protocol to 4 explicity will workaround this bug

Expected Output

the pickle should save ok

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 3e89b4c4b1580aa890023fc550774e63d499da25 python : 3.9.0.final.0 python-bits : 64 OS : Darwin OS-release : 19.6.0 Version : Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 machine : x86_64 processor : i386 byteorder : little LC_ALL : None LANG : zh_CN.UTF-8 LOCALE : zh_CN.UTF-8

pandas : 1.2.0 numpy : 1.19.4 pytz : 2020.5 dateutil : 2.8.1 pip : 20.0.2 setuptools : 45.2.0 Cython : 0.29.21 pytest : 6.2.1 hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : 4.6.2 html5lib : None pymysql : None psycopg2 : None jinja2 : 2.11.2 IPython : 7.19.0 pandas_datareader: 0.9.0 bs4 : None bottleneck : None fsspec : None fastparquet : None gcsfs : None matplotlib : 3.3.3 numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : None pyxlsb : None s3fs : None scipy : 1.5.4 sqlalchemy : None tables : None tabulate : 0.8.7 xarray : None xlrd : 2.0.1 xlwt : None numba : None