pandas: pd.where OverflowError with large numbers
Code Sample, a copy-pastable example if possible
import pandas as pd
import numpy as np
df = pd.DataFrame([[1.0, 2e25],[np.nan, 0.1]])
# Works when applied to individual columns
print(df[0].where(pd.notnull(df[0]), None))
print(df[1].where(pd.notnull(df[1]), None))
# Breaks for whole dataframe
print(df.where(pd.notnull(df), None))
Problem description
The above code does not work with 1.0.0, but used to work with at least 0.25.0.
Replacing large floats with pd.where breaks
Running pd.where on a dataframe that contains large float values and the replacement value is of a different dtype throws OverflowError: int too big to convert
:
applied = getattr(b, f)(**kwargs)
File "***\venv\lib\site-packages\pandas\core\internals\blocks.py", line 1426, in where
return self._maybe_downcast(blocks, "infer")
File "***\venv\lib\site-packages\pandas\core\internals\blocks.py", line 514, in _maybe_downcast
return _extend_blocks([b.downcast(downcast) for b in blocks])
File "***\venv\lib\site-packages\pandas\core\internals\blocks.py", line 514, in <listcomp>
return _extend_blocks([b.downcast(downcast) for b in blocks])
File "***\venv\lib\site-packages\pandas\core\internals\blocks.py", line 552, in downcast
return self.split_and_operate(None, f, False)
File "***\venv\lib\site-packages\pandas\core\internals\blocks.py", line 496, in split_and_operate
nv = f(m, v, i)
File "***\venv\lib\site-packages\pandas\core\internals\blocks.py", line 549, in f
val = maybe_downcast_to_dtype(val, dtype="infer")
File "***\venv\lib\site-packages\pandas\core\dtypes\cast.py", line 135, in maybe_downcast_to_dtype
converted = maybe_downcast_numeric(result, dtype, do_round)
File "***\venv\lib\site-packages\pandas\core\dtypes\cast.py", line 222, in maybe_downcast_numeric
new_result = trans(result).astype(dtype)
OverflowError: int too big to convert
Replacing data one column at a time works.
Expected Output
Name: 1, dtype: float64
0 1
0 1 2e+25
1 None 0.1
Output of pd.show_versions()
[paste the output of pd.show_versions()
here below this line]
INSTALLED VERSIONS
commit : None python : 3.7.3.final.0 python-bits : 64 OS : Windows OS-release : 10 machine : AMD64 processor : Intel64 Family 6 Model 142 Stepping 9, GenuineIntel byteorder : little LC_ALL : None LANG : None LOCALE : None.None pandas : 1.0.0 numpy : 1.16.2 pytz : 2018.9 dateutil : 2.8.0 pip : 10.0.1 setuptools : 39.1.0 Cython : None pytest : 4.4.0 hypothesis : None sphinx : 2.0.0 blosc : None feather : None xlsxwriter : None lxml.etree : 4.3.3 html5lib : None pymysql : None psycopg2 : None jinja2 : 2.11.1 IPython : None pandas_datareader: None bs4 : None bottleneck : None fastparquet : None gcsfs : None lxml.etree : 4.3.3 matplotlib : None numexpr : None odfpy : None openpyxl : 2.6.2 pandas_gbq : None pyarrow : None pytables : None pytest : 4.4.0 pyxlsb : None s3fs : None scipy : None sqlalchemy : 1.3.3 tables : None tabulate : None xarray : None xlrd : 1.2.0 xlwt : None xlsxwriter : None numba : None
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 16 (9 by maintainers)
Commits related to this issue
- test for pd.where overflow error #31687 — committed to mliu08/pandas by mliu08 2 years ago
@TomAugspurger adding the blocker tag as several users affected by this regression.