pandas: pd.where OverflowError with large numbers

Code Sample, a copy-pastable example if possible

import pandas as pd
import numpy as np

df = pd.DataFrame([[1.0, 2e25],[np.nan, 0.1]])

# Works when applied to individual columns
print(df[0].where(pd.notnull(df[0]), None))
print(df[1].where(pd.notnull(df[1]), None))

# Breaks for whole dataframe
print(df.where(pd.notnull(df), None))

Problem description

The above code does not work with 1.0.0, but used to work with at least 0.25.0. Replacing large floats with pd.where breaks Running pd.where on a dataframe that contains large float values and the replacement value is of a different dtype throws OverflowError: int too big to convert:

    applied = getattr(b, f)(**kwargs)
  File "***\venv\lib\site-packages\pandas\core\internals\blocks.py", line 1426, in where
    return self._maybe_downcast(blocks, "infer")
  File "***\venv\lib\site-packages\pandas\core\internals\blocks.py", line 514, in _maybe_downcast
    return _extend_blocks([b.downcast(downcast) for b in blocks])
  File "***\venv\lib\site-packages\pandas\core\internals\blocks.py", line 514, in <listcomp>
    return _extend_blocks([b.downcast(downcast) for b in blocks])
  File "***\venv\lib\site-packages\pandas\core\internals\blocks.py", line 552, in downcast
    return self.split_and_operate(None, f, False)
  File "***\venv\lib\site-packages\pandas\core\internals\blocks.py", line 496, in split_and_operate
    nv = f(m, v, i)
  File "***\venv\lib\site-packages\pandas\core\internals\blocks.py", line 549, in f
    val = maybe_downcast_to_dtype(val, dtype="infer")
  File "***\venv\lib\site-packages\pandas\core\dtypes\cast.py", line 135, in maybe_downcast_to_dtype
    converted = maybe_downcast_numeric(result, dtype, do_round)
  File "***\venv\lib\site-packages\pandas\core\dtypes\cast.py", line 222, in maybe_downcast_numeric
    new_result = trans(result).astype(dtype)
OverflowError: int too big to convert

Replacing data one column at a time works.

Expected Output

Name: 1, dtype: float64
      0      1
0     1  2e+25
1  None    0.1

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line] INSTALLED VERSIONS

commit : None python : 3.7.3.final.0 python-bits : 64 OS : Windows OS-release : 10 machine : AMD64 processor : Intel64 Family 6 Model 142 Stepping 9, GenuineIntel byteorder : little LC_ALL : None LANG : None LOCALE : None.None pandas : 1.0.0 numpy : 1.16.2 pytz : 2018.9 dateutil : 2.8.0 pip : 10.0.1 setuptools : 39.1.0 Cython : None pytest : 4.4.0 hypothesis : None sphinx : 2.0.0 blosc : None feather : None xlsxwriter : None lxml.etree : 4.3.3 html5lib : None pymysql : None psycopg2 : None jinja2 : 2.11.1 IPython : None pandas_datareader: None bs4 : None bottleneck : None fastparquet : None gcsfs : None lxml.etree : 4.3.3 matplotlib : None numexpr : None odfpy : None openpyxl : 2.6.2 pandas_gbq : None pyarrow : None pytables : None pytest : 4.4.0 pyxlsb : None s3fs : None scipy : None sqlalchemy : 1.3.3 tables : None tabulate : None xarray : None xlrd : 1.2.0 xlwt : None xlsxwriter : None numba : None

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 16 (9 by maintainers)

Commits related to this issue

Most upvoted comments

@TomAugspurger adding the blocker tag as several users affected by this regression.