pandas: BUG: df.fillna(dict, inplace=True) triggers FutureWarning in 1.5.0rc0 when DataFrame is empty

Pandas version checks

I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
df = pd.DataFrame(data=[[None, None], [None,None]], columns=['A', 'B'])
df.fillna({'A': 1, 'B': 2}, inplace=True)

Issue Description

With the most recent development version 1.5.0rc0, I see that the code example raises the following warning:

FutureWarning: In a future version, df.iloc[:, i] = newvalswill attempt to set the values inplace instead of always setting a new array. To retain the old behavior, use eitherdf[df.columns[i]] = newvalsor, if columns are non-unique,df.isetitem(i, newvals)``

The df. fillna() API should not use code that is ambiguous or deprecated.

Expected Behavior

df.fillna() should do its job, silently.

Installed Versions

INSTALLED VERSIONS ------------------ commit : 224458ee25d92ccdf289d1ae2741d178df4f323e python : 3.8.13.final.0 python-bits : 64 OS : Darwin OS-release : 21.6.0 Version : Darwin Kernel Version 21.6.0: Wed Aug 10 14:25:27 PDT 2022; root:xnu-8020.141.5~2/RELEASE_X86_64 machine : x86_64 processor : i386 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : None.UTF-8 pandas : 1.5.0rc0 numpy : 1.23.2 pytz : 2021.1 dateutil : 2.8.2 setuptools : 56.0.0 pip : 22.2.2 Cython : None pytest : 7.1.2 hypothesis : None sphinx : 1.8.6 blosc : None feather : None xlsxwriter : None lxml.etree : 4.9.1 html5lib : None pymysql : None psycopg2 : None jinja2 : 2.11.3 IPython : 7.21.0 pandas_datareader: None bs4 : None bottleneck : None brotli : None fastparquet : None fsspec : None gcsfs : None matplotlib : 3.4.1 numba : None numexpr : None odfpy : None openpyxl : 3.0.10 pandas_gbq : None pyarrow : None pyreadstat : None pyxlsb : None s3fs : None scipy : None snappy : None sqlalchemy : 1.4.40 tables : None tabulate : 0.8.10 xarray : None xlrd : None xlwt : None zstandard : None tzdata : None

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 19 (7 by maintainers)

Most upvoted comments

I confirm that I am not seeing the above warning after many tests. And, once I took care of some of the warnings related to some depreciated code, all worked perfectly with 2.0.0rc0. Thank you.

abedhammoud on Feb 24, 2023

thanks for checking!

MarcoGorelli on Feb 24, 2023

@MarcoGorelli the above warning is not happening with 2.0.0rc0. However, I am seeing other issues that I will report separately. Thank you.

abedhammoud on Feb 22, 2023

Hey @marekro

Thanks for having reported this - there’s now a release candidate for pandas 2.0.0, which you can install with

mamba install -c conda-forge/label/pandas_rc pandas==2.0.0rc0

python -m pip install --upgrade --pre pandas==2.0.0rc0

If you try it out and report bugs, then we can fix them before the final 2.0.0 release

MarcoGorelli on Feb 22, 2023

I’m using pandas 1.5.2 on MacOS 13.0.1, with Python@3.11.0 from Homebrew, and I can consistently reproduce the bug with the code below:

import pandas as pd

df = pd.DataFrame(data={'col1': ['a', pd.NA, 'c']}, index=range(3))
df = df.convert_dtypes()
df.fillna({'col1': '<NA>'}, inplace=True)

Three conditions together for the bug:

'col1' is object dtype at first, not int or float.
Call the convert_dtypes() method.
Set inplace=True in fillna().

Maybe some problem associated with string dtype?

And I also investigate other circumstances.

Using np.nan instead of pd.NA the problem will still be there.

Create a Series using the same list, and using fillna() method of Series will not trigger the bug. Codes below:

# No bug for Series.
s = pd.Series(['a', pd.NA, 'b'])
s = s.convert_dtypes()
s.fillna('<NA>', inplace=True)

Maybe some help.

caoyu-yiyue on Nov 30, 2022