pandas: BUG: inconsistent `DataFrame.agg` behavoir when passing as kwargs `numeric_only=True`
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
Python 3.11.0 | packaged by conda-forge | (main, Oct 25 2022, 06:12:32) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.__version__
'1.5.1'
>>> df = pd.DataFrame({'a': [1, 2, 3], 'b': list('abc')})
>>> df.agg('mean', numeric_only=True)
a 2.0
dtype: float64
>>> df.agg(['mean', 'std'], numeric_only=True)
<stdin>:1: FutureWarning: ['b'] did not aggregate successfully. If any error is raised this will raise in a future version of pandas. Drop these columns/ops to avoid this warning.
a
mean 2.0
std 1.0
>>> df.agg(pd.DataFrame.mean, numeric_only=True)
<stdin>:1: FutureWarning: Calling Series.mean with numeric_only=True and dtype object will raise a TypeError in the future
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\frame.py", line 9329, in aggregate
result = op.agg()
^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 773, in agg
result = self.obj.apply(self.orig_f, axis, args=self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\frame.py", line 9555, in apply
return op.apply().__finalize__(self, method="apply")
^^^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 746, in apply
return self.apply_standard()
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 873, in apply_standard
results, res_index = self.apply_series_generator()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 889, in apply_series_generator
results[i] = self.f(v)
^^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 139, in f
return func(x, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\generic.py", line 11847, in mean
return NDFrame.mean(self, axis, skipna, level, numeric_only, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\generic.py", line 11401, in mean
return self._stat_function(
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\generic.py", line 11353, in _stat_function
return self._reduce(
^^^^^^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\series.py", line 4812, in _reduce
raise NotImplementedError(
NotImplementedError: Series.mean does not implement numeric_only.
>>>
Issue Description
I expect DataFrame.agg to accept kwargs arguments passed to the aggregation functions.
It works in a seemingly inconsistent way.
df.agg('mean', numeric_only=True) works as expected.
But passing multiple functions as string in a list does not work as expected as we still get a FutureWarning showing that the argument numeric_only=True was not correctly passed to either mean, or std, or both
df.agg(['mean', 'std'], numeric_only=True)
>>>
<stdin>:1: FutureWarning: ['b'] did not aggregate successfully. If any error is raised this will raise in a future version of pandas. Drop these columns/ops to avoid this warning.
a
mean 2.0
std 1.0
Even more surprising, using the function pd.DataFrame.mean raises a NotImplementedError for Series.mean
>>> df.agg(pd.DataFrame.mean, numeric_only=True)
<stdin>:1: FutureWarning: Calling Series.mean with numeric_only=True and dtype object will raise a TypeError in the future
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\frame.py", line 9329, in aggregate
result = op.agg()
^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 773, in agg
result = self.obj.apply(self.orig_f, axis, args=self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\frame.py", line 9555, in apply
return op.apply().__finalize__(self, method="apply")
^^^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 746, in apply
return self.apply_standard()
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 873, in apply_standard
results, res_index = self.apply_series_generator()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 889, in apply_series_generator
results[i] = self.f(v)
^^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 139, in f
return func(x, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\generic.py", line 11847, in mean
return NDFrame.mean(self, axis, skipna, level, numeric_only, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\generic.py", line 11401, in mean
return self._stat_function(
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\generic.py", line 11353, in _stat_function
return self._reduce(
^^^^^^^^^^^^^
File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\series.py", line 4812, in _reduce
raise NotImplementedError(
NotImplementedError: Series.mean does not implement numeric_only.
Expected Behavior
Using kwargs arguments with DataFrame.agg must be correcly passed to all aggregation function, whatever the way they are given (string name or method reference)
Installed Versions
pd.show_versions() INSTALLED VERSIONS
commit : 91111fd99898d9dcaa6bf6bedb662db4108da6e6 python : 3.11.0.final.0 python-bits : 64 OS : Windows OS-release : 10 Version : 10.0.19044 machine : AMD64 processor : Intel64 Family 6 Model 140 Stepping 1, GenuineIntel byteorder : little LC_ALL : None LANG : None LOCALE : fr_FR.cp1252
pandas : 1.5.1 numpy : 1.23.4 pytz : 2022.5 dateutil : 2.8.2 setuptools : 65.5.0 pip : 22.3 Cython : None pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : None IPython : None pandas_datareader: None bs4 : None bottleneck : None brotli : None fastparquet : None fsspec : None gcsfs : None matplotlib : None numba : None numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : None pyreadstat : None pyxlsb : None s3fs : None scipy : None snappy : None sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None xlwt : None zstandard : None tzdata : None
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (14 by maintainers)
Just opened a documentation issue for the discussed subject here #49528
Marking as a regression since users shouldn’t be seeing a FutureWarning in this case.
Thanks for the report! The way
df.aggcurrently works is it splits the DataFrame up into a collection of Series and applies the function(s) provided to each Series. I believe this is what gives rise to your errors here.We should be applying
numeric_only=Trueindf.aggprior to splitting the DataFrame into Series. We then also need to not passnumeric_only=Trueto the functions when they are called on the Series. This should then also fix the issue with passingDataFrame.mean.