pandas: DEPR: Deprecate returning a DataFrame in Series.apply

Feature Type

Adding new functionality to pandas
Changing existing functionality in pandas
Removing existing functionality in pandas

Series.apply (more precisely SeriesApply.apply_standard) normally returns a Series, if supplied a callable, but if the callable returns a Series, it converts the returned array of Series into a Dataframe and returns that.

>>> small_ser = pd.Series(range(3))
>>> small_ser.apply(lambda x: pd.Series({"x": x, "x**2": x**2}))
   x  x**2
0  1     1
1  2     4
2  3     9

This approach is exceptionally slow and should be generally avoided. For example:

import pandas as pd
>>> ser = pd.Series(range(10_000))
>>> %timeit ser.apply(lambda x: pd.Series({"x": x, "x**2": x ** 2}))
658 ms ± 623 µs per loop

The result from the above method can in almost all cases be gotten faster by using other methods. For example can the above example be written much faster as:

>>> %timeit ser.pipe(lambda x: pd.DataFrame({"x": x, "x**2": x**2}))
80.6 µs ± 682 ns per loop

In the rare case where a fast approach isn’t possible, users should just construct the DataFrame manually from the list of Series instead of relying on the Series.apply method.

Also, allowing this construct complicates the return type of Series.Apply.apply_standard and therefore the apply method. By removing the option to return a DataFrame, the signature of SeriesApply.apply_standard will be simplified into simply Series.

All in all, this behavior is slow and makes things complicated without any benefit, so I propose to deprecate returning a DataFrame from SeriesApply.apply_standard.

About this issue

Original URL
State: closed
Created a year ago
Comments: 23 (15 by maintainers)

Most upvoted comments

The deprecation was reverted on 2.1.1

phofl on Oct 12, 2023

The doc string for json_normalize says that can be dict or list of dicts, so we can do:

>>> experiments = pd.DataFrame([{"train_config": {"num_epochs": 30, "patience": 10}, "model_config": {"num_layers": 3, "layer_dim": 256}}])
>>> pd.json_normalize(experiments.to_dict())
   train_config.0.num_epochs  train_config.0.patience  model_config.0.num_layers  model_config.0.layer_dim
0                         30                       10                          3                       256
>>>  pd.json_normalize(experiments["train_config"]). # single series
   num_epochs  patience
0          30        10

Does json_normalize fit your use case, @harrisonWilde?

topper-123 on Sep 11, 2023

I see now; I was thinking you were want to take non-duplicate keys and make them each columns regardless of the row they occurred on. Now it makes sense what you’re going for.

rhshadrach on Sep 7, 2023

Apologies, I meant to attach the message below too:

When I run the code above, I see these warnings:

...: FutureWarning: Returning a DataFrame from Series.apply when the supplied function returns a Series is deprecated and will be removed in a future version.
  train_configs = experiments["train_config"].apply(pd.Series)
...: FutureWarning: Returning a DataFrame from Series.apply when the supplied function returns a Series is deprecated and will be removed in a future version.
  model_configs = experiments["model_config"].apply(pd.Series)

This can be reproduced with version 2.1.0 of pandas and a dataframe of the following form:

experiments = pd.DataFrame([{"train_config": {"num_epochs": 30, "patience": 10}, "model_config": {"num_layers": 3, "layer_dim": 256}}])

A lot of the discourse online implies this is the “standard” approach to unpacking dictionaries in columns. I am now using this method, but wondered if there was a recommended workflow in light of the deprecation:

    experiments = experiments.join(
        pd.DataFrame(experiments.pop("train_config").values.tolist(), index=experiments.index)
    )
    experiments = experiments.join(
        pd.DataFrame(experiments.pop("model_config").values.tolist(), index=experiments.index)
    )

HarrisonWilde on Sep 5, 2023