pandas: BUG: Change of behavior in casting of datetime-like types in MultiIndex

However, I would also have to check for an inferred type of mixed, for when my column of datetime.date objects contains null dates, right?

I’d go for lib.infer_dtype(col, skipna=True) == "date" instead of checking for “mixed”

I guess we could maybe pass a convert_dates parameter through to the Categorical constructor from the MultiIndex constructor. @jbrockmendel wdyt?

It’s possible. Though we’d then have a breaking change for anyone relying on the 1.3 behavior.

Would it be to call pd.to_datetime(arg, errors=‘ignore’) where arg takes every column of the DataFrame

I’d check Index(col).inferred_type == "date"

in 1.2.5…

pd.Index(
    [
        datetime.date(2021, 8, 1),
        datetime.date(2021, 8, 2),
        datetime.date(2021, 8, 3),
    ]
)

gives

Index([2021-08-01, 2021-08-02, 2021-08-03], dtype='object')

and using the DataFrame from the OP

df.set_index(["date"]).index

gives

Index([2021-08-01, 2021-08-02, 2021-08-03], dtype='object', name='date')

whereas for a MultiIndex

arr = [
    datetime.date(2021, 8, 1),
    datetime.date(2021, 8, 2),
    datetime.date(2021, 8, 3),
]
pd.MultiIndex.from_arrays([arr, arr]).levels[0]

gives

DatetimeIndex(['2021-08-01', '2021-08-02', '2021-08-03'], dtype='datetime64[ns]', freq=None)

So the Index and MultiIndex constructors were inconsistent in the handling of object dtype arrays containing datetime objects in pandas 1.2.5.

As initially stated, «I fail to find in the What’s new page the reason for that change of behavior». Would you mind pointing it out to me?

The change of behavior in casting of datetime-like types in MultiIndex was done in #38552. Looking at the code changes in that PR, it is clear from the changed tests and comments added that this change was intentional. Unfortunately the release note added did not refer to changes in MultiIndex construction.

Anyway, the version policy clearly states that «API breaking changes should only occur in major releases», and «a deprecation path will be provided rather than an outright breaking change». Wouldn’t that be the case here, since we are talking about a breaking change that occurred between versions 1.2.5 and 1.3.0?

The policy also states

pandas will sometimes make behavior changing bug fixes, as part of minor or patch releases. Whether or not a change is a bug fix or an API-breaking change is a judgement call. We’ll do our best, and we invite you to participate in development discussion on the issue tracker or mailing list.

So the change in behavior could be considered a bug fix, since the MultiIndex constructor was inconsistent with the Index constructor and no further action.

However, the policy also states

Whenever possible, a deprecation path will be provided rather than an outright breaking change.

and

We will not introduce new deprecations in patch releases.

So, as an alternative, we could maybe restore the old behavior for 1.3.5 and add a deprecation of this behavior in 1.4

The only code change in #38552 was removing convert_dates=True from values = maybe_infer_to_datetimelike(values, convert_dates=True)

I guess we could maybe pass a convert_dates parameter through to the Categorical constructor from the MultiIndex constructor. @jbrockmendel wdyt?

@jgmarcel would likely take a community pull request

core can provide review

might be quite tricky as date have very little support

Thanks @jgmarcel for the report.

first bad commit: [545a942424a26c4163e1f959ac6130984fc3fb41] BUG: Index([date]).astype(“category”).astype(object) roundtrip (#38552)

I’ll mark as a regression for now pending further investigation.

Note that the set_index followed by a reset_index still creates a datetime64[ns] column from the original object column of date objects.

cc @jbrockmendel