pandas: Time Series Interpolation is wrong

@jreback @jbrockmendel @oricou @Jython1415 I independently became aware of this aspect. I think there is a relatively important issue here. Before I explain, I filed a PR that fixes how resample->interpolate in the case of upsampling works, here: https://github.com/pandas-dev/pandas/pull/56515 … But some failing test cases made me realize that the unreasonable behavior is treated as the “expected behavior” so far, so that fixing it would probably require a major version bump if we take semantic versioning seriously. Please read on.

The most important aspect in the code is the comments around the examples here: https://github.com/pandas-dev/pandas/blob/main/pandas/core/resample.py#L1064-L1080 … The case with 400ms resampling causes only 3 of the original data points to be included when the interpolation method is called. See this plot:

The blue points are the original example data (series). The green points are the 400ms resample-interpolate results with the current code. It is obvious that the green points are not a proper upsampling result for the blue reference points. However, the comment in the doc-string declares this as “desired” (or expected) behavior:

Internal reindexing with asfreq() prior to interpolation leads to an interpolated timeseries on the basis the reindexed timestamps (anchors). Since not all datapoints from original series become anchors, it can lead to misleading interpolation results […]

However, objectively nobody would “expect” this result, and it is correctly labeled as “misleading” in the comment. The purpose of such interpolation is precisely to either (a) increase the time resolution of a time series using some assumed underlying function (i.e. linear, polynomial, …) or to (b) align measurement data done at potentially arbitrary timestamps to a fixed grid. In neither case the current implementation leads to the expected results. It only works when the new desired sampling includes all of the known data points exactly. The orange points in the plot show you the interpolation result with my fixed code, achieved by first concatenating the non-covered data points with the new resampled index, then doing the interpolation, then discarding the excess data points. I’d argue this is the result most people would expect.

In the edge case, the current behavior leads to completely unreasonable results, such as:

index = pd.date_range('2000-01-01 00:01:00', periods=5, freq='2h')
df = pd.Series(np.arange(5.), index)
df.resample('1h').interpolate(method='linear')

printing

2000-01-01 00:00:00   NaN
2000-01-01 01:00:00   NaN
2000-01-01 02:00:00   NaN
2000-01-01 03:00:00   NaN
2000-01-01 04:00:00   NaN
2000-01-01 05:00:00   NaN
2000-01-01 06:00:00   NaN
2000-01-01 07:00:00   NaN
2000-01-01 08:00:00   NaN
Freq: H, dtype: float64

Here, a 1-second offset makes the upsampling lead to all-NaN results.

If I am given green light that it is okay to correct the interpolation behavior in version 2.x, I can fix the test cases which are based on hard-coded wrong expectations.

@MarcoGorelli Thank you for open ears and quick feedback 😃 Please note a PR is already there: https://github.com/pandas-dev/pandas/pull/56515 … I can target 3.0 instead (and adjust the implementation accordingly). But I want to stress that the behavior in 2.x is “wrong”, so it would be wise to carefully consider whether it could target 2.x to benefit users of pandas 2. What do you think?

die 12/06/23, ad 03h44, Joshua Shew @.***> dixit :

I am currently looking at this issue and the related pull request (#39886). Just wanted to make sure I’m not overlapping with anyone, however unlikely it may be.

You can work on it. You are most welcome!

Olivier.