pandas: BUG: Unclear FutureWarning regarding inplace iloc setitem

I’m not sure that’s a sufficient change @jbrockmendel because df.isetitem will not work in that case and there is no corresponding df.setitem. I also suspect it’s possible to get into a situation where if you do df[col] = data you get a SettingWithCopyWarning telling you to do df.loc[:, col] = data, but then when you do df.loc[:, col], you’ll get a warning telling you to do df[col] = data. I think people will find that confusing.

What’s the difference between inplace and setting a new array? Neither the warning, nor the release notes makes this very clear to me.

I have a script that runs df.loc[i:ii, c] = newvals in a loop over a number of dataframes and only some of the cases emits this warning, but in all cases values are set correctly.

I also don’t see how either df[df.columns[i]] = newvals or, df.isetitem(i, newvals) can be a substitute for either df.loc[i:ii, c] or df.iloc[i:ii, c]

… as I read the discussion here and in the PR, there doesn’t really seem to be a consensus about what to do with this warning. (Remove it, change it, …)

They must undo this warning because it creates highly undesirable consequences.

Next, they should book and complete an Ayahuasca therapy session because they obviously cannot resolve this issue at their current level of conciousness and understanding.

this issue is quite annoying…

@max0x7ba just out of curiosity, how much are you paying for this massively complex, 3.5k-issue, OSS that you are totally entitled to SLAs and world-class service on?

It’s a warning. Relax or find a different tool.

What’s the difference between inplace and setting a new array? Neither the warning, nor the release notes makes this very clear to me.

Definitely open to suggestions for better wording. Let me try to explain using the OP example:

import numpy as np
import pandas as pd

values = np.arange(4).reshape(2, 2)

df = pd.DataFrame(values, columns=["a", "b"])

At this point the DataFrame df is directly backed by the original values, so doing something like values[0, 0] = 99 would affect the values in df.

There are types of setting (e.g. df.iloc[0,0] = 11) that will be “inplace” and will edit the original values, and others (e.g. df["B"] = 42) that will create a new array and NOT edit the original values.

Unfortunately, because of [reasons], the existing behavior is not super-consistent in when we do inplace vs not-inplace. That’s why this particular case is deprecated, so in 2.0 we can make the behavior more consistent.

In particular, in your case:

new = np.array([10, 11]).astype(np.int16)
df.loc[:, "a"] = new  # <- issues the warning about future behavior changing

In the current behavior, df.loc[:, "a"] = new is NOT inplace, but in the future it will be. The warning here is just in case you really care about inplace-vs-not, to keep the old behavior you need to do df["a"] = new instead.

Is that helpful? If you have suggestions to make the warning or docs clearer, please let us know.

I’m getting this warning indirectly when I call df.update(). I believe it does need to fixed in pandas, at least in that spot.

Actually even looking at the release notes, I don’t think I understand exactly what is deprecated here. The relevant section starts with

Most of the time setting values with DataFrame.iloc() attempts to set values inplace, only falling back to inserting a new array if necessary.

But then it says

This behavior is deprecated. In a future version, setting an entire column with iloc will attempt to operate inplace.

Isn’t this what already happens (“most of the time”?) And if it’s about different dtypes, how will that work? Do you mean to say it will coerce the dtype when setting the data?

Closed by #50044

As an aside, I’d suggest enforcing the code of conduct more and banning toxic users, there’s nothing to be gained by having them around and they disincentivise people from contributing

@max0x7ba please refrain from such comments (the second paragraph), that’s not helpful at all

When a commit causes totally unnecessary consternation from the users, but the author of the commit doesn’t take any responsibility and keeps dragging his feet for 4 months by not resolving the otherwise unnecessary problem, you don’t get to choose the feedback you get, do you?

Get real and fix the problem promptly, please.

@jbrockmendel Yes, that is what I ended up doing, but that’s not what I was hoping. To the best of my knowledge, other warnings from pandas are things that I can take direct action upon to correct a potentially dangerous situation, or to prepare for a future change - making the appropriate changes in these cases silences the warnings.

In this case, it seems more informative than a true warning, at least in the case where I am OK with the setting being in-place in the future (which I am). I am OK with using the warnings filter, but I was hoping for some way to suppress it at the pandas level so downstream users don’t potentially see it as well.

Making it a deprecation warning is fine, imo. Then, as package developer, I can supress it from my logs once I have verified it’s not a problem and users won’t be bothered. I think the main problem right now is that end users get spammed with this warning for code that they have no control over and that might be perfectly fine.

yeah, you could use with tm.assert_produces_warning(None) for that (we should really run the whole test suite with -W error, but we’re not there yet, so for now let’s explicitly assert no warning is raised)

I’m getting this warning indirectly when I call df.update(). I believe it does need to fixed in pandas, at least in that spot.

Me too, df.update() does:

self.loc[:, col] = expressions.where(mask, this, that)

The warning message essentially says that DataFrame.update currently does not update but will in the future versions, which doesn’t sound right.