pandas: BUG: crosstab(dropna=False) does not work like value_counts(dropna=False)
If one uses series.value_counts(dropna=False) it includes NaN values, but crosstab(series1, series2, dropna=False) does not include NaN values. I think of cross_tab kind of like 2-dimensional value_counts(), so this is surprising.
In [31]: x = pd.Series(['a', 'b', 'a', None, 'a'])
In [32]: y = pd.Series(['c', 'd', 'd', 'c', None])
In [33]: y.value_counts(dropna=False)
Out[33]:
d 2
c 2
NaN 1
dtype: int64
In [34]: pd.crosstab(x, y, dropna=False)
Out[34]:
c d
row_0
a 1 1
b 0 1
I believe what crosstab(..., dropna=False) really does is just include rows/columns that would otherwise have all zeros, but that doesn’t seem the same as dropna=False to me. I would have expected to see a row and column entry for NaN, something like:
c d NaN
row_0
a 1 1 1
b 0 1 0
NaN 1 0 0
Thoughts on this?
This is on current master (almost 0.17).
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Reactions: 11
- Comments: 19 (11 by maintainers)
The current (0.24) Pandas documentation should say dropna: “Do not include columns OR ROWS whose entries are all NaN”, because that is what the current behavior actually seems to be: when rows/columns are entirely empty, rows/columns are dropped with default dropna = True.
With dropna=False, empty rows show up as 0 value entries in columns, and empty columns show up as 0 values in rows. Previous example shows that NaN is apparently not considered to be a possible row/column value with dropna=False.
It seems reasonable to suggest that NaN be added to the set of possible values for both rows and columns in the dataframe created when crosstab has dropna=False set by a user. That would replicate behavior in other statistical software (e.g., Stata when tab used with “missing” option). More importantly, using this option necessarily means you are interested in understanding the joint distribution of NaNs in the two series.
Current behavior of crosstab(‘y_variable’,‘x_variable’,dropna=False) appears to be equivalent to groupby(‘y_variable’).agg({‘x_variable’:‘count’}) with a binary dummy x variable.
To preserve this equivalence, it may be worthwhile to instead consider a switch adding NaN to the sets of tabulated values in a crosstab rather than changing behavior with dropna=False.
I also would like to see this fixed.
Is there a reasonably clean way to get the desired output within the context of Pandas as it is today?
Hi @Codeberg-AsGithubAlternative-buhtz
I didn’t blame you, it just sounded like you were interested in working on this
Are you asking about whether there’s agreement that the expected output from the original post is the one to aim for?
If so, then I agree that it would be an improvement. Seems like @dsaxton agrees too, and @TomAugspurger has welcomed a PR above. I presume @jreback agrees too, as the issue hasn’t been closed and he’s added labels to it
I agree this would make a nice improvement, especially since the
fillnafix doesn’t really work for categorical columns since the fill value needs to be an available level.@ceilingcat let us know if you need help starting a PR.