scikit-bio: Alpha Diversity indices produce zeros where previously undefined (NaN)
https://github.com/scikit-bio/scikit-bio/pull/1894 added short-circuits to produce 0 where previously the formula would produce NaN.
This is currently impacting QIIME 2’s diversity measures and represents a breaking change from our perspective. For the moment, we’re going to copy the effected indices prior to these updates into q2-diversity-lib, but we don’t want this to be a long-term solution.
What was the rationale for this particular change?
Reviewing Pielou’s J' for instance, there doesn’t appear to be a definition for zero-feature samples, so the formula producing NaN seems appropriate as it isn’t defined (this also lets the user know that they shouldn’t interpret that sample). Shannon has a similar reasoning.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 18 (13 by maintainers)
I’d like to suggest removing biom from this discussion. The transformation the model underneath operates on is inconsequential to the correctness and interpretation of the metrics.
I like this line of reasoning, but just to point out:
0 * NaNwould produceNaN, so I think the generalized and specialized metrics would still both produceNaNon empty communities (even if standardized by a zero-producing measure).I’m still fine with
0becomingNaNfor empty communities, but wanted to note the above.@ebolyen Thanks for the additional input! I agree with this.
All, I just did some investigation into the current and previous code. I am proposing the following solution:
NaNfor empty samples for all alpha diversity metrics.This is based on the observation that most of the currently implemented metrics involves fraction or zero division. Specifically, they include:
Only a few metrics don’t suffer from this and a 0 value is meaningful:
We could choose to return zero for these few metrics. However, a difficulty is that these metrics are somehow related to the fraction-based metrics. For example, species richness and Shannon can be uniformized into Hill number; Faith’ PD can be generalized into abundance-weighted phylogenetic diversity. If we make them special, we will get different results from the specialized and generalized metrics.
The
NaNshould be achieved through a short circuit, rather than from zero division, which will generate a ValueError (Python case) or a RuntimeWarning (NumPy case).The helper function
_validate_counts_vectorcan be modified to strip off all zero values. I hope that it can also automatically returnNaNfor the outer function, but it is not possible with Python. Therefore, each alpha diversity metric will check whether the return value of this helper function is empty, and returnsNaNin that case.I am going to work on this right away. Please let me know if you have other thoughts.
Hi @gregcaporaso I think the point regarding inputs into to downstream regression are appropriate. I’m in favor of reverting back to the original
np.nanimplementation, since it appears to cover the most edge cases.Yes, this was missed during review and should have been flagged earlier. Thanks for catching @ebolyen @gregcaporaso
Should we have a
baseparameter that enables users to easily swap bases without thinking too much? There are certainly scenarios where base 2 is preferable (it is often easier to do mental arithmetic on base 2 or 10, instead of e=2.71828…).I’m also curious, under what scenarios should we expect a sample to have 0 features? Shouldn’t we have an assert statement rather than return 0? If it returns 0, its basically a silent error, since this scenario shouldn’t exist in the first place.
EDIT: ignore the previous comment – there are certainly scenarios (aka sterile communities) where there isn’t anything present). The first comment still stands though – treating communities with 0 species and 1 species still seems off, there should at least be a way to distinguish between these two scenarios.