scikit-bio: Alpha Diversity indices produce zeros where previously undefined (NaN)

https://github.com/scikit-bio/scikit-bio/pull/1894 added short-circuits to produce 0 where previously the formula would produce NaN.

This is currently impacting QIIME 2’s diversity measures and represents a breaking change from our perspective. For the moment, we’re going to copy the effected indices prior to these updates into q2-diversity-lib, but we don’t want this to be a long-term solution.

What was the rationale for this particular change?

Reviewing Pielou’s J' for instance, there doesn’t appear to be a definition for zero-feature samples, so the formula producing NaN seems appropriate as it isn’t defined (this also lets the user know that they shouldn’t interpret that sample). Shannon has a similar reasoning.

cc @gregcaporaso

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 18 (13 by maintainers)

Most upvoted comments

I’d like to suggest removing biom from this discussion. The transformation the model underneath operates on is inconsequential to the correctness and interpretation of the metrics.

For example, species richness and Shannon can be uniformized into Hill number; Faith’ PD can be generalized into abundance-weighted phylogenetic diversity. If we make them special, we will get different results from the specialized and generalized metrics.

I like this line of reasoning, but just to point out: 0 * NaN would produce NaN, so I think the generalized and specialized metrics would still both produce NaN on empty communities (even if standardized by a zero-producing measure).

I’m still fine with 0 becoming NaN for empty communities, but wanted to note the above.

@ebolyen Thanks for the additional input! I agree with this.

All, I just did some investigation into the current and previous code. I am proposing the following solution:

  1. Uniformly return NaN for empty samples for all alpha diversity metrics.

This is based on the observation that most of the currently implemented metrics involves fraction or zero division. Specifically, they include:

  • berger_parker_d, brillouin_d, dominance, enspie, esty_ci, goods_coverage, kempton_taylor_q, margalef, mcintosh_d, mcintosh_e, menhinick, pielou_e, robbins, shannon, simpson, simpson_e, strong

Only a few metrics don’t suffer from this and a 0 value is meaningful:

  • doubles, sobs (i.e., observed features), singles, osd, faith_pd

We could choose to return zero for these few metrics. However, a difficulty is that these metrics are somehow related to the fraction-based metrics. For example, species richness and Shannon can be uniformized into Hill number; Faith’ PD can be generalized into abundance-weighted phylogenetic diversity. If we make them special, we will get different results from the specialized and generalized metrics.

  1. The NaN should be achieved through a short circuit, rather than from zero division, which will generate a ValueError (Python case) or a RuntimeWarning (NumPy case).

  2. The helper function _validate_counts_vector can be modified to strip off all zero values. I hope that it can also automatically return NaN for the outer function, but it is not possible with Python. Therefore, each alpha diversity metric will check whether the return value of this helper function is empty, and returns NaN in that case.

I am going to work on this right away. Please let me know if you have other thoughts.

Hi @gregcaporaso I think the point regarding inputs into to downstream regression are appropriate. I’m in favor of reverting back to the original np.nan implementation, since it appears to cover the most edge cases.

Yes, this was missed during review and should have been flagged earlier. Thanks for catching @ebolyen @gregcaporaso

Should we have a base parameter that enables users to easily swap bases without thinking too much? There are certainly scenarios where base 2 is preferable (it is often easier to do mental arithmetic on base 2 or 10, instead of e=2.71828…).

I’m also curious, under what scenarios should we expect a sample to have 0 features? Shouldn’t we have an assert statement rather than return 0? If it returns 0, its basically a silent error, since this scenario shouldn’t exist in the first place.

EDIT: ignore the previous comment – there are certainly scenarios (aka sterile communities) where there isn’t anything present). The first comment still stands though – treating communities with 0 species and 1 species still seems off, there should at least be a way to distinguish between these two scenarios.