torchmetrics: [metrics] AUROC Metric can't handle 0 observations of a class with multiclass classifier

I’m attempting to calculate AUROC for a multiclass problem where some classes are very rare, occasionally never seen, and I’m getting the following error: raise ValueError("No positive samples in targets, true positive value should be meaningless")

In the case of 0 observations, I feel the average='weighted' should work, since the contribution to the final AUROC should be 0 regardless. One can think of other scenarios where there are a very high number of classes, some of which will happen to not be seen in some dataset.

_Originally posted by @BeyondTheProof in https://github.com/PyTorchLightning/pytorch-lightning/issues/2210#issuecomment-872440776_

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 15 (13 by maintainers)

Most upvoted comments

Hi! thanks for your contribution!, great first issue!

github-actions[bot] on Jul 2, 2021

I’ve found a hack for this by subclassing AUROC:

class FixedAUC(torchmetrics.AUROC):
    def update(self, preds: torch.Tensor, target: torch.Tensor):
        num_classes = preds.shape[1]
        zero_obs_mask = torch.tensor([(target == c).sum() > 0 for c in range(num_classes)])
        preds = preds[:, zero_obs_mask]
        target = target[:, zero_obs_mask]
        self.num_classes = zero_obs_mask.sum().int()
        super().update(torch.softmax(preds, axis=-1).data, target.argmax(axis=-1).int().data)

I will be making a cleaner fix in torch and asking for a PR

BeyondTheProof on Jul 7, 2021