neptune-client: BUG: Neptune log error for multiple dataloaders

Describe the bug

Error gets thrown while logging the metric value. Having Pytorch lightning integration with Neptune. This error gets thrown only in the latest client of Neptune’s from neptune.new.integrations.pytorch_lightning import NeptuneLogger

Reproduction

https://colab.research.google.com/drive/13rRlztjGRQrv6Y3W-d21Dotoj8L2UtoZ?usp=sharing

Expected behavior

Experiment should keep running when without any error.

Traceback

Following trace as a result of invoking self.logger.log_metrics

    def __getattr__(self, attr):
>       raise AttributeError("{} has no attribute {}.".format(type(self), attr))
E       AttributeError: <class 'neptune.new.attributes.namespace.Namespace'> has no attribute log.
env/lib/python3.8/site-packages/neptune/new/attributes/attribute.py:35: AttributeError

image

If the value of attr is None, then it passes the if condition and am not facing any error. Facing the issue in the else condition. neptune.new.handler.Handler.log self._path = "val_loss"

Environment

The output of pip list:

neptune-contrib           0.27.2                   pypi_0    pypi
neptune-pytorch-lightning 0.9.7                    pypi_0    pypi

The operating system you’re using: Ubuntu The output of python --version: Python 3.8.10

Additional context It gets logged for all the metrics, only for this particular ‘val_loss’ key the error gets thrown. Happens only after migrating to new neptune client. Works fine with previous version. This error gets thrown only having more than one validation dataloader.

EDIT: If we have multiple dataloaders, then all of the parameters that gets logged will have name of the dataloader appended. Ex: Suppose my log is self.log('loss',0.2) It will get logged for each of the dataloader along with its index in the log name and its corresponding value: loss/dataloader_0 = 0.2 , loss/dataloader_1=0.4 and so on for every dataloader. Since my metric to monitor is ‘loss’, PTL also expects exact string ‘loss’ value to be logged, otherwise it throws below error

  
if not trainer.fit_loop.epoch_loop.val_loop._has_run:
              warning_cache.warn(m)
          else:
             raise MisconfigurationException(m)
E               pytorch_lightning.utilities.exceptions.MisconfigurationException: ModelCheckpoint(monitor='loss') not found in the   returned metrics: ['train_loss', 'train_loss_step', 'loss/dataloader_idx_0', 'loss/dataloader_idx_1', 'validation_f1',  'validation_precision', 'validation_recall', 'validation_accuracy']. HINT: Did you call self.log('loss', value) in the LightningModule?

But according to Neptune, ‘loss’ is now invalid once you have already logged ‘loss/dataloader_1’ (I guess) ? If so, you are both contradicting.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (9 by maintainers)

Most upvoted comments

Appreciate your reply.

I will pass this info to the product team, for the time being, I recommend to adjust loss names a bit

Sure, Thanks.