pytorch-lightning: Comet logger cannot be pickled after creating an experiment
🐛 Bug
The Comet logger cannot be pickled after an experiment (at least an OfflineExperiment) has been created.
To Reproduce
Steps to reproduce the behavior:
initialize the logger object (works fine)
from pytorch_lightning.loggers import CometLogger
import tests.base.utils as tutils
from pytorch_lightning import Trainer
import pickle
model, _ = tutils.get_default_model()
logger = CometLogger(save_dir='test')
pickle.dumps(logger)
initialize a Trainer object with the logger (works fine)
trainer = Trainer(
max_epochs=1,
logger=logger
)
pickle.dumps(logger)
pickle.dumps(trainer)
access the experiment attribute which creates the OfflineExperiment object (fails)
logger.experiment
pickle.dumps(logger)
>> TypeError: can't pickle _thread.lock objects
Expected behavior
We should be able to pickle loggers for distributed training.
Environment
- CUDA: - GPU: - available: False - version: None
- Packages: - numpy: 1.18.1 - pyTorch_debug: False - pyTorch_version: 1.4.0 - pytorch-lightning: 0.7.5 - tensorboard: 2.1.0 - tqdm: 4.42.0
- System: - OS: Darwin - architecture: - 64bit - - processor: i386 - python: 3.7.6 - version: Darwin Kernel Version 19.3.0: Thu Jan 9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 1
- Comments: 15 (2 by maintainers)
I still see this bug as well with WandB logger.
I don’t know if it can help or if it is the right place, but a similar error occurswhen running in ddp mode with the WandB logger.
WandB uses a lambda function at some point.
Does the logger have to pickled ? Couldn’t it log only on rank 0 at epoch_end ?
also related: #1704
Currently having this issue with wandbLogger.
I still have this error with 1.5.10 on macOS