❓ Questions & Help

When I start training the code, it is hanging in the beginning as below. How long do you expect to wait for the training to start? For my case, it is more than at least ~5 hrs. My very first run went through and it failed in the middle of the training because of CUDA issues.

[2021-09-16 01:46:18,272][openspeech.utils][INFO] - Operating System : Linux 5.11.0-27-generic
[2021-09-16 01:46:18,272][openspeech.utils][INFO] - Processor : x86_64
[2021-09-16 01:46:18,273][openspeech.utils][INFO] - device : NVIDIA GeForce RTX 2080 Ti
[2021-09-16 01:46:18,273][openspeech.utils][INFO] - CUDA is available : True
[2021-09-16 01:46:18,273][openspeech.utils][INFO] - CUDA version : 11.0
[2021-09-16 01:46:18,273][openspeech.utils][INFO] - PyTorch version : 1.7.0
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
/opt/conda/lib/python3.8/site-packages/pytorch_lightning/core/datamodule.py:423: LightningDeprecationWarning: DataModule.setup has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.setup.
  rank_zero_deprecation(
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results
wandb: Enter your choice: 3
wandb: You chose 'Don't visualize my results'
wandb: WARNING `resume` will be ignored since W&B syncing is set to `offline`. Starting a new run with run id tpyh7nfj.

CondaEnvException: Unable to determine environment

Please re-run this command with one of the following options:

* Provide an environment name via --name or -n
* Re-run this command inside an activated conda environment.

wandb: W&B syncing is set to `offline` in this directory.  Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.

  | Name      | Type                     | Params
-------------------------------------------------------
0 | criterion | JointCTCCrossEntropyLoss | 0
1 | encoder   | LSTMEncoder              | 36.6 M
2 | decoder   | LSTMAttentionDecoder     | 53.4 M
-------------------------------------------------------
90.0 M    Trainable params
0         Non-trainable params
90.0 M    Total params
359.991   Total estimated model params size (MB)
Global seed set to 1

Details

Env: Ubuntu 18.04 GPU: 2080TI 12GB