neptune-client: BUG: Neptune v1.3.2 is slow when using forked data loaders in Linux

We just released v1.4.1 which fixes this 🥳

I’ll be closing this issue now, but please feel free to let us know if you are still facing issues.

Hello all 👋

We have identified the root cause of this issue and have pushed a fix v1.4.1rc1 🎉

With this fix, you will not need to resort to any of the above-mentioned workarounds. ✅

We will be publishing the stable version soon, but if you want to test the pre-release version, you can update neptune by running pip install neptune==1.4.1rc1.

Please let us know if this works 🙏

We believe that changes introduced to better support forking processes in neptune==1.3.2 might be responsible for this. Can you please try downgrading to neptune==1.3.1 and let us know if it solves the issue?

Yeah, I can confirm this occurs to me with 1.3.2 but not with 1.3.1.

Hey all, We have found a workaround that should work for most cases.

Instead of using fork for multiprocessing, we recommend using spawn or forkserver. spawn is already the default in MacOS and Windows. forkserver is not supported on Windows.

To use spawn/forkserver, you will need to have a single entry point for your training script:

if __name__ == "__main__":
    multiprocessing.set_start_method("spawn")
    train_fn() # or trainer.train(), etc

Here is a minimal example of how you can do this --> Neptune slowdown issue workaround

Note that using spawn/forkserver comes with some limitations. You can check all the limitations here

If you do not want to make these changes, you can update the Neptune connection mode to “offline”, and then upload the data to Neptune servers using neptune sync once training is complete.

While we work on a more permanent solution, please let me know if either of these suggestions seem viable workarounds for now 🙏

@SiddhantSadangi I confirm that the issue does not happen when using neptune==1.3.1

Hello all,

We are having some trouble reproducing this issue. Would it be possible for anyone to share a reproducible example? Code, data sample, hardware details, etc. would help a lot pray

You can send these to support@neptune.ai.

I have sent this e-mail a reproduction of the issue in Colab.

Hello @dscarmo ,

You can use neptune clear to delete all unsynced information in the .neptune folder.

Since you also upgraded from neptune-client to neptune, I am sharing an upgrade guide that might be useful.

Please let me know if this helps.

It seems like I fixed my issue by clearing with Neptune clear, uninstalling Neptune, and installing neptune-client 0.16.16.

Note that neptune clear did not fix anything while still using 1.0.

I think others on this thread might fix their issues by using 0.16.16 too.

As additional information, here are screenshots of gpu usage of the same exact debug run (1 step in training and validation) with 1.0 and with 0.16.16:

1.0

0.16

Clearly something is bottlenecking the main training loop hard, maybe intensive disk operations interfering with IO? Might be the same bottleneck the Issue OP found in his profiling.

I don’t think any checkpoints are being logged because I’m always turning the training script off before it reaches 3000 steps (it’s too slow). When I look at the Artifacts tab it doesn’t show me any artifacts.

Here’s the run.

Thanks @SiddhantSadangi, I tried setting report_to to none as you suggested. Here’s the code and config:

import logging

import hydra
from transformers import Trainer, TrainingArguments
from transformers.integrations import NeptuneCallback

from dataset import LukalficationDataset
from dataset.classes import LEVEL1, LEVEL2, LEVEL3
from metrics import MetricsLogger
from model import LukalificationModel
from preprocessing import MultiLabelOneHotEncoder, Processor

logger = logging.getLogger(__name__)


@hydra.main(config_path=".", config_name="config.yaml", version_base="1.2")
def main(config):
    logger.info("Creating dataset")
    train_dataset = LukalficationDataset(**config.dataset.train)
    eval_dataset = LukalficationDataset(**config.dataset.eval)

    logger.info("Creating preprocessor")
    processor = Processor(
        tokenizer_config=config.tokenizer,
        label_encoders=[
            MultiLabelOneHotEncoder(LEVEL1),
            MultiLabelOneHotEncoder(LEVEL2),
            MultiLabelOneHotEncoder(LEVEL3),
        ],
    )

    logger.info("Creating model")
    model = LukalificationModel([len(LEVEL1), len(LEVEL2), len(LEVEL3)], **config.model)

    training_args = TrainingArguments(**config.training)

    logger.info("Training model")
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        data_collator=processor.data_collator,
        compute_metrics=MetricsLogger(
            label_encoders=processor.label_encoders
        ).compute_metrics,
        callbacks=[NeptuneCallback(log_checkpoints="best")],
    )

    trainer.train()

    logger.info("Training complete")


if __name__ == "__main__":
    main()

Config:

model:
  freeze_backbone: false
  checkpoint_path: null


dataset:
  train:
    csv_file: ${hydra:runtime.cwd}/data/train.csv
  eval:
    csv_file: ${hydra:runtime.cwd}/data/eval.csv


tokenizer:
  init:
    pretrained_model_name_or_path: "Unbabel/xlm-roberta-comet-small"
    truncation_side: "left"
  call:
    max_length: 512
    padding: longest
    truncation: true
    return_tensors: "pt"


training:
  output_dir: ${hydra:runtime.cwd}/checkpoints/${now:%Y-%m-%d}/${now:%H-%M}
  num_train_epochs: 50
  per_device_train_batch_size: 48
  per_device_eval_batch_size: 48
  learning_rate: 5e-5
  warmup_ratio: 0.1
  dataloader_num_workers: 8
  lr_scheduler_type: linear
  greater_is_better: true
  metric_for_best_model: "eval_global/f1-score"
  full_determinism: false
  no_cuda: false
  save_strategy: steps
  save_steps: 3000
  evaluation_strategy: steps
  eval_steps: 3000
  logging_strategy: steps
  logging_steps: 100
  optim: adamw_torch
  remove_unused_columns: false
  report_to: none
  load_best_model_at_end: true

The training script still seems to be slow, here’s the output from the profiler:

Hey @issamemari ,

I see that you have used report_to: null in your config file. It should be none instead of null. Can you update this as check if the issue is resolved?

Here’s the relevant API reference: https://huggingface.co/docs/transformers/v4.30.0/en/main_classes/trainer#transformers.TrainingArguments.report_to

I am not logging model checkpoints to Neptune, I’m just saving them to disk. I tried what you suggested, here’s the code:

import logging

import hydra
from transformers import Trainer, TrainingArguments
from transformers.integrations import NeptuneCallback

from dataset import LukalficationDataset
from dataset.classes import LEVEL1, LEVEL2, LEVEL3
from metrics import MetricsLogger
from model import LukalificationModel
from preprocessing import MultiLabelOneHotEncoder, Processor

logger = logging.getLogger(__name__)


@hydra.main(config_path=".", config_name="config.yaml", version_base="1.2")
def main(config):
    logger.info("Creating dataset")
    train_dataset = LukalficationDataset(**config.dataset.train)
    eval_dataset = LukalficationDataset(**config.dataset.eval)

    logger.info("Creating preprocessor")
    processor = Processor(
        tokenizer_config=config.tokenizer,
        label_encoders=[
            MultiLabelOneHotEncoder(LEVEL1),
            MultiLabelOneHotEncoder(LEVEL2),
            MultiLabelOneHotEncoder(LEVEL3),
        ],
    )

    logger.info("Creating model")
    model = LukalificationModel([len(LEVEL1), len(LEVEL2), len(LEVEL3)], **config.model)

    training_args = TrainingArguments(**config.training)

    logger.info("Training model")
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        data_collator=processor.data_collator,
        compute_metrics=MetricsLogger(
            label_encoders=processor.label_encoders
        ).compute_metrics,
        callbacks=[NeptuneCallback(log_checkpoints="best")],
    )

    trainer.train()

    logger.info("Training complete")


if __name__ == "__main__":
    main()

And here’s the config file:

model:
  freeze_backbone: false
  checkpoint_path: null


dataset:
  train:
    csv_file: ${hydra:runtime.cwd}/data/train.csv
  eval:
    csv_file: ${hydra:runtime.cwd}/data/eval.csv


tokenizer:
  init:
    pretrained_model_name_or_path: "Unbabel/xlm-roberta-comet-small"
    truncation_side: "left"
  call:
    max_length: 512
    padding: longest
    truncation: true
    return_tensors: "pt"


training:
  output_dir: ${hydra:runtime.cwd}/checkpoints/${now:%Y-%m-%d}/${now:%H-%M}
  num_train_epochs: 50
  per_device_train_batch_size: 48
  per_device_eval_batch_size: 48
  learning_rate: 5e-5
  warmup_ratio: 0.1
  dataloader_num_workers: 8
  lr_scheduler_type: linear
  greater_is_better: true
  metric_for_best_model: "eval_global/f1-score"
  full_determinism: false
  no_cuda: false
  save_strategy: steps
  save_steps: 3000
  evaluation_strategy: steps
  eval_steps: 3000
  logging_strategy: steps
  logging_steps: 100
  optim: adamw_torch
  remove_unused_columns: false
  report_to: null
  load_best_model_at_end: true

Here’s the output of my profiler with the Neptune callback:

And here’s the output without the Neptune callback (I explicitly called trainer.remove_callback(NeptuneCallback)):