torchio: Memory leak with TorchIO 0.20.1

Is there an existing issue for this?

  • I have searched the existing issues

Bug summary

A memory leak issue is observed in TorchIO version 20.0.1 during prolonged training sessions. The memory usage gradually increases over time, leading to an eventual out-of-memory error. This issue is not present in version 0.20.0, where memory usage remains stable under similar conditions.

The following graphs show the evolution of RAM utilization during a basic DL training session, using the provided example code, with TorchIO version 0.20.0: torch20_0_0

Here with TorchIO 0.20.1:

torchio20_0_1

Do you have any tips for preventing this problem with Torch 0.20.1?

Thanks 😃

Code for reproduction

import copy
from typing import Any
import torch
import torch.utils.data
import torchio as tio
import pytorch_lightning as pl
import monai
import psutil
import matplotlib.pyplot as plt


ram_usage = []
timestamps = []


def get_ram_usage():
    return psutil.virtual_memory().percent

# %% Lightning module
class BasicPLModule(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.net = monai.networks.nets.Unet(spatial_dims=3, in_channels=1, out_channels=1, channels=(8, 16, 32), strides=(2, 2)).float()

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=0.0001)
        return optimizer

    def training_step(self, batch, batch_idx):
        inputs = batch['t1'][tio.DATA].float()
        targets = batch['brain'][tio.DATA].float()
        preds = self.net(inputs)
        return torch.nn.functional.mse_loss(preds, targets)

    def on_train_epoch_end(self):
        ram_usage.append(get_ram_usage())
        timestamps.append(self.current_epoch)


# %% Main program
if __name__ == '__main__':

    subject = tio.datasets.Colin27()
    # convert to instance of new subject class
    subject = tio.Subject(
        t1=subject['t1'],
        head=subject['head'],
        brain=subject['brain'],
    )
    subjects = 10 * [subject]
    transforms = tio.Compose([
        tio.ZNormalization(masking_method = tio.ZNormalization.mean),
        tio.transforms.CropOrPad(target_shape=192)
    ])
    subject_dataset = tio.SubjectsDataset(subjects, transforms)
    training_loader = tio.SubjectsLoader(subject_dataset, batch_size=1)
    reg_net = BasicPLModule()

    trainer_args = {
        'max_epochs': 200,
    }

    trainer_reg = pl.Trainer(**trainer_args)
    trainer_reg.fit(reg_net, training_loader)

    # Plotting the data
    plt.plot(timestamps, ram_usage)
    plt.title('RAM Usage Over Time')
    plt.xlabel('Time')
    plt.ylabel('RAM Usage (%)')
    plt.grid(True)
    plt.show()

Actual outcome

After a long period : Process finished with exit code 137 (interrupted by signal 9: SIGKILL)

Error messages

No response

Expected outcome

It’s an example of code where the expected outcome is irrelevant, but the RAM utilization should remain relatively constant.

System info

Platform:   Windows-10-10.0.22631-SP0
TorchIO:    0.20.1
PyTorch:    2.4.1+cu121
SimpleITK:  2.4.0 (ITK 5.4)
NumPy:      2.0.2
Python:     3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)]

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 22 (16 by maintainers)

Most upvoted comments

@paulhager

Could you provide any additional information on your environment and when it happens? A MWE would be greatly appreciated to look into this issue. Also (if possible) show a graph of the memory usage over time.

Nico

Hi, all. This is hopefully fixed in v0.20.4 thanks to @nicoloesch’s work in

Hi folks,

sorry I did see the ping just now (pretty busy these days ^^).

@fepegar I only outsourced the existing implementation to a function to correctly handle subclasses of Subject 😃

That being said though, I agree with @nicoloesch on the issue and that it should be safe to use copy for most things. Probably the reason for going with deepcopy here is because python sometimes has weird rules to call by reference vs call by value (and therefore also when a copy vs. deepcopy is required). And with objects like tensors, where the object is merely a pointer to the actual consumed memory these things get even less clear.

My recommendation would be:

  • Check if just switching to a copy works as expected in all cases since it seems to reduce the memory footprint (we might need a separate deepcopy then though as we might still want to exclude images here)
  • if it doesn’t act as expected make a list of things that we want to be shallow copied and handle them manually in this function.

Sadly, I won’t have the time to work on it myself, but I’d be happy to review a PR if someone else takes this on 😃

Hi @romainVala, hi @FlorianScalvini,

Update: I think I found the memory leak - this time mostly definitive. I will provide a step-by-step overview of what I did. Let’s start with my environment:

Platform:   Linux-6.8.0-45-generic-x86_64-with-glibc2.35
TorchIO:    0.20.1
PyTorch:    2.3.1
SimpleITK:  2.4.0 (ITK 5.4)
NumPy:      2.0.2
Python:     3.10.15 (main, Oct  3 2024, 07:27:34) [GCC 11.2.0]

The following code is required to reproduce the memory leak:

import torchio as tio
from memory_profiler import profile
import torch

class DummyTransform(tio.Transform):

    @profile
    def apply_transform(self, subject: tio.Subject) -> tio.Subject:

        for img in subject.get_images_dict().values():
            # Make shallow copy - that must be the memory leak
            if isinstance(img, tio.Image):
                new_img = img.data.clone().float().numpy()
                new_img += 1
                new_img = torch.as_tensor(new_img)
                img.set_data(new_img)
            else:
                raise ValueError(f"Expected data to be of type np.ndarray or torch.Tensor but got {type(img.data)}")
        return subject
        
        
def run_main(
    length: int = 4,
    copy_compose: bool = True,
):
    transforms = tio.Compose([
        DummyTransform(copy=False, include=["t1c", "t1n", "t2w", "t2f"]),
        DummyTransform(copy=False, include=["t1c", "t1n", "t2w", "t2f"]),
        DummyTransform(copy=False, include=["t1c", "t1n", "t2w", "t2f"]),
    ], copy=copy_compose)

    # roughly 3GB in size (4 * 0.75GB)
    subject = tio.Subject(**{seq: tio.ScalarImage(tensor=torch.randn(1, 1024, 1024, 192)) for seq in ["t1c", "t1n", "t2w", "t2f"]})

    subjects_dataset = tio.SubjectsDataset(
        subjects=length * [subject],
        transform=transforms
    )

    subjects_loader = tio.SubjectsLoader(
        dataset=subjects_dataset,
        batch_size=1,
        num_workers=0  # increasing it will multiply the memory leak per worker
    )

    for subject in subjects_loader:
        print("Processed subject")

if __name__ == "__main__":
    run_main()

How to reproduce the memory leak? The memory leak appears as soon as copy_compose=True. As a result, the base class tio.Transform of tio.Compose will make a copy of the subject in __call__ (see L.162) [all the transforms would do that but I call my DummyTransform explicitly with copy=False to reduce the memory leak and just have it for Compose]:

...
if self.copy:
    subject = copy.copy(subject)
...

This in theory will create a shallow copy of the object (tio.Subject is of type dict), which calls __copy__ of the tio.Subject if implemented. The method is implemented in subject.py to apply custom logic:

def __copy__(self):
    return _subject_copy_helper(self, type(self))
    
...

def _subject_copy_helper(
    old_obj: Subject,
    new_subj_cls: Callable[[Dict[str, Any]], Subject],
):
    result_dict = {}
    for key, value in old_obj.items():
        if isinstance(value, Image):
            value = copy.copy(value)
        else:
            value = copy.deepcopy(value)
        result_dict[key] = value

    new = new_subj_cls(**result_dict)  # type: ignore[call-arg]
    new.applied_transforms = old_obj.applied_transforms[:]
    return new

And exactly here lies the issue. This method _subejct_copy_helper is responsible for the memory leak. You can see the origin of memory leak in my profiled data.

  1. subject_copycompose-False.txt: This runs the script with copy_compose=False. As a result, none of the transforms copies the tio.Subject and we do not call the __copy__ method (at least from the transforms). This results in moderate memory usage despite the rather large tensors of roughly 10GB at peak.

  2. subject_copycompose-True.txt: We run the same script but this time with copy_compose=True. This in turn invokes the __call__ of each tio.Subject, which calls the _subject_copy_helper under the hood. You see exploding memory in the logs of around 28GB at peak.

  3. subject_copycompose-True_comment-copy.txt: Run the same script as in 2. but this time we just comment the __copy__ method of tio.Subject in the source file. As a result, the copy.copy call will create a shallow copy without custom logic (apparently using __new__ and __init__ - unsure about the exact python semantics). You will see the memory usage plummets down from 28GB back to the original 10GB, indicative of a shallow copy with shared memory.

I was wondering why there is a differentiation betweeen the copying behaviour of a tio.Image and any other attribute of the dict in _subjets_copy_helper? Why are other attributes called with deepcopy despite we are calling the method in __copy__ and not in __deepcopy__?

Note: The memory behaviour changes between runs occasionally where sometimes memory is freed, sometimes memory is not freed. It still can be backtraced to the implementation of __copy__ and ultimately the question of why this is required as calling copy.copy and copy.deepcopy on the tio.Subject without having any implementation of __call__ works as intended. Maybe it is useful to figure out when it is absolutely necessary to call copy.deepcopy (expensive with large tensors) and when to call copy.copy opposed to a mixed version of both.

One way could be to define memory-safe __copy__ and __deepcopy__ methods as described here but I am so far unsure if that is required, as the default without any specification of __copy__ copies the subject as intended.

I hope this helps and would love to hear some thoughts 😃 Nico

Hello @nicoloesch and @romainVala!

My problem may be related to @nicoloesch’s concerning the missing de-allocation memory in transform functions.

Indeed, when the transformation application is disabled by setting the transform parameter to None (tio.SubjectsDataset(subjects, transform=None)) the memory problems disappear.

PS : I had to modify the initialization of the Subject object from my example to take into account the input size constraints of the Unet model (Divisible by 32), in order to test the impact of the preprocessing on memory

  subject = tio.Subject(
      t1=tio.ScalarImage(tensor=torch.rand(1, 192, 192, 192)),
      head=tio.ScalarImage(tensor=torch.rand(1, 192, 192, 192)),
      brain=tio.ScalarImage(tensor=torch.rand(1, 192, 192, 192)),
  )