torchio: Memory leak with TorchIO 0.20.1
Is there an existing issue for this?
- I have searched the existing issues
Bug summary
A memory leak issue is observed in TorchIO version 20.0.1 during prolonged training sessions. The memory usage gradually increases over time, leading to an eventual out-of-memory error. This issue is not present in version 0.20.0, where memory usage remains stable under similar conditions.
The following graphs show the evolution of RAM utilization during a basic DL training session, using the provided example code, with TorchIO version 0.20.0:
Here with TorchIO 0.20.1:
Do you have any tips for preventing this problem with Torch 0.20.1?
Thanks 😃
Code for reproduction
import copy
from typing import Any
import torch
import torch.utils.data
import torchio as tio
import pytorch_lightning as pl
import monai
import psutil
import matplotlib.pyplot as plt
ram_usage = []
timestamps = []
def get_ram_usage():
return psutil.virtual_memory().percent
# %% Lightning module
class BasicPLModule(pl.LightningModule):
def __init__(self):
super().__init__()
self.net = monai.networks.nets.Unet(spatial_dims=3, in_channels=1, out_channels=1, channels=(8, 16, 32), strides=(2, 2)).float()
def configure_optimizers(self):
optimizer = torch.optim.Adam(self.parameters(), lr=0.0001)
return optimizer
def training_step(self, batch, batch_idx):
inputs = batch['t1'][tio.DATA].float()
targets = batch['brain'][tio.DATA].float()
preds = self.net(inputs)
return torch.nn.functional.mse_loss(preds, targets)
def on_train_epoch_end(self):
ram_usage.append(get_ram_usage())
timestamps.append(self.current_epoch)
# %% Main program
if __name__ == '__main__':
subject = tio.datasets.Colin27()
# convert to instance of new subject class
subject = tio.Subject(
t1=subject['t1'],
head=subject['head'],
brain=subject['brain'],
)
subjects = 10 * [subject]
transforms = tio.Compose([
tio.ZNormalization(masking_method = tio.ZNormalization.mean),
tio.transforms.CropOrPad(target_shape=192)
])
subject_dataset = tio.SubjectsDataset(subjects, transforms)
training_loader = tio.SubjectsLoader(subject_dataset, batch_size=1)
reg_net = BasicPLModule()
trainer_args = {
'max_epochs': 200,
}
trainer_reg = pl.Trainer(**trainer_args)
trainer_reg.fit(reg_net, training_loader)
# Plotting the data
plt.plot(timestamps, ram_usage)
plt.title('RAM Usage Over Time')
plt.xlabel('Time')
plt.ylabel('RAM Usage (%)')
plt.grid(True)
plt.show()
Actual outcome
After a long period : Process finished with exit code 137 (interrupted by signal 9: SIGKILL)
Error messages
No response
Expected outcome
It’s an example of code where the expected outcome is irrelevant, but the RAM utilization should remain relatively constant.
System info
Platform: Windows-10-10.0.22631-SP0
TorchIO: 0.20.1
PyTorch: 2.4.1+cu121
SimpleITK: 2.4.0 (ITK 5.4)
NumPy: 2.0.2
Python: 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)]
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 22 (16 by maintainers)
@paulhager
Could you provide any additional information on your environment and when it happens? A MWE would be greatly appreciated to look into this issue. Also (if possible) show a graph of the memory usage over time.
Nico
Hi, all. This is hopefully fixed in
v0.20.4thanks to @nicoloesch’s work inHi folks,
sorry I did see the ping just now (pretty busy these days ^^).
@fepegar I only outsourced the existing implementation to a function to correctly handle subclasses of
Subject😃That being said though, I agree with @nicoloesch on the issue and that it should be safe to use
copyfor most things. Probably the reason for going with deepcopy here is because python sometimes has weird rules to call by reference vs call by value (and therefore also when a copy vs. deepcopy is required). And with objects like tensors, where the object is merely a pointer to the actual consumed memory these things get even less clear.My recommendation would be:
copyworks as expected in all cases since it seems to reduce the memory footprint (we might need a separate deepcopy then though as we might still want to exclude images here)Sadly, I won’t have the time to work on it myself, but I’d be happy to review a PR if someone else takes this on 😃
Hi @romainVala, hi @FlorianScalvini,
Update: I think I found the memory leak - this time mostly definitive. I will provide a step-by-step overview of what I did. Let’s start with my environment:
The following code is required to reproduce the memory leak:
How to reproduce the memory leak? The memory leak appears as soon as
copy_compose=True. As a result, the base classtio.Transformoftio.Composewill make a copy of the subject in__call__(see L.162) [all the transforms would do that but I call myDummyTransformexplicitly withcopy=Falseto reduce the memory leak and just have it forCompose]:This in theory will create a shallow copy of the object (
tio.Subjectis of typedict), which calls__copy__of thetio.Subjectif implemented. The method is implemented insubject.pyto apply custom logic:And exactly here lies the issue. This method
_subejct_copy_helperis responsible for the memory leak. You can see the origin of memory leak in my profiled data.subject_copycompose-False.txt: This runs the script with
copy_compose=False. As a result, none of the transforms copies thetio.Subjectand we do not call the__copy__method (at least from the transforms). This results in moderate memory usage despite the rather large tensors of roughly 10GB at peak.subject_copycompose-True.txt: We run the same script but this time with
copy_compose=True. This in turn invokes the__call__of eachtio.Subject, which calls the_subject_copy_helperunder the hood. You see exploding memory in the logs of around 28GB at peak.subject_copycompose-True_comment-copy.txt: Run the same script as in 2. but this time we just comment the
__copy__method oftio.Subjectin the source file. As a result, thecopy.copycall will create a shallow copy without custom logic (apparently using__new__and__init__- unsure about the exact python semantics). You will see the memory usage plummets down from 28GB back to the original 10GB, indicative of a shallow copy with shared memory.I was wondering why there is a differentiation betweeen the copying behaviour of a
tio.Imageand any other attribute of the dict in_subjets_copy_helper? Why are other attributes called withdeepcopydespite we are calling the method in__copy__and not in__deepcopy__?Note: The memory behaviour changes between runs occasionally where sometimes memory is freed, sometimes memory is not freed. It still can be backtraced to the implementation of
__copy__and ultimately the question of why this is required as callingcopy.copyandcopy.deepcopyon thetio.Subjectwithout having any implementation of__call__works as intended. Maybe it is useful to figure out when it is absolutely necessary to callcopy.deepcopy(expensive with large tensors) and when to callcopy.copyopposed to a mixed version of both.One way could be to define memory-safe
__copy__and__deepcopy__methods as described here but I am so far unsure if that is required, as the default without any specification of__copy__copies the subject as intended.I hope this helps and would love to hear some thoughts 😃 Nico
Hello @nicoloesch and @romainVala!
My problem may be related to @nicoloesch’s concerning the missing de-allocation memory in transform functions.
Indeed, when the transformation application is disabled by setting the transform parameter to None (
tio.SubjectsDataset(subjects, transform=None)) the memory problems disappear.PS : I had to modify the initialization of the Subject object from my example to take into account the input size constraints of the Unet model (Divisible by 32), in order to test the impact of the preprocessing on memory