spikeinterface: CUDA out of memory error from running kilosort
Hello,
When running kilosort4 via spikeinterface, CUDA out of memory error occurs immediately after the first clustering.
The error still occurs when setting clear_cache = True. But not when running kilosort directly from its gui.
Could you examine the error and provide any idea of how to fix? Thank you!!
kilosort issue <CUDA out of memory>
Kilosort version: 4.0.15 spikeinterface version: 0.100.0.dev0
File ~\.conda\envs\SI\spikeinterface\src\spikeinterface\sorters\launcher.py:306, in run_sorter_by_property(sorter_name, recording, grouping_property, folder, mode_if_folder_exists, engine, engine_kwargs, verbose, docker_image, singularity_image, working_folder, **sorter_params)
[295](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/launcher.py:295) job = dict(
[296](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/launcher.py:296) sorter_name=sorter_name,
[297](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/launcher.py:297) recording=rec,
(...)
[302](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/launcher.py:302) **sorter_params,
[303](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/launcher.py:303) )
[304](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/launcher.py:304) job_list.append(job)
--> [306](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/launcher.py:306) sorting_list = run_sorter_jobs(job_list, engine=engine, engine_kwargs=engine_kwargs, return_output=True)
[308](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/launcher.py:308) unit_groups = []
[309](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/launcher.py:309) for sorting, group in zip(sorting_list, recording_dict.keys()):
File ~\.conda\envs\SI\spikeinterface\src\spikeinterface\sorters\launcher.py:106, in run_sorter_jobs(job_list, engine, engine_kwargs, return_output)
[103](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/launcher.py:103) if engine == "loop":
[104](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/launcher.py:104) # simple loop in main process
[105](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/launcher.py:105) for kwargs in job_list:
--> [106](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/launcher.py:106) sorting = run_sorter(**kwargs)
[107](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/launcher.py:107) if return_output:
[108](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/launcher.py:108) out.append(sorting)
File ~\.conda\envs\SI\spikeinterface\src\spikeinterface\sorters\runsorter.py:216, in run_sorter(sorter_name, recording, folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, output_folder, **sorter_params)
[205](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/runsorter.py:205) raise RuntimeError(
[206](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/runsorter.py:206) "The python `spython` package must be installed to "
[207](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/runsorter.py:207) "run singularity. Install with `pip install spython`"
[208](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/runsorter.py:208) )
[210](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/runsorter.py:210) return run_sorter_container(
[211](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/runsorter.py:211) container_image=container_image,
[212](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/runsorter.py:212) mode=mode,
[213](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/runsorter.py:213) **common_kwargs,
[214](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/runsorter.py:214) )
--> [216](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/runsorter.py:216) return run_sorter_local(**common_kwargs)
File ~\.conda\envs\SI\spikeinterface\src\spikeinterface\sorters\runsorter.py:276, in run_sorter_local(sorter_name, recording, folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, output_folder, **sorter_params)
[274](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/runsorter.py:274) SorterClass.set_params_to_folder(recording, folder, sorter_params, verbose)
[275](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/runsorter.py:275) SorterClass.setup_recording(recording, folder, verbose=verbose)
--> [276](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/runsorter.py:276) SorterClass.run_from_folder(folder, raise_error, verbose)
[277](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/runsorter.py:277) if with_output:
[278](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/runsorter.py:278) sorting = SorterClass.get_result_from_folder(folder, register_recording=True, sorting_info=True)
File ~\.conda\envs\SI\spikeinterface\src\spikeinterface\sorters\basesorter.py:301, in BaseSorter.run_from_folder(cls, output_folder, raise_error, verbose)
[298](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/basesorter.py:298) print(f"{sorter_name} run time {run_time:0.2f}s")
[300](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/basesorter.py:300) if has_error and raise_error:
--> [301](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/basesorter.py:301) raise SpikeSortingError(
[302](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/basesorter.py:302) f"Spike sorting error trace:\n{error_log_to_display}\n"
[303](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/basesorter.py:303) f"Spike sorting failed. You can inspect the runtime trace in {output_folder}/spikeinterface_log.json."
[304](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/basesorter.py:304) )
[306](https://file+.vscode-resource.vscode-cdn.net/d%3A/YK_pyfile/SI/SI_Proc/~/.conda/envs/SI/spikeinterface/src/spikeinterface/sorters/basesorter.py:306) return run_time
SpikeSortingError: Spike sorting error trace:
Traceback (most recent call last):
File "C:\Users\Lenovo\.conda\envs\SI\spikeinterface\src\spikeinterface\sorters\basesorter.py", line 261, in run_from_folder
SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
File "C:\Users\Lenovo\.conda\envs\SI\spikeinterface\src\spikeinterface\sorters\external\kilosort4.py", line 273, in _run_from_folder
st, tF, _, _ = detect_spikes(ops, device, bfile, tic0=tic0, progress_bar=progress_bar)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\Lenovo\.conda\envs\SI\Lib\site-packages\kilosort\run_kilosort.py", line 611, in detect_spikes
st, tF, ops = template_matching.extract(ops, bfile, Wall3, device=device,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\Lenovo\.conda\envs\SI\Lib\site-packages\kilosort\template_matching.py", line 26, in extract
ctc = prepare_matching(ops, U)
^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\Lenovo\.conda\envs\SI\Lib\site-packages\kilosort\template_matching.py", line 108, in prepare_matching
ctc = torch.einsum('ijkm, kml -> ijl', UtU, WtW)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\Lenovo\.conda\envs\SI\Lib\site-packages\torch\functional.py", line 380, in einsum
return _VF.einsum(equation, operands) # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 53.24 GiB. GPU 0 has a total capacity of 10.00 GiB of which 0 bytes is free. Of the allocated memory 31.18 GiB is allocated by PyTorch, and 43.14 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 16 (10 by maintainers)
Deal! I actually always wanted to do that, but have been too lazy 😛
Hi @Hobart10, thanks for helping to get to the bottom of this.
can you double-check:
recording.save(..., format="binary"). Then load from the saved binary (the.datfile in the saved recording folder) either through KS4 GUI or theirrun_kilosortfunction (e.g. here. (basically, skip all SI wrapping and see if it is breaking from on the SI-saved binary).I have a colleague who is experiencing a similar error even when running the raw data directly in KS4. It would be good to find out whether this a) a problem with the spikeinterface wrapper b) a native KS4 problem c) in this test ensure that KS4 is definitely getting the binary and not defaulting to the spikeinterface wrapper c) some strange interaction between the spikeinterface preprocessing and KS4 (i.e. the error does not occur when running from the raw binary through KS4 but does occur on a spikeinterface preprocessed / saved binary). Could you also post the script you are running?