spikeinterface: Error using si.run_sorter [Mountainsort5] with TDT data on Windows 10
Hi, I am encountering an error when running mountainsort5. I follow the instructions as describe in #2471
- n_jobs = 1
- create a tmp dir in a not nested folder
OS: Windows 10 Ram: 128 SSD available: 1.2Tb Data Size after filter and zarr compress: 340mb Si version: ‘0.100.0’ MS5 version: ‘0.5.3’
the code that I am using is:
# imports
import numcodecs
from pathlib import Path
import spikeinterface.full as si
import spikeinterface.extractors as se
import probeinterface as pi
from probeinterface import read_probeinterface
from probeinterface.plotting import plot_probe_group
print(f"SpikeInterface version: {si.__version__}") # 0.100.0
#%%
import matplotlib.pyplot as plt
import numpy as np
from pathlib import Path
import os
import warnings
warnings.simplefilter("ignore")
# %matplotlib widget
%matplotlib widget
save_folder = drv_dir.joinpath("preprocessed_compressed.zarr")
if (save_folder).is_dir():
recording_saved = si.read_zarr(save_folder)
else:
import numcodecs
compressor = numcodecs.Blosc(cname="zstd", clevel=9, shuffle=numcodecs.Blosc.BITSHUFFLE)
recording_saved = recording_sub.save(format="zarr", folder=save_folder,
compressor=compressor,
**job_kwargs)
#%%
si.get_default_sorter_params('mountainsort5')
#%%
tmp_folder = Path.home().joinpath('.tmp')
#%%
sorter_params = {'filter': False,
'temporary_base_dir': tmp_folder.__str__(),
'n_jobs_for_preprocessing': 1}
#%%
sorting_ = si.run_sorter(sorter_name='mountainsort5', recording=recording_saved, remove_existing_folder=True,
output_folder=tmp_folder.joinpath('results_').__str__(),
**sorter_params,
verbose=True)
# params
{
"sorter_name": "mountainsort5",
"sorter_params": {
"scheme": "2",
"detect_threshold": 5.5,
"detect_sign": -1,
"detect_time_radius_msec": 0.5,
"snippet_T1": 20,
"snippet_T2": 20,
"npca_per_channel": 3,
"npca_per_subdivision": 10,
"snippet_mask_radius": 250,
"scheme1_detect_channel_radius": 150,
"scheme2_phase1_detect_channel_radius": 200,
"scheme2_detect_channel_radius": 50,
"scheme2_max_num_snippets_per_training_batch": 200,
"scheme2_training_duration_sec": 300,
"scheme2_training_recording_sampling_mode": "uniform",
"scheme3_block_duration_sec": 1800,
"freq_min": 300,
"freq_max": 6000,
"filter": false,
"whiten": true,
"temporary_base_dir": "C:\\Users\\jesus.penalozaa\\.tmp",
"n_jobs_for_preprocessing": 1
}
}
# Error
{
"sorter_name": "mountainsort5",
"sorter_version": "0.5.3",
"datetime": "2024-02-14T17:37:37.833633",
"runtime_trace": [],
"error": true,
"error_trace": "Traceback (most recent call last):\n File \"c:\\Users\\jesus.penalozaa\\AppData\\Local\\miniconda3\\envs\\roboz\\Lib\\shutil.py\", line 634, in _rmtree_unsafe\n os.unlink(fullname)\nPermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\\\Users\\\\jesus.penalozaa\\\\.tmp\\\\tmp5b8tjy78\\\\recording.dat'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"c:\\Users\\jesus.penalozaa\\AppData\\Local\\miniconda3\\envs\\roboz\\Lib\\site-packages\\spikeinterface\\sorters\\basesorter.py\", line 258, in run_from_folder\n SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)\n File \"c:\\Users\\jesus.penalozaa\\AppData\\Local\\miniconda3\\envs\\roboz\\Lib\\site-packages\\spikeinterface\\sorters\\external\\mountainsort5.py\", line 189, in _run_from_folder\n with TemporaryDirectory(dir=p[\"temporary_base_dir\"]) as tmpdir:\n File \"c:\\Users\\jesus.penalozaa\\AppData\\Local\\miniconda3\\envs\\roboz\\Lib\\tempfile.py\", line 946, in __exit__\n self.cleanup()\n File \"c:\\Users\\jesus.penalozaa\\AppData\\Local\\miniconda3\\envs\\roboz\\Lib\\tempfile.py\", line 950, in cleanup\n self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)\n File \"c:\\Users\\jesus.penalozaa\\AppData\\Local\\miniconda3\\envs\\roboz\\Lib\\tempfile.py\", line 930, in _rmtree\n _shutil.rmtree(name, onexc=onexc)\n File \"c:\\Users\\jesus.penalozaa\\AppData\\Local\\miniconda3\\envs\\roboz\\Lib\\shutil.py\", line 808, in rmtree\n return _rmtree_unsafe(path, onexc)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"c:\\Users\\jesus.penalozaa\\AppData\\Local\\miniconda3\\envs\\roboz\\Lib\\shutil.py\", line 636, in _rmtree_unsafe\n onexc(os.unlink, fullname, err)\n File \"c:\\Users\\jesus.penalozaa\\AppData\\Local\\miniconda3\\envs\\roboz\\Lib\\tempfile.py\", line 905, in onexc\n _os.unlink(path)\nPermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\\\Users\\\\jesus.penalozaa\\\\.tmp\\\\tmp5b8tjy78\\\\recording.dat'\n",
"run_time": null
}
About this issue
- Original URL
- State: open
- Created 5 months ago
- Comments: 16
There’s not a discord, but there is a slack. I’m not sure who admins that so I’ll tag @alejoe91 and I’m sure he can help get you set up. In general probe based questions can go to probeinterface waveform extraction questions would go here. And if you have electrophysiology reading (ie from one of the extractors in the
spikeinterface.extractorsmodule) would go to neo. This issue tracker is pretty responsive so most questions can go here just so they can help the overall community (I know other libraries really try to divide between a chat and the issue tracker, but again @alejoe91 can give the definitive answer for that).Completely my fault. I mistyped. It’s
ignore_cleanup_errors. Notice thesthat I forgot to include.@jesusdpa1,
If you are comfortable installing from source (which it doesn’t look like you’re currently so happy to help you do that if you want) and using at least python 3.10 (which it seems like you are) then the work around is to open the file
mounainsort5.pyand edit this line:https://github.com/SpikeInterface/spikeinterface/blob/d459b4ecb0108c2a6889fa69466b896086dd3767/src/spikeinterface/sorters/external/mountainsort5.py#L189
Then you just have to go and delete the temp folder yourself that you put in
temporary_base_dir. In this case for some files it might succeed and you don’t need to cleanup for others the cleanup will fail, but it won’t crash the program. It will just pass and return the sorting object into your terminal.Otherwise we wait for mountainsort5 to be edited.
https://docs.python.org/3/library/tempfile.html
Here are the docs @jesusdpa1. Basically when using tempdir the user has to clean up. The goal of TemporaryDirectory as a context manager is that it handles all the cleanup. Unfortunately it is a well known problem with Windows that the file is permanently locked once opened (until the full process completes) and it is actually non-trivial to get the file to close. In other OSes like macOS and Linux files to be deleted are flagged for deletion and deleted when the OS is able to, but on Windows it tries to do the process right away which leads to the failures. So the options for a Windows user are to try and try again until it works or use WSL to use Linux file system management rather than Windows. (Unless the context manager is removed from mountainsort5).
So I just got this Mountainsort5 error to repeat. This has come up before with trying to use TemporaryDirectory on Windows. In 3.12 they finally decided to add a delete argument just to prevent this error on windows. Then the user needs to cleanup themselves. @magland, is there any reason why you can’t do this the way
kilosort.pydoes this (ie, write the binary file to use and then give an argument to delete the binary itself later). Otherwise I wonder if you’ll lose Windows users. This seems to be intermittent failures in the cleanup. I can’t predict how or why this happens to be honest. The context manager has extra arguments added in 3.10 and 3.12 to make it useable by Windows users, but “useable” just means that it won’t try to delete and then the user would need to go delete anyway.