spikeinterface: Binary file saving doesn't allow multiple jobs

Running recording.save(folder=file_path_pre,n_jobs=2,format='binary') results in the error trace below. Anything above n_jobs=1 causes this error, so it must have something to do with parallel processing. It seems that my full python script is rerun (i.e., everything I print to terminal before calling .save is printed again). This causes .save to create the output folder multiple times, resulting in the error below.

Traceback (most recent call last): File “<string>”, line 1, in <module> File “C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1008.0_x64__qbz5n2kfra8p0\Lib\multiprocessing\spawn.py”, line 120, in spawn_main exitcode = _main(fd, parent_sentinel) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1008.0_x64__qbz5n2kfra8p0\Lib\multiprocessing\spawn.py”, line 129, in _main prepare(preparation_data) File “C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1008.0_x64__qbz5n2kfra8p0\Lib\multiprocessing\spawn.py”, line 240, in prepare _fixup_main_from_path(data[‘init_main_from_path’]) File “C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1008.0_x64__qbz5n2kfra8p0\Lib\multiprocessing\spawn.py”, line 291, in _fixup_main_from_path main_content = runpy.run_path(main_path, ^^^^^^^^^^^^^^^^^^^^^^^^^ File “<frozen runpy>”, line 291, in run_path File “<frozen runpy>”, line 98, in _run_module_code File “<frozen runpy>”, line 88, in _run_code File “C:\Users\Cheetah\PycharmProjects\spikeInterface\main.py”, line 65, in <module> recording_saved = recording_cmr.save(folder=file_path_pre,n_jobs=2, format=‘binary’) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\Cheetah\PycharmProjects\spikeInterface\venv\Lib\site-packages\spikeinterface\core\base.py”, line 621, in save loaded_extractor = self.save_to_folder(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\Cheetah\PycharmProjects\spikeInterface\venv\Lib\site-packages\spikeinterface\core\base.py”, line 684, in save_to_folder assert not folder.exists(), f’folder {folder} already exists, choose another name’ AssertionError: folder C:\CheetahData\bench\preprocessed\2023-04-13_13-56-33 already exists, choose another name Traceback (most recent call last): File “<string>”, line 1, in <module> File “C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1008.0_x64__qbz5n2kfra8p0\Lib\multiprocessing\spawn.py”, line 120, in spawn_main exitcode = _main(fd, parent_sentinel) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1008.0_x64__qbz5n2kfra8p0\Lib\multiprocessing\spawn.py”, line 129, in _main prepare(preparation_data) File “C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1008.0_x64__qbz5n2kfra8p0\Lib\multiprocessing\spawn.py”, line 240, in prepare _fixup_main_from_path(data[‘init_main_from_path’]) File “C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1008.0_x64__qbz5n2kfra8p0\Lib\multiprocessing\spawn.py”, line 291, in _fixup_main_from_path main_content = runpy.run_path(main_path, ^^^^^^^^^^^^^^^^^^^^^^^^^ File “<frozen runpy>”, line 291, in run_path File “<frozen runpy>”, line 98, in _run_module_code File “<frozen runpy>”, line 88, in _run_code File “C:\Users\Cheetah\PycharmProjects\spikeInterface\main.py”, line 65, in <module> recording_saved = recording_cmr.save(folder=file_path_pre,n_jobs=2, format=‘binary’) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\Cheetah\PycharmProjects\spikeInterface\venv\Lib\site-packages\spikeinterface\core\base.py”, line 621, in save loaded_extractor = self.save_to_folder(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\Cheetah\PycharmProjects\spikeInterface\venv\Lib\site-packages\spikeinterface\core\base.py”, line 684, in save_to_folder assert not folder.exists(), f’folder {folder} already exists, choose another name’ AssertionError: folder C:\CheetahData\bench\preprocessed\2023-04-13_13-56-33 already exists, choose another name write_binary_recording: 0%| | 0/6585 [00:02<?, ?it/s] Traceback (most recent call last): File “C:\Users\Cheetah\PycharmProjects\spikeInterface\main.py”, line 65, in <module> recording_saved = recording_cmr.save(folder=file_path_pre,n_jobs=2, format=‘binary’) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\Cheetah\PycharmProjects\spikeInterface\venv\Lib\site-packages\spikeinterface\core\base.py”, line 621, in save loaded_extractor = self.save_to_folder(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\Cheetah\PycharmProjects\spikeInterface\venv\Lib\site-packages\spikeinterface\core\base.py”, line 700, in save_to_folder cached = self._save(folder=folder, verbose=verbose, **save_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\Cheetah\PycharmProjects\spikeInterface\venv\Lib\site-packages\spikeinterface\core\baserecording.py”, line 297, in _save write_binary_recording(self, file_paths=file_paths, dtype=dtype, **job_kwargs) File “C:\Users\Cheetah\PycharmProjects\spikeInterface\venv\Lib\site-packages\spikeinterface\core\core_tools.py”, line 280, in write_binary_recording executor.run() File “C:\Users\Cheetah\PycharmProjects\spikeInterface\venv\Lib\site-packages\spikeinterface\core\job_tools.py”, line 364, in run for res in results: File “C:\Users\Cheetah\PycharmProjects\spikeInterface\venv\Lib\site-packages\tqdm\std.py”, line 1178, in iter for obj in iterable: File “C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1008.0_x64__qbz5n2kfra8p0\Lib\concurrent\futures\process.py”, line 602, in _chain_from_iterable_of_lists for element in iterable: File “C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1008.0_x64__qbz5n2kfra8p0\Lib\concurrent\futures_base.py”, line 619, in result_iterator yield _result_or_cancel(fs.pop()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1008.0_x64__qbz5n2kfra8p0\Lib\concurrent\futures_base.py”, line 317, in _result_or_cancel return fut.result(timeout) ^^^^^^^^^^^^^^^^^^^ File “C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1008.0_x64__qbz5n2kfra8p0\Lib\concurrent\futures_base.py”, line 456, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File “C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1008.0_x64__qbz5n2kfra8p0\Lib\concurrent\futures_base.py”, line 401, in __get_result raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Process finished with exit code 1