spikeinterface: KS4: CUDNN_STATUS_NOT_SUPPORTED error

Hi,

I upgraded my spikeinterface to “0.101.0rc0” tried to run kilosort 4.0.12 with singularity image. However, I got the following error:

---------------------------------------------------------------------------                                                                                                                                                                                                                                                                                      
SpikeSortingError                         Traceback (most recent call last)                                                                                                                                                                   
File ~/Documents/git/analysis/analysis/preprocessing/test_ks4.py:119                                                                                                                                                                          
    116 os.chdir(preprocess_folder)                                                                                                                                                                                                           
    118 # sort spikes                                                                                                                                                                                                                         
--> 119 sorting = ss.run_sorter(                                                                                                                                                                                                              
    120     sorter_name='kilosort4',                                                                                                                                                                                                          
    121     recording=ap_mcd,                                                                                                                                                                                                                 
    122     folder=str(base_folder/"ks4_full"),                                                                                                                                                                                               
    123     singularity_image=True,                                                                                                                                                                                                           
    124     remove_existing_folder=True,                                                                                                                                                                                                      
    125     #**ks4_params,                                                                                                                                                                                                                    
    126 )                                                                                                                                                                                                                                     
    128 # curate                                                                                                                                                                                                                              
    129 sorting = scur.remove_duplicated_spikes(                                                                                                                                                                                              
    130     sorting,                                                                                                                                                                                                                          
    131     censored_period_ms=0.3,                                                                                                                                                                                                           
    132     method="keep_first_iterative",                                                                                                                                                                                                    
    133 )                                                                                                                                                                                                                                     
       
File ~/.conda/envs/ks4/lib/python3.11/site-packages/spikeinterface/sorters/runsorter.py:169, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, **sorter_params)
    167         else:
    168             container_image = singularity_image
--> 169     return run_sorter_container(
    170         container_image=container_image,
    171         mode=mode,
    172         **common_kwargs,
    173     )
    175 return run_sorter_local(**common_kwargs)

File ~/.conda/envs/ks4/lib/python3.11/site-packages/spikeinterface/sorters/runsorter.py:595, in run_sorter_container(sorter_name, recording, mode, container_image, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, delete_container_files, extra_requirements, installation_mode, spikeinterface_version, spikeinterface_folder_source, **sorter_params)
    593 if run_error:
    594     if raise_error:
--> 595         raise SpikeSortingError(f"Spike sorting in {mode} failed with the following error:\n{run_sorter_output}")
    596 else:
    597     if with_output:

SpikeSortingError: Spike sorting in singularity failed with the following error:
  0%|          | 0/844 [00:00<?, ?it/s]/home/miniconda3/lib/python3.11/site-packages/kilosort/spikedetect.py:134: UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at ../aten/src/ATen/native/cudnn/Conv_v8.cpp:919.)
  B = conv1d(X.unsqueeze(1), W, padding=nt//2)
100%|██████████| 844/844 [03:44<00:00,  3.76it/s]
100%|██████████| 844/844 [03:44<00:00,  3.76it/s]
100%|██████████| 96/96 [02:00<00:00,  1.25s/it]
/home/miniconda3/lib/python3.11/site-packages/kilosort/template_matching.py:78: UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at ../aten/src/ATen/native/cudnn/Conv_v8.cpp:919.)
  X = conv1d(X.unsqueeze(1), ops['wTEMP'].unsqueeze(1), padding=ops['nt']//2)
100%|██████████| 844/844 [01:07<00:00, 12.50it/s]
100%|██████████| 96/96 [01:47<00:00,  1.12s/it]
/home/amz/running_data/npx/raw/20240711_az_PANT06/si_preprocess/in_container_python_base/lib/python3.11/site-packages/spikeinterface/core/basesorting.py:264: UserWarning: The registered recording will not be persistent on disk, but only available in memory
  warnings.warn("The registered recording will not be persistent on disk, but only available in memory")

> /home/amz/.conda/envs/ks4/lib/python3.11/site-packages/spikeinterface/sorters/runsorter.py(595)run_sorter_container()
    593     if run_error:
    594         if raise_error:
--> 595             raise SpikeSortingError(f"Spike sorting in {mode} failed with the following error:\n{run_sorter_output}")
    596     else:
    597         if with_output:

Could anyone please help? Thanks!

Best, Arthur

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (5 by maintainers)

Most upvoted comments

Related:

https://github.com/SpikeInterface/spikeinterface/pull/3156

And:

https://github.com/SpikeInterface/spikeinterface/pull/3103

If it not too much to ask maybe you coul run from main with the suggestions that @zm711 gave on https://github.com/SpikeInterface/spikeinterface/issues/3192

Otherwise, a release is coming that should contain all of those fixes in the next week.

I think we had the ks4 fixes by rc0, but maybe not. You could also try from main in that case.

Hey @a-zmz,

this is an error in the kilosort runtime. You can see the error here is within the kilosort package.

%|          | 0/844 [00:00<?, ?it/s]/home/miniconda3/lib/python3.11/site-packages/kilosort/spikedetect.py:134: UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at ../aten/src/ATen/native/cudnn/Conv_v8.cpp:919.)

It would probably be best to open the issue directly on their repo so they can let you know what to do for the cudnn errors. Then if they say the problem is something with our wrapper you can bring the issues here. But since this error is inside kilosort they would know better than us.