spikeinterface: `ProcessPoolExecutor` hanging in `compute_principal_components` and `compute_quality_metrics`

Starting from version 0.100.0, execution of compute_principal_components sometimes and compute_quality_metrics always (when requesting PCA metrics) hang at 0% when I use n_jobs > 1. To work around this, I have reverted portions of these functions involving ProcessPoolExecutor to their 0.99.1 versions that instead use joblib with appropriate changes to upstream functions. This allows me to use parallel processing with version 0.100.5.

Is anyone else experiencing a similar issue? I’m using Ubuntu 20.04.5 LTS and python 3.10.14.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 16 (9 by maintainers)

Most upvoted comments

Maybe a dedicated “Troubleshooting” page in the docs?

Cool, I tried a .ipynb notebook in Jupyter, and it reproduces the behavior of .py scripts. So it finally fails when computing quality metrics in parallel.

Ah thanks for pointing that out! I swapped my recording and sorting because I didn’t use keyword arguments for them.

Now create_sorting_analyzer() seems to work; I call it as follows:

analyzer = sf.create_sorting_analyzer(sorting=sorting, recording=filtered,
                                      format='binary_folder', folder=analyzer_path,
                                      sparse=True, n_jobs=8)

I now have narrowed down the problem a bit further. It seems that the hang only occurs during or after the second time a parallelized analyzer.compute() is called, and somehow it’s worse for ipynb notebooks (I’m using VSCode) and for more jobs.

For example, the following works for .ipynb notebooks and .py scripts:

analyzer.compute('random_spikes')
analyzer.compute('waveforms')
analyzer.compute('templates')
analyzer.compute('principal_components', n_jobs=8)

The following also works for .ipynb notebooks and .py scripts:

analyzer.compute('random_spikes')
analyzer.compute('waveforms', n_jobs=2)
analyzer.compute('templates')
analyzer.compute('principal_components', n_jobs=2)

The following hangs on computing principal components for .ipynb notebooks, but it works for .py scripts:

analyzer.compute('random_spikes')
analyzer.compute('waveforms', n_jobs=8)
analyzer.compute('templates')
analyzer.compute('principal_components', n_jobs=8)

And the following hangs on computing quality metrics even for .py scripts:

analyzer.compute('random_spikes')
analyzer.compute('waveforms', n_jobs=2)
analyzer.compute('templates')
analyzer.compute('principal_components', n_jobs=2)
analyzer.compute('spike_amplitudes')
analyzer.compute('noise_levels')
analyzer.compute('correlograms')
analyzer.compute('template_similarity')
analyzer.compute('quality_metrics', n_jobs=2)

So it appears that notebooks hang on the second parallelized analyzer.compute() and scripts hang on the third [edit: Actually this is a bit too simplistic… in any case, it seems that PCA-related computations are particularly troublesome.]. To be clear, “hang” for me means that the progress bar doesn’t change for a long time and the CPUs show no usage. I’ve also tried using format='memory' for create_sorting_analyzer() with the same outcome. Interestingly, analyzer.compute() doesn’t mind that I run other functions beforehand in parallel, like create_sorting_analyzer(), and recording.save()