spikeinterface: "hard" merge of units in sorting analyzer fails on extensions computations
Hi all,
I am encountering an error during the extension computation (amplitude_scalings + spike_amplitudes + spike_locations) when applying curation (with merges) on an analyzer.
This error happens only when merging mode is set to "hard" in the call of apply_curation(). I’m not very familiar with the internal spikeinterface code for computing extensions so I’m struggling to debug this and I would love a bit of help. Note that when applying the curation to the sorting object, it works fine.
You’ll find here the analyzer, sorting and recording object together with the curation_dict.json file that I’m using so you can reproduce (hopefully) this behavior : https://instituteicm-my.sharepoint.com/:f:/g/personal/anthony_pinto_icm-institute_org/EhU1nQWpaNBFtVovmQaDUeEB5RJnkYReYfWu4ChMQ088OA?e=Aw9XSe
Here is also the code that you need to be able to reproduce :
import spikeinterface as si
import json
from spikeinterface.curation import apply_curation
raw_sorting = si.load_extractor(r'path_to_sorting')
raw_analyzer = si.load_sorting_analyzer(r'path_to_analyzer')
recording = si.load_extractor(r'path_to_recording')
raw_analyzer._recording = recording
with open(r'path_to_curation_dict', 'r') as f:
curation_dict = json.load(f)
print(recording), print(raw_sorting), print(raw_analyzer), print(curation_dict)
curated_analyzer = apply_curation(raw_analyzer, curation_dict, new_id_strategy='take_first', merging_mode='hard')
Help would be really appreciated ! Thank you !
Cheers, Anthony
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 34 (34 by maintainers)
Hello, I managed to reproduce the issue! I haven’t solved it yet but I’ve found something worth looking at. If I run
it’s fine. But the
all_nodeslist (a variable used inrun_node_pipelineinsortinganalyzer.py) is in a different order than when the merging takes place. For thecompute_several_extensions, all_nodes is equal towhile during the merging, all_nodes is equal to:
If I change the order:
I get the familiar error…
So I suspect it’s to do with the order of computing the extensions. @samuelgarcia is there a reason that
amplitude_scalingsshould be computed afterspike_amplitudes? Or maybe the nodes need to be reordered duringcompute_several_extensions?Should be fixed here: #3642
Hey @Antho2422 . Maybe while we try and figure this out, you could only compute
amplitude_scalingsat the very end. I guess you don’t need these to help figure out if you want to merge or not. So I suggestLoad your sorting analyzer Delete the amplitude scalings extension Do your merging Compute amplitude scalings (if you really need them!)
And we’ll let you know when this is fixed!
same
I think this depend on the numpy version installed that behave strictly or not. we need to fix anyway
I think, there are spikes too near the border and we are not safe enought after the merging in mode “hard”. But this is strange.
let’s see if we can reproduce it on windows!
Can you try from the anaconda prompt to do:
?
haha okay I trust you @alejoe91 ! Thank you
I am on main I’m 100% sure, the code I’m using is here and as you can see I have the updates from the last commit (except the one you just made a few minutes ago)
I already ran it with one job only same issue. I think it is not Python related aswell because I just tried with Python3.11 and I have the same issue. Another collegue managed to pass the line that was causing problem (On windows) (and got another issue). So maybe this has something to do with a dependency ? I will investigate and compare environments.
@alejoe91 oops I completly forgot this point !
Here it is :