phy: Assertion Error when opening spike sorted file
Hi,
I recently ran a recording through Kilosort2.5, where it reported to find spikes and good units (if few), but when I try to open it up in phy I get this error:
[33m16:07:42.861 [W] model:545 Unreferenced clusters found in templates (generally not a problem)[0m
[33m16:07:42.938 [W] model:567 Unreferenced clusters found in spike_clusters (generally not a problem)[0m
[33m16:07:42.965 [W] model:54 4279/40000 values are nan in C:\Users\idavalik\Data\Neuropixels\Socal_track_day5_g0\Socal_track_day5_g0_imec0\similar_templates.npy, replacing by zero.[0m
[31m16:07:43.826 [E] __init__:62 An error has occurred (AssertionError):
Traceback (most recent call last):
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\idavalik\AppData\Local\Continuum\anaconda3\envs\phy2\Scripts\phy.exe\__main__.py", line 7, in <module>
sys.exit(phycli())
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\click\core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\click\decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\phy\apps\__init__.py", line 159, in cli_template_gui
template_gui(params_path, **kwargs)
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\phy\apps\template\gui.py", line 199, in template_gui
gui = controller.create_gui()
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\phy\apps\base.py", line 1638, in create_gui
self.supervisor.attach(gui)
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\phy\cluster\supervisor.py", line 942, in attach
gui=gui, sort=gui.state.get('ClusterView', {}).get('current_sort', None))
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\phy\cluster\supervisor.py", line 760, in _create_views
gui, data=self.cluster_info, columns=self.columns, sort=sort)
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\phy\cluster\supervisor.py", line 916, in cluster_info
return [self.get_cluster_info(cluster_id) for cluster_id in self.clustering.cluster_ids]
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\phy\cluster\supervisor.py", line 916, in <listcomp>
return [self.get_cluster_info(cluster_id) for cluster_id in self.clustering.cluster_ids]
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\phy\cluster\supervisor.py", line 745, in get_cluster_info
out[key] = func(cluster_id)
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\phy\apps\base.py", line 1150, in get_best_channel_label
return self._get_channel_labels([self.get_best_channel(cluster_id)])[0]
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\phy\utils\context.py", line 154, in memcached
out = f(*args, **kwargs)
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\phy\apps\base.py", line 1144, in get_best_channel
channel_ids = self.get_best_channels(cluster_id)
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\phy\utils\context.py", line 154, in memcached
out = f(*args, **kwargs)
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\phy\apps\template\gui.py", line 149, in get_best_channels
template = self.model.get_template(template_id)
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\phylib\io\model.py", line 855, in get_template
return self._get_template_dense(template_id, channel_ids=channel_ids)
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\phylib\io\model.py", line 798, in _get_template_dense
channel_ids_, amplitude, best_channel = self._find_best_channels(template)
File "c:\users\idavalik\appdata\local\continuum\anaconda3\envs\phy2\lib\site-packages\phylib\io\model.py", line 787, in _find_best_channels
assert best_channel in channel_ids
AssertionError
[0m
The recording was very noisy and I fear/suspect some probe damage, but I have had no problems opening noisy files before.
About this issue
- Original URL
- State: open
- Created 4 years ago
- Comments: 43 (5 by maintainers)
I pushed a tentative fix on phylib (master branch), empty templates in KS2.5 are NaN which phy doesn’t support for some reason – the test dataset I have seems to load correctly after replacing NaNs with zeros
I am experiencing this issue as well. I tried the workaround suggested by @JoseGuzman without success. @rossant, is there any update? I am also willing to share the recording I am using via dropbox if that would be helpful.
@torbenott I was able to resolve this issue by downgrading Kilosort to the 2.0 release. I saw that @Hankslab-ABG tried this too which resulted in some weird sorting results, but in my case, the auto-sorting done by Kilosort2 looks perfectly reasonable. For what it’s worth I am using the Neuropixels 1.0 probes.
I tried re-running after deleting any of the files generated by KS2.5 but I still get this error. I don’t know the exact cause of this error but I suspect it has something to do with the fact that in my dataset (and maybe @Idavr as well?), a few channels are disabled. Then
assert best_channel in channel_ids
fails because the best channel for one cluster is mis-assigned to one of the disabled channels. @marius10p do you think this is possible?For some recordings I have been able to work around it by connecting all channels (including reference), but this issue is not consistent across all my recordings. In the cases where I am completely unable to work around it with KS2.5, I use KS2 without issues, but I haven’t found a consistent reason for these phy issues when spike-sorting with KS2.5.
Any updates on this?
For what it’s worth, I re-ran a couple of the datasets that wouldn’t open in Phy when using KS2.5. I used KS2 instead of 2.5 and was able to open them in Phy. However, what I found was that all of the clusters were based around very ill-defined waveforms that did not look spike-like at all. Has anyone else tried this? I know that some had mentioned noisy datasets might be behind this issue, so I’d be interested to see if anyone was able to get sensible-looking waveforms when re-running in KS2.
@rossant Sent!
sure!
@rossant would a Google Drive link work for you?
I’m now also getting a similar error trying to run phy on a KS 2.5 sorted session where a reference channel on an NPx probe was disabled. Any insights since the last posting?
Sent.
Any file-sharing like Dropbox, at cyrille.rossant at gmail.com
Le sam. 14 nov. 2020 à 16:28, Idavr notifications@github.com a écrit :
I’ve also tried re-running it multiple times, getting the same error. Since this happened after something funky happened with the channels during the actual recording (the above mentioned noise) I am unable to open any sessions done after this weirdness, even if the raw stream of the recording does not have the same kind of noise. There are still very - very - few good units being detected by KS2.5, so I can not say whether that is a confounding factor but when there has been no spikes or units previously I haven’t been able to run KS at all. A sudden channel mislabeling (for whatever reason) would make sense, but I am very interested to hear what Marius has to say on the subject.