spikeinterface: SpikeGLXRecordingExtractor fails when gate > 0

Hi @samuelgarcia and all, Initializing a spikeinterface.extractors.SpikeGLXRecordingExtractor object from a probe directory with gate index 0 works properly:

$ ls subject_dir/run_name_g0/run_name_g0_imec0
1-6-2022_g0_t0.imec0.lf.bin   1-6-2022_g0_t1.imec0.lf.bin   1-6-2022_g0_t2.imec0.lf.bin
1-6-2022_g0_t0.imec0.lf.meta  1-6-2022_g0_t1.imec0.lf.meta  1-6-2022_g0_t2.imec0.lf.meta

se.SpikeGLXRecordingExtractor("subject_dir/run_name_g0/run_name_g0_imec0", "imec0.lf") # Assuming folder-per-probe organisation

SpikeGLXRecordingExtractor: 384 channels - 1 segments - 2.5kHz - 7200.001s

… But it fails when the gate index is >0

$ ls subject_dir/run_name_g1/run_name_g1_imec0
1-6-2022_g1_t0.imec0.lf.bin   1-6-2022_g1_t2.imec0.lf.meta  1-6-2022_g1_t5.imec0.lf.bin   1-6-2022_g1_t7.imec0.lf.meta
1-6-2022_g1_t0.imec0.lf.meta  1-6-2022_g1_t3.imec0.lf.bin   1-6-2022_g1_t5.imec0.lf.meta  1-6-2022_g1_t8.imec0.lf.bin
1-6-2022_g1_t1.imec0.lf.bin   1-6-2022_g1_t3.imec0.lf.meta  1-6-2022_g1_t6.imec0.lf.bin   1-6-2022_g1_t8.imec0.lf.meta
1-6-2022_g1_t1.imec0.lf.meta  1-6-2022_g1_t4.imec0.lf.bin   1-6-2022_g1_t6.imec0.lf.meta  1-6-2022_g1_t9.imec0.lf.bin
1-6-2022_g1_t2.imec0.lf.bin   1-6-2022_g1_t4.imec0.lf.meta  1-6-2022_g1_t7.imec0.lf.bin   1-6-2022_g1_t9.imec0.lf.meta

se.SpikeGLXRecordingExtractor("subject_dir/run_name_g1/run_name_g1_imec0", "imec0.lf")

KeyError                                  Traceback (most recent call last)
Input In [47], in <cell line: 1>()
----> 1 se.SpikeGLXRecordingExtractor('.', 'imec0.lf')

File ~/projects/spikeinterface/spikeinterface/extractors/neoextractors/spikeglx.py:38, in SpikeGLXRecordingExtractor.__init__(self, folder_path, stream_id)
     36 if HAS_NEO_10_2:
     37     neo_kwargs['load_sync_channel'] = False
---> 38 NeoBaseRecordingExtractor.__init__(self, stream_id=stream_id, **neo_kwargs)
     40 #~ # open the corresponding stream probe
     41 if HAS_NEO_10_2 and '.ap' in self.stream_id:

File ~/projects/spikeinterface/spikeinterface/extractors/neoextractors/neobaseextractor.py:26, in NeoBaseRecordingExtractor.__init__(self, stream_id, **neo_kwargs)
     24 def __init__(self, stream_id=None, **neo_kwargs):
---> 26     _NeoBaseExtractor.__init__(self, **neo_kwargs)
     28     # check channel
     29     # TODO propose a meachanisim to select the appropriate channel groups
     30     # in neo one channel group have the same dtype/sampling_rate/group_id
     31     # ~ channel_indexes_list = self.neo_reader.get_group_signal_channel_indexes()
     32     stream_channels = self.neo_reader.header['signal_streams']

File ~/projects/spikeinterface/spikeinterface/extractors/neoextractors/neobaseextractor.py:16, in _NeoBaseExtractor.__init__(self, **neo_kwargs)
     14 neoIOclass = eval('neo.rawio.' + self.NeoRawIOClass)
     15 self.neo_reader = neoIOclass(**neo_kwargs)
---> 16 self.neo_reader.parse_header()
     18 assert self.neo_reader.block_count() == 1, \
     19     'This file is neo multi block spikeinterface support one block only dataset'

File ~/miniconda3/envs/rebooting/lib/python3.10/site-packages/neo/rawio/baserawio.py:185, in BaseRawIO.parse_header(self)
    172 def parse_header(self):
    173     """
    174     This must parse the file header to get all stuff for fast use later on.
    175
   (...)
    183
    184     """
--> 185     self._parse_header()
    186     self._check_stream_signal_channel_characteristics()

File ~/miniconda3/envs/rebooting/lib/python3.10/site-packages/neo/rawio/spikeglxrawio.py:106, in SpikeGLXRawIO._parse_header(self)
    103 signal_channels = []
    104 for stream_name in stream_names:
    105     # take first segment
--> 106     info = self.signals_info_dict[0, stream_name]
    108     stream_id = stream_name
    109     stream_index = stream_names.index(info['stream_name'])

KeyError: (0, 'imec0.lf')

Same issue when instantiating directly the SpikeGLXRawIO object from neo. It seems like the gate (and not only the stream) should be either passed as a kwarg or parsed from the folder name?

Thanks!

Spikeinterface version = 0.94.1.dev0 neo version = 0.12.2

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 31 (18 by maintainers)

Most upvoted comments

@grahamfindlay @billkarsh : I propose to move the discussion where I did the code here : https://github.com/NeuralEnsemble/python-neo/pull/1125

samuelgarcia on Jun 1, 2022

Hi Graham. Thanks a lot for this. I will push myself to gin repo. This look absolutly perfect. bin files are small and I think we cover many case.

samuelgarcia on May 20, 2022

I am also adding here a collection of CatGT output produced from these sample files. This output uses the latest version of CatGT (v3.0). It should be enough to properly test ingest of CatGT output in Neo (it covers concatenation of t-files within and across g-indices, concatenation of continuous and discontinuous recordings, supercat concatenation of multiple runs, and output in both folder-per-probe and single-folder format).

Data: sample_data_v2.zip

grahamfindlay on May 20, 2022

I’m a super friendly guy, trust me. I don’t have any free time and have a very full agenda already. I’m not keen on working on a parallel version in SI (or other) of what I already cleared off my plate in CatGT. It’s a command line tool so that it can plug into other workflows. Also the source code is there which answers essentially all questions about how I actually do it, so attending meetings to say “how I think I did it” wouldn’t even be as accurate. BTW, searching the CatGT sources for [“xxx”] finds CatGT reads of metadata items, so you can see what items are required and how they are parsed and interpreted. That’s my first order answer while I reconsider, since I am trying to help as best I can.

billkarsh on May 19, 2022

(@alejoe91 I just updated my comment)

TomBugnon on May 18, 2022