mne-bids: Splits are fetched by BIDSPath.fpath only when extension (and/or) suffix are missing

Is there a reason for such behavior? I’m asking because I’m using BIDSPath as a kind of template tool for paths in my dataset. For instance, I like to use the same BIDSPath for writing a file and then loading it at the later processing steps.

The problem occurs when occasionally file sizes exceed the 2 Gb limit and have to be split. In this case, writing is still fine with split=None since MNE Python supports bids-style splits when saving raw data. Reading on the other hand becomes problematic because I have to either keep track of all the files which happen to have splits and update my templates separately, which is tedious or use the template BIDSPaths with suffix or extension unset, which I didn’t realize until looking at the source code and which seem a little random to guess.

https://github.com/mne-tools/mne-bids/blob/74ac41a3f2035e4469f9930e725fe36f53f07146/mne_bids/path.py#L414-L446

So my question is can splits be handled irrespective of suffix and extension? And also can this fetching behavior of BIDSPath.fpath (and BIDSPath.__str__ for that matter) become more transparent? At least mentioning these rules in the docstrings would make a big difference I think.

Also maybe adding some flag like fetch_from_filesystem=True to make the behavior more explicit and controllable.

About this issue

Original URL
State: open
Created 3 years ago
Reactions: 1
Comments: 24 (15 by maintainers)

Most upvoted comments

defaulting to ‘.’ yes would simplify things

agramfort on Mar 1, 2021

@adam2392 can you comment in to clarify what you expect fpath to return in different situations? It will be the basis of an improved docstring

and sorry @dmalt for the slow reaction and the time it took me to understand the issue…

I think the documentation could definitely be improved.

Rn:

if root is not passed in, then fpath is just a relative path: e.g. sub-01/ses-01/<datatype>/sub-01_ses-01_task-01_run-01_<suffix>.<extension>
If root is passed in, and a full filename is defined (e.g. suffix and extension), then it will just be like the above, but with root prepended: <root>/sub-01/ses-01/<datatype>/sub-01_ses-01_task-01_run-01_<suffix>.<extension>
if root is passed in, then a full fpath will try to get inferred if one is missing the suffix and extension.

The third case is probably where it’s potentially weird, and I also don’t get what’s going on w/ split files cuz I don’t use them. So in this case, you might store data as BrainVision, and so you could theoretically leave out the extension='vhdr', and you can still use fpath to get the full filepath to the dataset (assuming you uniquely defined it with the other BIDS entities). Same w/ suffix.

adam2392 on Feb 28, 2021

And I do care about setting both extension and suffix because I want to use the same BIDSPath for both reading and writing data. It feels logical and readable and requires very little code. I managed to make it work for my project, but the process was rough for me which is why I wanted to discuss it here.

dmalt on Feb 22, 2021

if the files are split then

bp_template.fpath is indeed bids_root/sub-01/meg/sub-01_split-01_meg.fif

I don’t understand why you say you need to update your template manually. mne_bids should as shown above direct you to split-01 file automatically.

i’ve had to deal with split files in a real dataset using https://github.com/mne-tools/mne-study-template/ and I had no problem after my last fix.

agramfort on Feb 21, 2021

Discussion in #692 is relevant I think

dmalt on Feb 20, 2021