mne-bids-pipeline: Critical error: Missing file: proc-ica_split-01_epo.fif

I have a pretty good machine but I’m surprised it takes so long on yours

If things are still the way I remember them, the data is stored on an NFS that is super slow esp during the initialization step(s)

and you can also for example set subjects = ['01'], hopefully this would run in minutes not hours!

I don’t know a quick workaround from the top of my head, sorry. I hope @larsoner may find some time to look into this, as I’m rather busy for the entire week. One temporary “solution” that comes to mind is making the epochs shorter such that you don’t end up with split files. I know this is not what you eventually will want to use, but that way you could at least produce a “preview dataset” your student can use to start implementing their analysis, until the “actual” dataset is ready

Oooohhhh good catch!

… stupid split files … We should probably have better handling for them in MNE-BIDS

Until then, we can probably work around this problem somehow

@larsoner I just saw that before the pipeline breaks, these two files were written, i.e. the filenames are wrong. I must have missed those before even though I looked through the folder, maybe because I ran several subjects in parallel (at least this seems a more likely explanation, than their magical appearance after running the same thing n times). I also ran it on a different machine, but with access to the same server for the data, and the same mne setup.

sub-155_task-tiwm_proc-ica_split-01_split-01_epo.fif sub-155_task-tiwm_proc-ica_split-01_split-02_epo.fif

Any idea how to prevent these erroneous filenames?

Modified the uploaded script to have:

bids_root = Path(__file__).parent
deriv_root = bids_root / "derivatives"

and re-ran. Observations unrelated to the issue that I figured you should know:

• You have ica_reject_components which is not used, I recommend removing it so that it’s clear it’s not needed. (I recently found a similar error in one of our test configs which is why I noticed!)
• All steps were recomputed for me as expected, and it only took ~20 minutes to get to the preprocessing/_08a_apply_ica step. I have a pretty good machine but I’m surprised it takes so long on yours if it’s hours for just subject 155 (?). After about an hour it was on some decoding step but the error happens before that so not necessary to keep going at that point.

I was unable to reproduce the error. Can you try latest main of MNE-Python, MNE-BIDS, and MNE-BIDS-Pipeline? Here’s my sys_info for those for reference:

├☑ mne               1.7.0.dev120+g38e2839a36 (devel, latest release is 1.6.1)
├☑ mne-bids          0.15.0.dev31+gb85a0dc7.d20240216
├☑ mne-bids-pipeline 1.6.0.dev18+g9f8b170

still running… for 1 SB with 8 runs - the epochs are very long though. I will report how it went tomorrow

I am getting a similar error again, it checks for the files that are supposed to be the output:

File "/home/sh254795/Documents/REPOS/mne-study-template/mne_bids_pipeline/_run.py", line 393, in _prep_out_files

    out_files = _path_to_str_hash(

  File "/home/sh254795/Documents/REPOS/mne-study-template/mne_bids_pipeline/_run.py", line 412, in _path_to_str_hash

    assert v.exists(), f'missing {kind}_files["{k}"] = {v}'

Argh … I’m sure @larsoner will have an idea here

But you’re saying that the file the pipeline complains about as being missing is, in fact, not missing now?

everything before 08_apply_ICA seems to work (it ran), but I cannot apply the ICA

01 and 02