pyprep: Unable to finish running PyPREP

@barlehmann Yeah indeed Im seeing the problem. The ransac is saying that too many channels were marked as bad channels so it cannot continue to interpolate (which he needs to do to function). In particular this will happen if 25% of the number of good channels so far is less or equal to 3.

Now you could disable the ransac for now. For that you would need to set the following in your code:

prep = PrepPipeline(raw_copy, prep_params, montage,ransac=False)

I think @sappelhoff may have some better feedback regarding a workaround since he directly translated the ransac.

Using mne-python PR #7609 and changing the mne.filter.notch_filter(...) call in PrepPipeline.fit() to:

self.EEG_clean = mne.filter.notch_filter(
    self.EEG_new,
    Fs=self.sfreq,
    freqs=linenoise,
    method="spectrum_fit",
    mt_bandwidth=2,
    p_value=0.01,
    filter_length='10s'
)

seems to work fine for me. Without it, the fit() call hangs forever because scipy.signal.windows.dpss() is called with full length signals (no epoching), which takes forever.

@yjmantilla thank you so much for the assistance with this. That line of code works that sets prep to not use ransac works perfect - using this code is much better than nothing. Your help also makes me understand that the recordings we use will need to be of a higher quality. Also thank you @sappelhoff for clarifying this issue as well, I totally understand and greatly value both of your assistance on using this - It’s awesome that you have adapted PyPrep for python even if only at an alpha version so far - I am very glad I found out about this. If there are other free and similar EEG pre-processing-related pipelines for python (apart from this and FASTER) that you recommend as well, I will of course be glad to hear of anything that might be of use.

@sappelhoff and @wmvanvliet thank you both very much for your helpful feedback on the data cleaning process, and the importance of corroborating with your own ICA - I will definitely need to get started on learning this. Great to know also about the Faster implementation on python as well!

@yjmantilla Thank you so much, I actually got it to work right before you sent your last message - I really appreciate your help through this though. To give you more specific information on what changed for it to work: I had been having trouble with MNE reading the channel names from my .edf file, so I had to specifically draw this out in my code:

ch_names = [‘fp1’, ‘fp2’, ‘f3’, ‘f4’, ‘c3’, ‘c4’, ‘p3’, ‘p4’, ‘o1’, ‘o2’, ‘f7’, ‘f8’, ‘t3’, ‘t4’, ‘t5’, ‘t6’, ‘fz’, ‘cz’, ‘pz’] ch_types = [‘eeg’,‘eeg’,‘eeg’,‘eeg’,‘eeg’,‘eeg’,‘eeg’,‘eeg’,‘eeg’,‘eeg’, ‘eeg’,‘eeg’,‘eeg’,‘eeg’,‘eeg’,‘eeg’,‘eeg’,‘eeg’,‘eeg’] #ch_names = [‘FP1’, ‘FP2’, ‘F3’, ‘F4’, ‘C3’, ‘C4’, ‘P3’, ‘P4’, ‘O1’, ‘O2’, ‘F7’, ‘F8’, ‘T3’, ‘T4’, ‘T5’, ‘T6’, ‘FZ’, ‘CZ’, ‘PZ’, ‘A2-A1’] info = mne.create_info(ch_names=ch_names, sfreq=256, ch_types=ch_types) raw.info = info

However, this coding was, in some way (I am not aware exactly how), causing or contributing to the non-finishing pyprep. When I replaced the above code that ranames my channels with the following line of code:

raw.rename_channels(lambda s: s.split(’ ‘)[1].split(’-')[0])

The problem disappeared!

Also thank you @sappelhoff for your information on the patch. I greatly appreciate your support through this.

Lastly, I understand that PyPrep does some pre-processing, but does not deal with eye blink artifacts, or other user-generated muscle or motion artifacts. I just wanted to ask you guys if there is any additional automated pipeline with MNE that you recommend for removing such user-generated artifacts and/or performing ICA automatically?

Im suspecting the problem is actually the multitaper filter.

indeed … we’ve had several points now that showed that not all is well with that filter. +1 for a PR to include a way to skip this filter.

… that’d provide a short term workaround. In the longterm, we need to prepare a proper (working) implementation of that filter … or fix it in MNE.

Thats weird. Im taking a look at it.

• Does it show any errors?
• How much time have you let it run without apparently doing anything?

Its weird because 14 minutes of 19 channels at 256Hz shouldnt be that much of a computational stress. Specifically at that point it should be doing the robust referencing. That process can take a lot of time because of the RANSAC criteria.

In my case I’ve got 5 minutes of 58 channels at 1000Hz . That does take a long time to run for a single EEG (between 30minutes to 1 hour). Now in your case it shouldn’t take that long. Let it run for maximum an 1 hour. If it does not finish a single eeg recording then something weird is going on. It my estimates yours shouldn’t take more than 15 minutes.

Do notice that even though there is no output in the console, that does not mean it isn’t doing anything. Maybe we should put in the code more a kind of percentage output that feedbacks to the user the stage of the process that is being done.