mne-bids: problem with overwrite=True for raw ctf data

Hi mne-bids folks! I am in the process of converting my raw CTF data to BIDS format before analyses. My raw dir is as such:

|--------------------------- mASD03_MMNASD-pduret_20181113_03.ds
|------------------------------ mASD03_MMNASD-pduret_20181113_03.acq
|------------------------------ mASD03_MMNASD-pduret_20181113_03.eeg
|------------------------------ bad.segments.bak
|------------------------------ bad.segments
|------------------------------ ChannelGroupSet.cfg
|------------------------------ mASD03_MMNASD-pduret_20181113_03.infods
|------------------------------ MarkerFile.mrk
|------------------------------ mASD03_MMNASD-pduret_20181113_03.hist
|------------------------------ BadChannels
|------------------------------ mASD03_MMNASD-pduret_20181113_03.hc
|------------------------------ mASD03_MMNASD-pduret_20181113_03.res4
|------------------------------ mASD03_MMNASD-pduret_20181113_03.newds
|------------------------------ ClassFile.cls
|------------------------------ MarkerFile.mrk.bak
|------------------------------ mASD03_MMNASD-pduret_20181113_03.infods.bak
|------------------------------ mASD03_MMNASD-pduret_20181113_03.meg4
|------------------------------ hz.ds
|--------------------------------- hz.res4
|--------------------------------- hz.meg4
|--------------------------------- hz.hist
|--------------------------------- hz.hc
|--------------------------------- hz.newds
|--------------------------------- BadChannels
|--------------------------------- hz.acq
|--------------------------------- hz.infods

And my output dir is as such:

--------------------- mMNE
|------------------------ mASD03_MMNASD-pduret_20181113_03.eeg
|------------------------ participants.tsv
|------------------------ participants.json
|------------------------ dataset_description.json
|------------------------ sub-mASD03
|--------------------------- sub-mASD03_scans.tsv
|--------------------------- meg
|------------------------------ sub-mASD03_task-ContextOdd_run-2_channels.tsv
|------------------------------ sub-mASD03_task-ContextOdd_run-2_events.tsv
|------------------------------ sub-mASD03_coordsystem.json
|------------------------------ sub-mASD03_task-ContextOdd_run-2_meg.json
|------------------------------ sub-mASD03_task-ContextOdd_run-2_meg.ds
|--------------------------------- mASD03_MMNASD-pduret_20181113_03.acq
|--------------------------------- mASD03_MMNASD-pduret_20181113_03.eeg
|--------------------------------- bad.segments.bak
|--------------------------------- bad.segments
|--------------------------------- ChannelGroupSet.cfg
|--------------------------------- mASD03_MMNASD-pduret_20181113_03.infods
|--------------------------------- MarkerFile.mrk
|--------------------------------- mASD03_MMNASD-pduret_20181113_03.hist
|--------------------------------- BadChannels
|--------------------------------- mASD03_MMNASD-pduret_20181113_03.hc
|--------------------------------- mASD03_MMNASD-pduret_20181113_03.res4
|--------------------------------- mASD03_MMNASD-pduret_20181113_03.newds
|--------------------------------- ClassFile.cls
|--------------------------------- MarkerFile.mrk.bak
|--------------------------------- mASD03_MMNASD-pduret_20181113_03.infods.bak
|--------------------------------- mASD03_MMNASD-pduret_20181113_03.meg4
|--------------------------------- hz.ds
|------------------------------------ hz.res4
|------------------------------------ hz.meg4
|------------------------------------ hz.hist
|------------------------------------ hz.hc
|------------------------------------ hz.newds
|------------------------------------ BadChannels
|------------------------------------ hz.acq
|------------------------------------ hz.infods

As you can see, the data is copied but what’s inside the .ds folder does not get renamed correctly. This is for the first iteration of the loop over subj and runs, the script would break and I get the following error:

Copying data files to /sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/mMNE/sub-mASD03/meg/sub-mASD03_task-ContextOdd_run-2_meg.ds
Traceback (most recent call last):

  File "<ipython-input-40-672e36af4084>", line 16, in <module>
    event_id=event_id, overwrite=True)

  File "/pbs/home/p/pduret/anaconda3/lib/python3.7/site-packages/mne_bids/mne_bids.py", line 736, in write_raw_bids
    copyfile_ctf(raw_fname, bids_fname)

  File "/pbs/home/p/pduret/anaconda3/lib/python3.7/site-packages/mne_bids/utils.py", line 531, in copyfile_ctf
    sh.copytree(src, dest)

  File "/pbs/home/p/pduret/anaconda3/lib/python3.7/shutil.py", line 359, in copytree
    raise Error(errors)

Error: [('/sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/Raw_Data/MEG/mASD03_MMNASD-pduret_20181113_03.ds/mASD03_MMNASD-pduret_20181113_03.acq', '/sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/mMNE/sub-mASD03/meg/sub-mASD03_task-ContextOdd_run-2_meg.ds/mASD03_MMNASD-pduret_20181113_03.acq', "[Errno 22] Invalid argument: '/sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/Raw_Data/MEG/mASD03_MMNASD-pduret_20181113_03.ds/mASD03_MMNASD-pduret_20181113_03.acq'"), ('/sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/Raw_Data/MEG/mASD03_MMNASD-pduret_20181113_03.ds/mASD03_MMNASD-pduret_20181113_03.eeg', '/sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/mMNE/sub-mASD03/meg/sub-mASD03_task-ContextOdd_run-2_meg.ds/mASD03_MMNASD-pduret_20181113_03.eeg', "[Errno 22] Invalid argument: '/sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/Raw_Data/MEG/mASD03_MMNASD-pduret_20181113_03.ds/mASD03_MMNASD-pduret_20181113_03.eeg'"), ('/sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/Raw_Data/MEG/mASD03_MMNASD-pduret_20181113_03.ds/bad.segments.bak', '/sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/mMNE/sub-mASD03/meg/sub-mASD03_task-ContextOdd_run-2_meg.ds/bad.segments.bak', "[Errno 22] Invalid argument: '/sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/Raw_Data/MEG/mASD03_MMNASD-pduret_20181113_03.ds/bad.segments.bak'"), ` .....

And so on for all the files in the .ds . I don’t get why this is happening, is there something wrong in my data architecture?

Thank you!

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 17 (9 by maintainers)

Most upvoted comments

@paulineduret you have become quite a hacker!

okay, so it seems the clean solution would be to use Python 3.7, right? Apparently, they cannot backport to Python 3.6 anymore. I think this is useful information and we should document this somewhere.

+2
jasmainak on Feb 25, 2019

Hi, I finally found interesting documentation on my error here: https://bugs.python.org/issue24564

It is not a permission issue but an issue linked to python and NFS. Does anyone know how if this getxattr is important or if there is a way to skip this step?

Thx

+1
paulineduret on Feb 25, 2019

I had a very similar issue when working on BIDSHandler. I found shutil gives errors like that when it tries to move/copy files to a location that would have a file name longer than the maximum value allowed on an operating system. Not sure if this is the same issue but it may be something to keep in mind…

+1
monkeyman192 on Feb 11, 2019
Read more comments on GitHub
← mne-bids: Problem with accentuated characters
mne-bids: read_raw_bids() respects info['bads'], but it really shouldn't →