pybids: Returning hidden files, names starting with dot
While trying to get all the anat images in a given BIDS folder (ds114_R2), I noticed I am receiving some hidden files too e.g. ._sub-09_ses-retest_T1w.nii.gz in addition to sub-09_ses-retest_T1w.nii.gz . Hidden files must have come from open-fmri or elsewhere (I didn’t create them). Was this by design or is it a bug? I can filter them at my end for now, but not returning them in the first place would be a better choice I think.
I queried it with bids_layout.get(**types) where types={'extensions': ('nii', 'nii.gz'), 'modality': 'anat'}
File(filename='/home/praamana/BIDS/ds114_R2/sub-09/ses-retest/anat/._sub-09_ses-retest_T1w.nii.gz', subject='09', session='retest', type='', modality='anat'),
File(filename='/home/praamana/BIDS/ds114_R2/sub-09/ses-retest/anat/sub-09_ses-retest_T1w.nii.gz', subject='09', session='retest', type='T1w', modality='anat'),
File(filename='/home/praamana/BIDS/ds114_R2/sub-09/ses-test/anat/._sub-09_ses-test_T1w.nii.gz', subject='09', session='test', type='', modality='anat'),
File(filename='/home/praamana/BIDS/ds114_R2/sub-09/ses-test/anat/sub-09_ses-test_T1w.nii.gz', subject='09', session='test', type='T1w', modality='anat'),
File(filename='/home/praamana/BIDS/ds114_R2/sub-10/ses-retest/anat/._sub-10_ses-retest_T1w.nii.gz', subject='10', session='retest', type='', modality='anat'),
File(filename='/home/praamana/BIDS/ds114_R2/sub-10/ses-retest/anat/sub-10_ses-retest_T1w.nii.gz', subject='10', session='retest', type='T1w', modality='anat'),
File(filename='/home/praamana/BIDS/ds114_R2/sub-10/ses-test/anat/._sub-10_ses-test_T1w.nii.gz', subject='10', session='test', type='', modality='anat'),
File(filename='/home/praamana/BIDS/ds114_R2/sub-10/ses-test/anat/sub-10_ses-test_T1w.nii.gz', subject='10', session='test', type='T1w', modality='anat')]
cc @chrisfilo
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 15 (5 by maintainers)
Commits related to this issue
- set validation to True by default; closes #244 — committed to bids-standard/pybids by tyarkoni 6 years ago
I’m a little inclined to exclude non-conformant files unless otherwise flagged (or given an additional spec that provides parsing information).
It’ll be a moderate pain while developing new apps that have as-yet-unstandardized entities, but I think the default behavior should be targeted at novices and reflect the constraints of the spec as much as possible.
What would be kind of nice is three levels: strict, permissive, and warn, where warn would behave permissively but raise warnings to let you know if something will be ignored or cause an error in strict mode. (I would also think of these as “error”, “silent”, and “warn”.) Then a natural progression of development would be to work in permissive until release time, when you switch to warn and figure out what needs standardization or custom spec modifications.
@KirstieJane derivatives will be handled differently… the thinking right now is that directories like
derivatives/andcode/will be excluded by default, but you can explicitly say things likeinclude=['derivatives/', 'code/']at initialization to index them. That’s independent of thevalidatesetting, which will apply within those directories too (i.e., you can have valid or non-valid files withinderivatives/).@raamana - I hear you on providing what is expected by a new user. That definitely supports
validate=Trueas the default.Anyone working in derivatives or an extension will need to set
validate=Falsewhich I think is not a small number of users of pybids…but as you said, they’re likely to be able to set the flag as needed (so long as its documented!)Thoughts, @effigies, @chrisfilo, @satra, @adelavega? The question, to save you some reading, is whether we should exclude or include non-BIDS files from pybids indexing by default. Right now the default is to not apply any validation, so this would be a change in behavior. But the next major version will introduce a bunch of breaking changes, so now would be a good time for this.