fmriprep: fmriprep 1.4.1 file not found error

I am running a Singularity container of fmriprep 1.4.1. When we try to run the following command, we get a Traceback error indicating that it cannot find a file which I think it was supposed to have created itself.

The command I run is

singularity run -B /sw:/sw /tmp/fmriprep-1.4.1.simg $PWD --fs-license-file=/sw/arcts/centos7/freesurfer/6.0.0/license.txt --participant-label=sub-NDARINVZZZNB0XC output participant

I get the same traceback using that same command but with the added options --nthreads 1 --omp-nthreads 1.

It runs through setting up the nipype nodes successfully it seems, then prints up to the ### References, at which point it produces this output

190820-15:39:04,174 nipype.workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_NDARINVZZZNB0XC_wf.func_preproc_ses_baselineYear1Arm1_task_mid_run_01_wf.func_derivatives_wf.raw_sources" in "/tmp/fmriprep/work/fmriprep_wf/single_subject_NDARINVZZZNB0XC_wf/func_preproc_ses_baselineYear1Arm1_task_mid_run_01_wf/func_derivatives_wf/raw_sources".
190820-15:39:04,175 nipype.workflow INFO:
	 [Node] Running "raw_sources" ("nipype.interfaces.utility.wrappers.Function")
exception calling callback for <Future at 0x2af37ff648d0 state=finished raised OSError>
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 487, in run
    self, report_type='postexec', is_mapnode=isinstance(self, MapNode))
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 149, in write_report
    result = node.result  # Locally cache result
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 197, in result
    return _load_resultfile(self.output_dir(), self.name)[0]
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 359, in load_resultfile
    basedir=path).items()):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 478, in modify_paths
    val, relative=relative, basedir=basedir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 484, in modify_paths
    modify_paths(val, relative=relative, basedir=basedir))
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 498, in modify_paths
    raise IOError('File %s not found' % out)
OSError: File /tmp/fmriprep/work/fmriprep_wf/single_subject_NDARINVZZZNB0XC_wf/func_preproc_ses_baselineYear1Arm1_task_mid_run_01_wf/func_derivatives_wf/raw_sources/sub-NDARINVZZZNB0XC/ses-baselineYear1Arm1/func/sub-NDARINVZZZNB0XC_ses-baselineYear1Arm1_task-mid_run-01_bold.nii not found

It appears that this directory does exist: /tmp/fmriprep/work/fmriprep_wf/single_subject_NDARINVZZZNB0XC_wf/func_preproc_ses_baselineYear1Arm1_task_mid_run_01_wf/func_derivatives_wf/raw_sources, but that there is no subject directory underneath it.

$ ls /tmp/fmriprep/work/fmriprep_wf/single_subject_NDARINVZZZNB0XC_wf/func_preproc_ses_baselineYear1Arm1_task_mid_run_01_wf/func_derivatives_wf/raw_sources
_0x763007af6eb61212a00403b1b1920887.json  _inputs.pklz  _node.pklz  _report  result_raw_sources.pklz

Under the BIDS_ROOT directory, the path sub-NDARINVZZZNB0XC/ses-baselineYear1Arm1/func/sub-NDARINVZZZNB0XC_ses-baselineYear1Arm1_task-mid_run-01_bold.nii.

The same subject gets a good deal past that stage when run using a Singularity container built for fmriprep-1.1.8, so I think it is not a problem with the data. Since we tried this with the options to limit threading to 1 process and 1 thread, it should not be asynchronous threading.

The system on which I am running is CentOS Linux release 7.6.1810 (Core) on a Dell R740 with dual Intel® Xeon® Gold 6140 CPU @ 2.30GHz chips. The kernel version is 3.10.0-957.10.1.el7.x86_64.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 16 (6 by maintainers)

Most upvoted comments

We’ve been tearing our hair out over the same error. fmriprep 1.4.1 running in a singularity container, works on some subjects, throws the same error you got on others. I’m glad it’s not just us…

@effigies just an FYI, this issue does seem to be very related to issue #1758. I tried launching fmriprep in the same way as in the end of that thread, but without using the workaround.yml file and it does seem to get beyond the raw_sources step quickly.

Define “slowly”? We have no progress after 20 hours.

Yes, we were prematurely jubilant, because we got past our initial blocker. We ran into the same problem.