fmriprep: Low process ulimits can result in missing output failures (here in fsl2itk)

What happened?

Tried running several participants from two different datasets with a singularity container built from the most recent fmriprep release and they all died with an error from fsl2itk. The fsl2itk node says that it couldn’t find MAT_0000 in the mcflirt directory, but when I check the working directory, MAT_0000 is present at the path and looks like a valid affine transformation.

In the commands below, I know one of them had a smriprep directory monkeypatched in, but that’s the subject where I saved the working directory and am able to debug a bit more. The other command there for 24704 doesn’t have the monkey patch and threw the same error.

What command did you use?

(   export TMPDIR=/lscratch/$SLURM_JOB_ID \
    &&     . /usr/local/current/singularity/app_conf/sing_binds \
    &&    mkdir -p $TMPDIR/out \
    &&     mkdir -p $TMPDIR/wrk \
    &&     singularity run --cleanenv \
               -B /data/EDB/TMSpilot/code/smriprep/smriprep:/opt/conda/envs/fmriprep/lib/python3.10/site-packages/smriprep \
               /data/EDB/TMSpilot/code/fmriprep-23.1.3.simg /data/MLDSST/nielsond/target_test/data/ds002330     \
               $TMPDIR/out participant --participant_label 02 -w $TMPDIR/wrk \
               --nprocs $SLURM_CPUS_PER_TASK --mem_mb $SLURM_MEM_PER_NODE \
               --fs-license-file /data/MLDSST/nielsond/target_test/notebooks/license.txt \
               --output-spaces T1w MNI152NLin6Asym:res-2 fsLR --dummy-scans 5 --use-aroma \
               --error-on-aroma-warnings --cifti-output 91k -vvv;
    [...]
)

( export TMPDIR=/lscratch/${SLURM_JOBID} ;  \
   lscratch_mon --detail --log /data/EDB/TMSpilot/slurm/${SLURM_JOBID}.lscratch --interval 2 \
   & mkdir -p ${TMPDIR}/24704.out ${TMPDIR}/24704.wrk ; \
  singularity run /data/EDB/TMSpilot/code/fmriprep-23.1.3.simg /data/EDB/TMSpilot/BIDS \
  ${TMPDIR}/24704.out participant --participant_label 24704 -w ${TMPDIR}/24704.wrk  \
   --use-aroma --notrack --output-space MNI152NLin6Asym:res-2 T1w fsaverage fsLR fsnative \
   --nthreads ${SLURM_CPUS_PER_TASK} --omp-nthreads ${SLURM_CPUS_PER_TASK} --me-output-echos \
   --cifti-output 91k --skip_bids_validation --fs-license /data/EDB/TMSpilot/code/license.txt \
    --project-goodvoxels ;        \
   rsync -a --chown=:EDB ${TMPDIR}/24704.out/ /data/EDB/TMSpilot/derivatives/fmriprep-23-1-1.bugfix/ ;  \
    rm -rf ${TMPDIR}/24704.out ${TMPDIR}/24704.wrk )

What version of fMRIPrep are you running?

23.1.3

How are you running fMRIPrep?

Singularity

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

Error:

 File: /lscratch/4382593/out/sub-02/log/20230712-124027_8d2929db-51a7-4d50-9593-8454ce70c643/crash-20230712-124447-nielsond-fsl2itk-d31fdad1-d569-4b4d-8241-90aea122985d.txt
Working Directory: /lscratch/4382593/wrk/fmriprep_23_1_wf/single_subject_02_wf/func_preproc_task_rest_run_01_wf/bold_hmc_wf/fsl2itk
Inputs:

    in_files:
    in_reference:
    in_source:
    num_threads: 8

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node fsl2itk.

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
	    setattr(outputs, key, val)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
	    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
	    self.error(objekt, name, str(value))
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
	    raise TraitError(
	traits.trait_errors.TraitError: The 'itk_transform' trait of a C3dAffineToolOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/lscratch/4382593/wrk/fmriprep_23_1_wf/single_subject_02_wf/func_preproc_task_rest_run_01_wf/bold_hmc_wf/fsl2itk/tmp-j0rzc3hv/MAT_0002_itk-00002.txt'  was specified.

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 397, in run
	    runtime = self._run_interface(runtime)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/niworkflows/interfaces/itk.py", line 78, in _run_interface
	    itk_outs = list(
	  File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
	    yield _result_or_cancel(fs.pop())
	  File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
	    return fut.result(timeout)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 458, in result
	    return self.__get_result()
	  File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
	    raise self._exception
	  File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/thread.py", line 58, in run
	    result = self.fn(*self.args, **self.kwargs)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/niworkflows/interfaces/itk.py", line 225, in _mat2itk
	    ).run()
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 400, in run
	    outputs = self.aggregate_outputs(runtime)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
	    raise FileNotFoundError(msg)
	FileNotFoundError: No such file or directory '/lscratch/4382593/wrk/fmriprep_23_1_wf/single_subject_02_wf/func_preproc_task_rest_run_01_wf/bold_hmc_wf/fsl2itk/tmp-j0rzc3hv/MAT_0002_itk-00002.txt' for output 'itk_transform' of a C3dAffineTool interface

Node Name: fmriprep_23_1_wf.single_subject_02_wf.func_preproc_task_rest_run_02_wf.bold_hmc_wf.fsl2itk

Output of nipypecli show /data/MLDSST/nielsond/target_test/data/derivatives/ds002330/fmriprep/fmriprep_v23.1.3/wrk/wrk/fmriprep_23_1_wf/single_subject_02_wf/func_preproc_task_rest_run_01_wf/bold_hmc_wf/fsl2itk/_node.pklz

(/data/MLDSST/nielsond/target_test/env) [nielsond@biowulf fsl2itk]$ nipypecli show _node.pklz 
230713-10:01:30,595 nipype.utils WARNING:
         No metadata was found in the pkl file. Make sure you are currently using the same Nipype version from the generated pkl.
Traceback (most recent call last):
  File "/data/MLDSST/nielsond/target_test/env/bin/nipypecli", line 8, in <module>
    sys.exit(cli())
  File "/gpfs/gsfs10/users/MLDSST/nielsond/target_test/env/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/gpfs/gsfs10/users/MLDSST/nielsond/target_test/env/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/gpfs/gsfs10/users/MLDSST/nielsond/target_test/env/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/gpfs/gsfs10/users/MLDSST/nielsond/target_test/env/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/gpfs/gsfs10/users/MLDSST/nielsond/target_test/env/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/gpfs/gsfs10/users/MLDSST/nielsond/target_test/env/lib/python3.9/site-packages/nipype/scripts/cli.py", line 110, in show
    pkl_data = loadpkl(pklz_file)
  File "/gpfs/gsfs10/users/MLDSST/nielsond/target_test/env/lib/python3.9/site-packages/nipype/utils/filemanip.py", line 666, in loadpkl
    raise e
  File "/gpfs/gsfs10/users/MLDSST/nielsond/target_test/env/lib/python3.9/site-packages/nipype/utils/filemanip.py", line 643, in loadpkl
    unpkl = pickle.loads(pkl_contents)
  File "/gpfs/gsfs10/users/MLDSST/nielsond/target_test/env/lib/python3.9/site-packages/traits/has_traits.py", line 1410, in __setstate__
    self.trait_set(trait_change_notify=trait_change_notify, **state)
  File "/gpfs/gsfs10/users/MLDSST/nielsond/target_test/env/lib/python3.9/site-packages/traits/has_traits.py", line 1520, in trait_set
    setattr(self, name, value)
  File "/gpfs/gsfs10/users/MLDSST/nielsond/target_test/env/lib/python3.9/site-packages/nipype/interfaces/base/traits_extension.py", line 425, in validate
    value = super(MultiObject, self).validate(objekt, name, newvalue)
  File "/gpfs/gsfs10/users/MLDSST/nielsond/target_test/env/lib/python3.9/site-packages/traits/trait_types.py", line 2699, in validate
    return TraitListObject(self, object, name, value)
  File "/gpfs/gsfs10/users/MLDSST/nielsond/target_test/env/lib/python3.9/site-packages/traits/trait_list_object.py", line 582, in __init__
    super().__init__(
  File "/gpfs/gsfs10/users/MLDSST/nielsond/target_test/env/lib/python3.9/site-packages/traits/trait_list_object.py", line 213, in __init__
    super().__init__(self.item_validator(item) for item in iterable)
  File "/gpfs/gsfs10/users/MLDSST/nielsond/target_test/env/lib/python3.9/site-packages/traits/trait_list_object.py", line 213, in <genexpr>
    super().__init__(self.item_validator(item) for item in iterable)
  File "/gpfs/gsfs10/users/MLDSST/nielsond/target_test/env/lib/python3.9/site-packages/traits/trait_list_object.py", line 865, in _item_validator
    return trait_validator(object, self.name, value)
  File "/gpfs/gsfs10/users/MLDSST/nielsond/target_test/env/lib/python3.9/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
  File "/gpfs/gsfs10/users/MLDSST/nielsond/target_test/env/lib/python3.9/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
    self.error(objekt, name, str(value))
  File "/gpfs/gsfs10/users/MLDSST/nielsond/target_test/env/lib/python3.9/site-packages/traits/base_trait_handler.py", line 74, in error
    raise TraitError(
traits.trait_errors.TraitError: Each element of the 'in_files' trait of a _MCFLIRT2ITKInputSpec instance must be a pathlike object or string representing an existing file, but a value of '/lscratch/4382593/wrk/fmriprep_23_1_wf/single_subject_02_wf/func_preproc_task_rest_run_01_wf/bold_hmc_wf/mcflirt/sub-02_task-rest_run-01_bold_mcf.nii.gz.mat/MAT_0000' <class 'str'> was specified.

(/data/MLDSST/nielsond/target_test/env) [nielsond@biowulf fsl2itk]$ ls -ltah ../mcflirt/sub-02_task-rest_run-01_bold_mcf.nii.gz.mat/MAT_0000
-rw-rw----. 1 nielsond MBDU 152 Jul 12 12:43 ../mcflirt/sub-02_task-rest_run-01_bold_mcf.nii.gz.mat/MAT_0000
(/data/MLDSST/nielsond/target_test/env) [nielsond@biowulf fsl2itk]$ cat ../mcflirt/sub-02_task-rest_run-01_bold_mcf.nii.gz.mat/MAT_0000
1.000000 0.000368 0.000000 -0.017003 
-0.000368 1.000000 0.000370 -0.049489 
0.000000 -0.000370 1.000000 0.029035 
0.000000 0.000000 0.000000 1.000000 
(/data/MLDSST/nielsond/target_test/e

Additional information / screenshots

No response