smriprep: Autorecon1 Crashing

What happened?

Autorecon 1 appears to be failing and not providing a helpful error message.

What command did you use?

singularity run -e -B ${scratch} $IMG ${scratch}/${subject}_db ${scratch}/${subject}_db/derivatives  \ 
participant --participant_label ${subject:4} -w $scratch/${subject}_work \ 
--fs-license-file  ${scratch}/${subject}_db/code/license.txt --output-layout legacy \
--skip-bids-validation --stop-on-first-crash -t $task  \
--fd-spike-threshold .9 --dvars-spike-threshold 3 --mem 31500  \
--output-spaces T1w MNI152NLin6Asym:res-2

What version of fMRIPrep are you running?

22.0.0rc0

How are you running fMRIPrep?

Singularity

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

[Node] Executing "autorecon1" <smriprep.interfaces.freesurfer.ReconAll>
220528-11:48:40,992 nipype.workflow INFO:
	 [Node] Finished "autorecon1", elapsed time 0.77419s.
220528-11:48:41,7 nipype.workflow WARNING:
	 [Node] Error on "fmriprep_22_0_wf.single_subject_NDARAA306NT2_wf.anat_preproc_wf.surface_recon_wf.autorecon1" (/om2/scratch/Sat/smeisler/HBN_DM_SCRATCH/sub-NDARAA306NT2_work/fmriprep_22_0_wf/single_subject_NDARAA306NT2_wf/anat_preproc_wf/surface_recon_wf/autorecon1)
220528-11:48:42,834 nipype.workflow ERROR:
	 Node autorecon1 failed to run on host node039.
220528-11:48:42,845 nipype.workflow ERROR:
	 Saving crash info to /om2/scratch/Sat/smeisler/HBN_DM_SCRATCH/sub-NDARAA306NT2_db/derivatives/fmriprep/sub-NDARAA306NT2/log/20220528-114715_003f85be-839f-44cd-8d1f-fe81ffec540d/crash-20220528-114842-smeisler-autorecon1-e0ad7322-dd4c-4a03-9b22-70f6614a2539.txt
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node autorecon1.

RuntimeError: subprocess exited with code 1.

220528-11:48:42,895 nipype.workflow CRITICAL:
	 fMRIPrep failed: Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node autorecon1.

RuntimeError: subprocess exited with code 1.

220528-11:48:43,994 cli ERROR:
	 Preprocessing did not finish successfully. Errors occurred while processing data from participants: NDARAA306NT2 (1). Check the HTML reports for details.
220528-11:48:43,994 cli ERROR:
	 Preprocessing did not finish successfully. Errors occurred while processing data from participants: NDARAA306NT2 (1). Check the HTML reports for details.

Additional information / screenshots

Each subject is its own job, 32GB 8CPU

Not all subjects have this error, but all subjects came from the same dataset. Looking now I cannot think of anything that discerns the crashing vs passing subjects.

Centos7.6

From the crash file mentioned in the log output:

Node: fmriprep_22_0_wf.single_subject_NDARAA306NT2_wf.anat_preproc_wf.surface_recon_wf.autorecon1
Working directory: /om2/scratch/Sat/smeisler/HBN_DM_SCRATCH/sub-NDARAA306NT2_work/fmriprep_22_0_wf/single_subject_NDARAA306NT2_wf/anat_preproc_wf/surface_recon_wf/autorecon1

Node inputs:

FLAIR_file = <undefined>
T1_files = <undefined>
T2_file = <undefined>
args = <undefined>
big_ventricles = <undefined>
brainstem = <undefined>
directive = autorecon1
environ = {}
expert = <undefined>
flags = <undefined>
hemi = <undefined>
hippocampal_subfields_T1 = <undefined>
hippocampal_subfields_T2 = <undefined>
hires = <undefined>
mprage = <undefined>
mri_aparc2aseg = <undefined>
mri_ca_label = <undefined>
mri_ca_normalize = <undefined>
mri_ca_register = <undefined>
mri_edit_wm_with_aseg = <undefined>
mri_em_register = <undefined>
mri_fill = <undefined>
mri_mask = <undefined>
mri_normalize = <undefined>
mri_pretess = <undefined>
mri_remove_neck = <undefined>
mri_segment = <undefined>
mri_segstats = <undefined>
mri_tessellate = <undefined>
mri_watershed = <undefined>
mris_anatomical_stats = <undefined>
mris_ca_label = <undefined>
mris_fix_topology = <undefined>
mris_inflate = <undefined>
mris_make_surfaces = <undefined>
mris_register = <undefined>
mris_smooth = <undefined>
mris_sphere = <undefined>
mris_surf2vol = <undefined>
mrisp_paint = <undefined>
openmp = 8
parallel = <undefined>
steps = <undefined>
subject_id = recon_all
subjects_dir = <undefined>
talairach = <undefined>
use_FLAIR = <undefined>
use_T2 = <undefined>
xopts = <undefined>

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node autorecon1.

RuntimeError: subprocess exited with code 1.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 48 (9 by maintainers)

Most upvoted comments

Ok, looking at the log a little closer, autorecon1 instantly fails:

[Node] Executing "autorecon1" <smriprep.interfaces.freesurfer.ReconAll>
220528-11:48:40,992 nipype.workflow INFO:
	 [Node] Finished "autorecon1", elapsed time 0.77419s.

Perhaps the best way to debug this would be to:

  1. Shell into the container (either with the --shell option on the docker wrapper, or singularity shell), being sure to include all the original mounts
  2. Running the command located in the file (hopefully it exists)
<workdir>/fmriprep_22_0_wf/single_subject_<subject>_wf/anat_preproc_wf/surface_recon_wf/autorecon1/command.txt
+2
mgxd on Jun 3, 2022

Just following up to this, I updated to 22.0.2 and ran again. This time received an informative error message relating to failure to create symlinks in the output directory. Changed output directory to a local drive instead of a network drive (which didn’t support symlinks) and fmriprep ran successfully.

+1
BrynGibson on Oct 4, 2022

Thank you @smeisler for tracking that down!

+1
mgxd on Jun 27, 2022

This may be because, in a similar fashion, getfullpath is not in the docker container, but may be called by FreeSurfer.

+1
smeisler on Jun 26, 2022

@mafortin I’ve just tagged sMRIPrep 0.9.1 and fMRIPrep 22.0.0rc1. When those images hit DockerHub, you should be good to go.

+1
effigies on Jun 13, 2022

Hello,

I ran the command in <workdir>/fmriprep_22_0_wf/single_subject_<subject>_wf/anat_preproc_wf/surface_recon_wf/autorecon1/command.txt:

INFO: hi-res volumes are conformed to the min voxel size
fsr-checkxopts: Command not found.
Linux openmind7 3.10.0-1062.el7.x86_64 #1 SMP Wed Aug 7 18:08:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

recon-all -s sub-inside7003 exited with ERRORS at Fri Jun 10 19:53:28 EDT 2022

For more details, see the log file 
To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting

Tab autocomplete in the singularity shell revealed the only command in PATH that begins “fsr” is fsr-getxopts.

+1
smeisler on Jun 10, 2022

–cleanenv is not a recognized argument for smriprep-docker

--cleanenv is a singularity option (not a smriprep/fmriprep argument) which makes sure none of your local environmental variables are brought into the container.

+1
smeisler on Jun 2, 2022

At least on my end (using fMRIPrep), a <FSsubjects-dir>/<subject> directory was not even created. The fsaverage directory is there.

+1
smeisler on Jun 2, 2022
Read more comments on GitHub
← mriqc: pybids recent changes break compatibility: "No scans were found"
smriprep: Wrong probseg fmriprep output from freesurfer pipeline →