qsiprep: synthseg crash

I’m running qsiprep 0.18.0 on a standard (bids-validated) dataset like this:

tree bids/sub-001
bids/sub-001
|-- anat
|   |-- sub-001_acq-MPRAGE_rec-biascorr_run-1_T1w.json
|   |-- sub-001_acq-MPRAGE_rec-biascorr_run-1_T1w.nii.gz
|   |-- sub-001_acq-TSE_rec-biascorr_run-1_T2w.json
|   |-- sub-001_acq-TSE_rec-biascorr_run-1_T2w.nii.gz
|   |-- sub-001_run-1_UNIT1.json
|   |-- sub-001_run-1_UNIT1.nii.gz
|   |-- sub-001_run-1_echo-1_part-mag_MEGRE.json
|   |-- sub-001_run-1_echo-1_part-mag_MEGRE.nii.gz
|   |-- sub-001_run-1_echo-1_part-phase_MEGRE.json
|   |-- sub-001_run-1_echo-1_part-phase_MEGRE.nii.gz
|   |-- [..]
|   |-- sub-001_run-1_echo-9_part-mag_MEGRE.json
|   |-- sub-001_run-1_echo-9_part-mag_MEGRE.nii.gz
|   |-- sub-001_run-1_echo-9_part-phase_MEGRE.json
|   |-- sub-001_run-1_echo-9_part-phase_MEGRE.nii.gz
|   |-- sub-001_run-1_inv-1_MP2RAGE.json
|   |-- sub-001_run-1_inv-1_MP2RAGE.nii.gz
|   |-- sub-001_run-1_inv-2_MP2RAGE.json
|   `-- sub-001_run-1_inv-2_MP2RAGE.nii.gz
|-- dwi
|   |-- sub-001_acq-HARDI_dir-AP_run-1_dwi.bval
|   |-- sub-001_acq-HARDI_dir-AP_run-1_dwi.bvec
|   |-- sub-001_acq-HARDI_dir-AP_run-1_dwi.json
|   |-- sub-001_acq-HARDI_dir-AP_run-1_dwi.nii.gz
|   |-- sub-001_acq-HARDI_dir-AP_run-1_sbref.json
|   `-- sub-001_acq-HARDI_dir-AP_run-1_sbref.nii.gz
`-- sub-001_scans.tsv

qsub -l nodes=1:ppn=2,walltime=72:00:00,mem=20000mb,file=50gb -N qsiprep_sub-001  <<EOF
cd /project/3015046.06
apptainer run --cleanenv  --bind \$TMPDIR:/tmp /opt/qsiprep/0.18.0/qsiprep-0.18.0.sif bids bids/derivatives participant --output-resolution 1.8 -w \$TMPDIR/sub-001 --participant-label 001 --skip-bids-validation --fs-license-file /opt_host/fmriprep/license.txt --mem_mb 20000 --omp-nthreads 2 --nthreads 2 
EOF

It runs for several hours but crashes on mri_synthseg:

QSIPrep failed: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node synthseg.

Cmdline:
        mri_synthseg --i /scratch/marzwi/49347907.dccn-l029.dccn.nl/sub-260/qsiprep_wf/single_subject_260_wf/anat_preproc_wf/pad_anat_reference_wf/resample_skulled_to_reference/sub-260_acq-MPRAGE_rec-biascorr_run-1_T1w_lps_trans.nii.gz --threads 2 --post sub-260_acq-MPRAGE_rec-biascorr_run-1_T1w_lps_trans_post.nii.gz --qc sub-260_acq-MPRAGE_rec-biascorr_run-1_T1w_lps_trans_qc.csv --o sub-260_acq-MPRAGE_rec-biascorr_run-1_T1w_lps_trans_aseg.nii.gz
Stdout:

Stderr:
        Aborted (core dumped)
Traceback:
        Traceback (most recent call last):
          File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
            setattr(outputs, key, val)
          File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
            value = super(File, self).validate(objekt, name, value, return_pathlike=True)
          File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
            self.error(objekt, name, str(value))
          File "/usr/local/miniconda/lib/python3.8/site-packages/traits/base_trait_handler.py", line 74, in error
            raise TraitError(
        traits.trait_errors.TraitError: The 'out_seg' trait of a _SynthSegOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/scratch/marzwi/49347907.dccn-l029.dccn.nl/sub-260/qsiprep_wf/single_subject_260_wf/anat_preproc_wf/synthseg_anat_wf/synthseg/sub-260_acq-MPRAGE_rec-biascorr_run-1_T1w_lps_trans_aseg.nii.gz' <class 'str'> was specified.

        During handling of the above exception, another exception occurred:
[..]

See the attached files for more details. I’m stuck here, does anybody have any suggestions on what to do now?

qsiprep_sub-001.e49347653.log qsiprep_sub-001.o49347653.log crash-20230627-123725-marzwi-synthseg-859f4773-2ab1-45b4-8376-7d61b7d426c3.txt

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 23

Most upvoted comments

@mattcieslak sorry, typo. I used containall. On buster I am getting always the bmat error with singularity 2.6.1-dist. With --clearall, --containall, wtih export SINGULARITY_PYTHONNOUSERSITE=1 With docker the traits error.

Wtih bullseye (different server, different ~ dir) the traits error,

The servers are older ones with 80 cores Intel® Xeon® CPU E7- 4870 @ 2.40GHz, 512 GB mem.

I’ve been using --containall recently, it will also keep the environment variables from being included. Lots of problems recently are from python libraries in home directories