fmriprep: Multi echo: tCompCor - numpy.linalg.linalg.LinAlgError: SVD did not converge

Hello all, An older error I saw in a previous solved issue seems to have reappeared when I tried to run fmriprep on a multi echo dataset. Version 1.2.3 previously ran on the same data without any errors, but when I updated to 1.2.4 today, I got an error saying that SVD did not converge. Previous work directories were removed, and outputs were moved elsewhere before running fmriprep 1.2.4. The full output is as follows:

[Node] Error on "fmriprep_wf.single_subject_4154_wf.func_preproc_task_co_run_2_echo_1_wf.bold_confounds_wf.tcompcor" (/data/work/fmriprep_wf/single_subject_4154_wf/func_preproc_task_co_run_2_echo_1_wf/bold_confounds_wf/tcompcor)
181204-15:14:53,737 nipype.workflow ERROR:
	 Node tcompcor failed to run on host pavlov.amc.nl.
181204-15:14:53,743 nipype.workflow ERROR:
	 Saving crash info to /data/fmriprep/sub-4154/log/20181204-143741_072fe7eb-6c6e-4559-aafd-94fe1bdf66a4/crash-20181204-151453-gbrunner-tcompcor-7402206b-78a1-4ed0-b54e-53fe7c787030.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 473, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 557, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 637, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 511, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/patches.py", line 51, in _run_interface
    runtime = super(RobustTCompCor, self)._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/algorithms/confounds.py", line 539, in _run_interface
    self.inputs.pre_filter, degree, self.inputs.high_pass_cutoff, TR)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/algorithms/confounds.py", line 1189, in compute_noise_components
    u, _, _ = linalg.svd(M, full_matrices=False)
  File "/usr/local/miniconda/lib/python3.6/site-packages/scipy/linalg/decomp_svd.py", line 132, in svd
    raise LinAlgError("SVD did not converge")
numpy.linalg.linalg.LinAlgError: SVD did not converge

Preprocessing did not finish successfully. Errors occurred while processing data from participants: 4154 (1). Check the HTML reports for details.
Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 431, in main
    fmriprep_wf.run(**plugin_settings)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 597, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 162, in run
    self._clean_queue(jobid, graph, result=result))
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 225, in _clean_queue
    raise RuntimeError("".join(result['traceback']))
RuntimeError: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 473, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 557, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 637, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 511, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/patches.py", line 51, in _run_interface
    runtime = super(RobustTCompCor, self)._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/algorithms/confounds.py", line 539, in _run_interface
    self.inputs.pre_filter, degree, self.inputs.high_pass_cutoff, TR)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/algorithms/confounds.py", line 1189, in compute_noise_components
    u, _, _ = linalg.svd(M, full_matrices=False)
  File "/usr/local/miniconda/lib/python3.6/site-packages/scipy/linalg/decomp_svd.py", line 132, in svd
    raise LinAlgError("SVD did not converge")
numpy.linalg.linalg.LinAlgError: SVD did not converge

Sentry is attempting to send 3 pending error messages
Waiting up to 2.0 seconds
Press Ctrl-C to quit

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 26 (11 by maintainers)

Most upvoted comments

@gb4944 Thanks for the response. Just to be sure, your derivatives otherwise look sensible? You should be getting reports and derivatives that don’t depend on the confounds, and the reports should show the masks of voxels passed to CompCor. With SVD failures, the main thing we can control is whether we’re giving sensible data to the process; if the data is good and it’s still just failing, all we can try to do is handle that case more gracefully. @shirareznik Same questions for you.

I would propose that we catch LinAlgErrors in CompCor interfaces, and return columns of NaNs, producing a warning that SVD did not converge.

It sounds like we don’t want to disable CompCor altogether with ME, at present, as we don’t have MEICA to replace it, and there’s no guarantee that users aren’t using it. I also found this Power, et al. paper from February that seems to indicate from a cursory reading that there’s value to CompCor in ME datasets.

cc @oesteban @chrisfilo

effigies on Dec 10, 2018