rsatoolbox: Runtime warnings and errors when using searchlight

Hello,

I’m currently getting runtime warnings and errors when using searchlight (specifically from evaluate_models_searchlight).

Background:

• OS-X 10.14.3, Python 3.7.3, rsatoolbox 0.0.4, numpy 1.21.6, scipy 1.7.3, pandas 1.3.5, nibabel 4.0.1

• I’ve been successful in getting basic RSA toolbox functionality to work (i.e., defining a data RDM and a model RDM and fitting the model to the data using rsatoolbox.inference.eval_fixed). I don’t get any warnings or errors and the results look sensible. It’s only when attempting to use searchlight that I run into problems.

• As far as I have explored, the problem isn’t fixed by options, parameter settings, data size, etc.

1) runtime warning:

I get the following warning when i) running the searchlight example in the documentation or ii) running searchlight on toy data (generated using numpy random):

$ ./searchlight_demo.py
Finding searchlights...: 100%|███████████| 61044/61044 [00:14<00:00, 4308.06it/s]
Found 56954 searchlights
Calculating RDMs...: 100%|████████████| 100/100 [05:46<00:00,  3.47s/it]
Evaluating models for each searchlight: 
/usr/local/lib/python3.7/site-packages/rsatoolbox/inference/evaluate.py:255: RuntimeWarning: Degrees of freedom <= 0 for slice
  variances = np.cov(evaluations[0], ddof=1) \
/usr/local/lib/python3.7/site-packages/numpy/lib/function_base.py:2542: RuntimeWarning: divide by zero encountered in true_divide
  c *= np.true_divide(1, fact)
/usr/local/lib/python3.7/site-packages/numpy/lib/function_base.py:2542: RuntimeWarning: invalid value encountered in multiply
  c *= np.true_divide(1, fact)

[ warning repeats two additional times ]

Evaluating models for each searchlight: 100%|██████████| 56954/56954 [01:50<00:00, 517.41it/s]
$

The analysis completes, but the divide by zero warning is still concerning.

2) runtime error

When trying to run searchlight on real (i.e., our) data I get a different warning and then an error. What’s different here is this uses data saved from Matlab and then input using io.matlab.loadmat (code to follow):

$ ./searchlight_realdatatest.py
Finding searchlights...: 100%|███████████████████| 254906/254906 [00:22<00:00, 11444.65it/s]
Found 218370 searchlights
Calculating RDMs...:   0%|                                                                                                                                                                                     | /usr/local/lib/python3.7/site-packages/rsatoolbox/rdm/calc.py:209: RuntimeWarning: invalid value encountered in true_divide
  ma /= np.sqrt(np.einsum('ij,ij->i', ma, ma))[:, None]
Calculating RDMs...: 100%|████████████████████| 100/100 [03:38<00:00,  2.19s/it]
Evaluating models for each searchlight:   0%|                                                                            
joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 436, in _process_worker
    r = call_item()
  File "/usr/local/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 288, in __call__
    return self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 595, in __call__
    return self.func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/joblib/parallel.py", line 263, in __call__
    for func, args, kwargs in self.items]
  File "/usr/local/lib/python3.7/site-packages/joblib/parallel.py", line 263, in <listcomp>
    for func, args, kwargs in self.items]
  File "/usr/local/lib/python3.7/site-packages/rsatoolbox/inference/evaluate.py", line 251, in eval_fixed
    evaluations[k] = compare(rdm_pred, data, method)
  File "/usr/local/lib/python3.7/site-packages/rsatoolbox/rdm/compare.py", line 62, in compare
    sim = compare_spearman(rdm1, rdm2)
  File "/usr/local/lib/python3.7/site-packages/rsatoolbox/rdm/compare.py", line 176, in compare_spearman
    vector1, vector2, _ = _parse_input_rdms(rdm1, rdm2)
  File "/usr/local/lib/python3.7/site-packages/rsatoolbox/rdm/compare.py", line 615, in _parse_input_rdms
    raise ValueError('rdm1 and rdm2 have different nan positions')
ValueError: rdm1 and rdm2 have different nan positions
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./aamod_searchlight_hack.py", line 138, in <module>
    eval_results = evaluate_models_searchlight(SL_RDM, model, eval_fixed, method='spearman', n_jobs=njobs)
  File "/usr/local/lib/python3.7/site-packages/rsatoolbox/util/searchlight.py", line 207, in evaluate_models_searchlight
    sl_RDM, desc='Evaluating models for each searchlight'))
  File "/usr/local/lib/python3.7/site-packages/joblib/parallel.py", line 1056, in __call__
    self.retrieve()
  File "/usr/local/lib/python3.7/site-packages/joblib/parallel.py", line 935, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/usr/local/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 542, in wrap_future_result
    return future.result(timeout=timeout)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
ValueError: rdm1 and rdm2 have different nan positions
$

Here’s my code (which I mostly stole from the docs)

#!/usr/bin/env python3

import numpy
import rsatoolbox
from rsatoolbox.inference import eval_fixed

from rsatoolbox.util.searchlight import get_volume_searchlight, get_searchlight_RDMs, evaluate_models_searchlight

from scipy import io


# ----- load data ---------------------------------------------------

# 2D array (nconditions x nvoxels) saved from matlab

dataFile = '/path/to/data.mat'

temp = io.matlab.loadmat(dataFile)
varkey = list(temp)[-1]
data = temp[varkey]

# ---- load model ---------------------------------------------

# 2D array (nconditions x nfeatures) saved from matlab

modelFile = '/path/to/model.mat'

temp = io.matlab.loadmat(modelFile)
varkey = list(temp)[-1]
model_def = temp[varkey]

model_features = [rsatoolbox.data.Dataset(model_def)]
model_RDM = rsatoolbox.rdm.calc_rdm(model_features)
model = rsatoolbox.model.ModelFixed('fixed_model',model_RDM)

# ---- load mask -------------------------------------------

# 0/1 mask saved as 3D array from matlab
# [nx ny nz] where nx*ny*nz = nvoxels
# converted to boolean (although 0/1 seems to work)

maskFile = '/path/to/mask.mat'

temp = io.matlab.loadmat(maskFile)
varkey = list(temp)[-1]
imask = temp[varkey]
mask = imask > 0

# ---- searchlight -----------------------------------------

image_value = numpy.arange(data.shape[0])

# can convert nan here or convert nan in matlab before the
# file is saved (or both) - you get the same result

data = numpy.nan_to_num(data)

centers, neighbors = get_volume_searchlight(mask)
SL_RDM = get_searchlight_RDMs(data, centers, neighbors, image_value, method='correlation')
eval_results = evaluate_models_searchlight(SL_RDM, model, eval_fixed, method='spearman', n_jobs=3)

Any help is much appreciated, Mike