scipy: BUG: sparse.splu: Memory Error
Describe your issue.
Hi. I have been using scipy.sparse.linalg.splu for sparse LU factorization. I kept running into memory issues with problems that were nowhere near the RAM capacity of the system. After more experimenting, I am quite sure that the error has nothing to do with memory at all. I was able to reproduce the error on 2 different systems, one 128GPs of RAM and the other one 32 GB of RAM. The reproducing code example reliable leads to the error on the 32GP system, going nowhere near the available RAM. Thanks for any advice in advance.
Reproducing Code Example
from numpy.random import default_rng
import numpy as np
from scipy.sparse.linalg import splu
import scipy.sparse as sparse
import time
from scipy.sparse.linalg import use_solver
def add_to_diag(res, vector):
diag = sparse.eye(len(vector), format="coo") ##make variance
diag.setdiag(vector) ##make variance
sparse_covariance = res + diag ##add variance
return sparse_covariance
def maked(n,m):
rng1 = default_rng()
rows = rng1.choice(n, size=m)
rng2 = default_rng()
cols = rng2.choice(n, size=m)
data = np.random.rand(m)
return rows,cols,data
use_solver(useUmfpack=False)
N = 10000
n = 2000000
matrix = sparse.coo_matrix((N,N))
r,c,d = maked(N,n)
res = sparse.coo_matrix((d,(r,c)), dtype = np.float64)
res = res @ res.T
vector = np.random.rand(N) * 1.0
res = add_to_diag(res,vector)
print("Number of Non-Zero Entries: ",res.count_nonzero())
st = time.time()
#########LU###########
A = res.tocsc()
print("Transferred after: ", time.time() - st)
LU = splu(A)
print(LU)
print("concluded after: ", time.time() - st)
Error message
---------------------------------------------------------------------------
MemoryError Traceback (most recent call last)
Input In [2], in <module>
43 A = res.tocsc()
44 print("Transferred after: ", time.time() - st)
---> 45 LU = splu(A)
47 print(LU)
48 print("concluded after: ", time.time() - st)
File ~/VirtualEnvironments/fvgp_env/lib/python3.8/site-packages/scipy/sparse/linalg/dsolve/linsolve.py:337, in splu(A, permc_spec, diag_pivot_thresh, relax, panel_size, options)
334 if (_options["ColPerm"] == "NATURAL"):
335 _options["SymmetricMode"] = True
--> 337 return _superlu.gstrf(N, A.nnz, A.data, A.indices, A.indptr,
338 csc_construct_func=csc_construct_func,
339 ilu=False, options=_options)
MemoryError:
SciPy/NumPy/Python version information
1.7.3 1.22.1 sys.version_info(major=3, minor=8, micro=10, releaselevel=‘final’, serial=0)
About this issue
- Original URL
- State: open
- Created 5 years ago
- Comments: 47 (14 by maintainers)
Hi @GregStroot. This is a SuperLU “issue”, meaning that it is partly expected behavior because of the structure of the matrices. I am working with the developer of SuperLU on trying different sorting mechanisms. I’ll post any results here.
Thanks for all the work you guys are doing this, But sorry, I was not so lucky showing that it works:
Can’t expand MemType 1: jcol 653457 Traceback (most recent call last): File “C:\SINAS\coding\pysinas\src\sinas\patsolver.py”, line 135, in applySolver Qdec = splu(Q) File “C:\miniconda\envs\pysinas313\Lib\site-packages\scipy\sparse\linalg_dsolve\linsolve.py”, line 428, in splu return _superlu.gstrf(N, A.nnz, A.data, indices, indptr, ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ csc_construct_func=csc_construct_func, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ilu=False, options=_options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ MemoryError
Like before this is not an issue of running out of physical memory. When this happen the task manager was stating that I still had 60+ GB RAM available.
I am using anaconda/miniconda. The list of modules/packages installed is shown below. As can be seen I am using the latest scipy version (1.15.2). However I am not using the latest numpy version. But the sparse functionality is not part of numpy, as far as I know.
packages in environment at C:\miniconda\envs\pysinas313:
Name Version Build Channel
altgraph 0.17.3 py313haa95532_0
attrs 24.3.0 py313haa95532_0
blas 1.0 mkl
blosc 1.21.3 h6c2663c_0
bzip2 1.0.8 h2bbff1b_6
c-blosc2 2.12.0 h2f4ed9d_0
ca-certificates 2025.2.25 haa95532_0
colorama 0.4.6 py313haa95532_0
expat 2.6.4 h8ddb27b_0
hdf5 1.14.5 ha36df97_2
icc_rt 2022.1.0 h6049295_2
intel-openmp 2023.1.0 h59b6b97_46320
jsonschema 4.23.0 py313haa95532_0
jsonschema-specifications 2023.7.1 py313haa95532_0
libffi 3.4.4 hd77b12b_1
libmpdec 4.0.0 h827c3e9_0
llvmlite 0.44.0 py313h8b1c7eb_1
lz4-c 1.9.4 h2bbff1b_1
lzo 2.10 he774522_2
mkl 2023.1.0 h6b88ed4_46358
mkl-service 2.4.0 py313h827c3e9_2
mkl_fft 1.3.11 py313h827c3e9_0
mkl_random 1.2.8 py313hce38976_0
numba 0.61.0 py313h5da7b33_1
numexpr 2.10.1 py313h4cd664f_0
numpy 2.1.3 py313he6dc315_0
numpy-base 2.1.3 py313h6011491_0
openssl 3.0.16 h3f729d1_0
packaging 24.2 py313haa95532_0
pefile 2023.2.7 py313haa95532_0
pip 25.0 py313haa95532_0
py-cpuinfo 9.0.0 py313haa95532_0
pyinstaller 6.12.0 py313h0416ee5_0
pyinstaller-hooks-contrib 2025.1 py313haa95532_0
pytables 3.10.2 py313he42c613_1
python 3.13.2 hadb2040_100_cp313
python_abi 3.13 0_cp313
pywin32-ctypes 0.2.2 py313haa95532_0
referencing 0.30.2 py313haa95532_0
rpds-py 0.22.3 py313h636fa0f_0
scipy 1.15.2 py313hd4070db_1
setuptools 72.1.0 py313haa95532_0
sqlite 3.45.3 h2bbff1b_0
tbb 2021.8.0 h59b6b97_0
tk 8.6.14 h0416ee5_0
typing-extensions 4.12.2 py313haa95532_0
typing_extensions 4.12.2 py313haa95532_0
tzdata 2025a h04d1e81_0
vc 14.42 haa95532_4
vs2015_runtime 14.42.34433 he0abc0d_4
wheel 0.45.1 py313haa95532_0
xz 5.6.4 h4754444_1
zlib 1.2.13 h8cc25b3_1
zlib-ng 2.0.7 h2bbff1b_0
zstd 1.5.6 h8880b57_0
Any clue?
Maybe even some of the scipy patch can be upstreamed to SuperLU?
No. Someone will have to do the work to replace the version that is vendored in SciPy (the README file says 5.2.x) with the latest release. We currently have a patch file (
scipychanges.patch) containing changes that SciPy made to the vendored copy, so the update is not a trivial task.I think we want to update, but someone will have to do the work to make it happen, and I don’t know if any of the current devs has this on their “to do” list at the moment. Anyone who takes on this task should have some familiarity with C, and be willing to dig into the history to understand the changes in the patch file.
The SuperLU developer fixed the issue. This can now be included in scipy. Please let me know if this will be done and, if so, when it’s done. Thank you all for figuring this out. The 6.0.0 release is here: https://github.com/xiaoyeli/superlu/releases
@nicolasdonati The solution I found was to switch solvers (e.g. not use scipy). I am now using petsc4py, which allows you to access the MUMPS LU solver. I have successfully ran LU decompositions on matricies (N x N) of size N = 9e6 with complex double data-types, which took ~600GB of memory
Here is some code that will allow you to use it
Feel free to send a message if you get stuck. The documentation is a bit unwieldy