scikit-learn: Python crashes when calculating large t-SNE
Python 2.7.9, scikit-learn 0.16.1 MacBook Pro, 16GB RAM, MacOS 10.10.3
When running the following code in an iPython notebook, it runs for a long time producing no output, and then the iPython kernel crashes and has to restart.
import numpy as np
from sklearn.manifold import TSNE
X = np.random.random((100000, 100))
tsne = TSNE(n_components=2, perplexity=40, learning_rate=100, verbose=2).fit_transform(X)
Running the same code from the command line produces one line of output
[t-SNE] Computing pairwise distances...
and then it dies.
The error is pretty much the same as reported in this older issue. I’ll attach a crash log next, but I think it’s out of memory because if I reduce the rows of X to 10000, then it runs to completion without error.
Some suggested improvements:
- Estimate the memory required to carry out this operation and warn or error out earlier.
- Try to recover from OOM errors in libBLAS more gracefully.
- If possible, reduce the memory requirements of sklearn.manifold.TSNE or implement out-of-core
- Document the memory requirements in sklearn.manifold (and sklearn.decomposition), and perhaps warn about potential OOM errors and what they’ll look like to users. You’ve already nicely documented the runtime complexities of the algorithms but have not documented their space complexities.
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Reactions: 3
- Comments: 26 (9 by maintainers)
@michaelbrundage Thanks for your prompt answer! Just to let you know, you can use the python wrapper of the C++ implementation of the Bhtsne available on Laurent’s GitHub. It tested it with (300k, 50) and it works good! Maybe not as tunable as the one in Sklearn but it does the job pretty well.
@rjurney It is an integer overflow resulting in an attempt to write to an OS reserved address. The amount of physical RAM does not matter. The error can be produced on any 64-bit system. The root of the problem is the use of a 32-bit signed integer as indexer in the signature of the auxillary BLAS function
catlas_dset. Whoever wrote the function did not know (or care) about the difference between a Cintand asize_t.I have a machine with 1TB of RAM and am seeing this error. I do not think it is the RAM running out. I have tried installing
nomklwith no effect 😦C++ implementation with python wrapper https://github.com/lvdmaaten/bhtsne It was able to process data on which sklearn.manifold.TSNE was failing with Memory Error
I have this problem too, as the person above pointed out, the other implementation works fine. This is unfortunate. With 16gb ram this implementation fails with just (100000,20), how can this even be considered useful?
Edit: with (35000,20) it fails on “[t-SNE] Computing 91 nearest neighbors…”. Just what exactly is this implementation useful for? Iris data?
It’s an out-of-memory error, so the only workaround is to reduce your dataset size.