deeplearning4j: java.lang.OutOfMemoryError: Cannot allocate new DoublePointer(10000000): totalBytes = 4G, physicalBytes = 7G
My code runs vector embedding refresh every night based on linux crontab. Code does following
- Builds the Dl4J embedding files for records of size 2500 using GoogleNews Word2Vec model.
- Loads embeddings into JVM cache in a HashMap for a faster access by API servers
I get following exception in my code while performing second step above, where I try to read Dl4J Embeddings File using WordVectorSerializer.readWord2VecModel()
This does not happen all the time. After start the server and run it for 4-5 days and cron job invokes above process, in 5th or 6th day I see following OutOfMemory . My current jvm option is set to -Xmx5G. In 4-5 days it will kill my server . Question is what is the real solution to this problem , instead of keep increasing -Xmx ?
My code where it triggers the error after 4-5 days of invoking this piece of code once a day
File embeddings_file = "/Users/legalizenet/embeddings/dl4j_embeddings.txt";
Word2Vec entity_embeddings_w2v = WordVectorSerializer.readWord2VecModel(embeddings_file.getAbsolutePath());
Exception:
java.lang.OutOfMemoryError: Cannot allocate new DoublePointer(10000000): totalBytes = 4G, physicalBytes = 7G at org.bytedeco.javacpp.DoublePointer.<init>(DoublePointer.java:76) at org.nd4j.linalg.api.buffer.BaseDataBuffer.<init>(BaseDataBuffer.java:584) at org.nd4j.linalg.api.buffer.BaseDataBuffer.<init>(BaseDataBuffer.java:570) at org.nd4j.linalg.api.buffer.DoubleBuffer.<init>(DoubleBuffer.java:51) at org.nd4j.linalg.api.buffer.factory.DefaultDataBufferFactory.createDouble(DefaultDataBufferFactory.java:237) at org.nd4j.rng.NativeRandom.<init>(NativeRandom.java:77) at org.nd4j.rng.NativeRandom.<init>(NativeRandom.java:66) at org.nd4j.rng.NativeRandom.<init>(NativeRandom.java:62) at org.nd4j.linalg.cpu.nativecpu.rng.CpuNativeRandom.<init>(CpuNativeRandom.java:14) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at org.nd4j.linalg.factory.RandomFactory.getRandom(RandomFactory.java:28) at org.nd4j.linalg.factory.Nd4j.getRandom(Nd4j.java:542) at org.deeplearning4j.models.embeddings.inmemory.InMemoryLookupTable.<init>(InMemoryLookupTable.java:62) at org.deeplearning4j.models.embeddings.loader.WordVectorSerializer.readWord2VecModel(WordVectorSerializer.java:2208) at org.deeplearning4j.models.embeddings.loader.WordVectorSerializer.readWord2VecModel(WordVectorSerializer.java:2160) at org.deeplearning4j.models.embeddings.loader.WordVectorSerializer.readWord2VecModel(WordVectorSerializer.java:2176) … at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.OutOfMemoryError: Native allocator returned address == 0 at org.bytedeco.javacpp.DoublePointer.<init>(DoublePointer.java:70)
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 25 (15 by maintainers)
We have discovered some interesting facts & solution to this problem, about this in terms of hardware/memory/configuration settings combinations around java on heap vs off heap settings. @rizwan-talentsky (as he is the one who did the hard work finding different options out) will share that with you guys at some point, so that it will be useful for other developers and they do not spend/waste so much time dealing with this.
cc @legalizenet
I tried the suggested config changes on my mac:
The application runs fine but ultimately fails with OOM (same exception as mentioned earlier in the thread) even after doing the above suggested config changes. Also, noticed that increasing the javacpp maxbytes only delayed the OOM. This led me to suspect that there is possibly a memory allocation issue somewhere in the cpp layer.
The OOM issues goes away if I bump up the version dependency for org.bytedeco.javacpp-presets:openblas from the current version “0.2.20-1.4.1” to “0.3.0-1.4.2”.
Do you see any issue with the above workaround ?
I am using the following jvm options : -Xmx2G -Dorg.bytedeco.javacpp.maxbytes=5G -Dorg.bytedeco.javacpp.maxphysicalbytes=5G
This looks like a acceptable solution for us until a higher version of org.bytedeco.javacpp-presets:openblas is officially supported.
So whats the final verdict on this, How do I fix this issue
Anything else I missed ?