ort: OutOfMemoryError when running the scanner

Hi,

I have a fairly large Maven project (about 500 entries in the dependency tree) that I want to scan with ORT. The analyzer works as expected and produces an analyzer-result.json file of about 1 M.

When I run the scanner, though, It runs for a while and then terminates with the following error message:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

The exact point of failure varies with every run, but generally it happens at a rather advanced stage of the scan (say at 392/486 packages scanned).

A typical stack trace looks like this:

15:26:15.236 [FileBasedStorage with XZCompressedLocalFileStorage backend-4] INFO  kotlinx.coroutines.CoroutineScope - Looking for stored scan results for Maven:org.eclipse.persistence:org.eclipse.persistence.jpa
.modelgen.processor:2.7.3 and ScannerDetails(name=ScanCode, version=3.0.2, configuration=--copyright --license --ignore *.ort.yml --info --strip-root --timeout 300 --ignore HERE_NOTICE --ignore META-INF/DEPENDEN
CIES --json-pp --license-diag) (384/486).
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:944)
        at java.base/java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2266)
        at java.base/java.lang.ProcessImpl.initStreams(ProcessImpl.java:389)
        at java.base/java.lang.ProcessImpl.lambda$new$0(ProcessImpl.java:352)
        at java.base/java.lang.ProcessImpl$$Lambda$144/0x0000000800551040.run(Unknown Source)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:351)
        at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:271)
        at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
        at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
        at org.ossreviewtoolkit.utils.ProcessCapture.<init>(ProcessCapture.kt:86)
        at org.ossreviewtoolkit.utils.ProcessCapture.<init>(ProcessCapture.kt:29)
        at org.ossreviewtoolkit.utils.ProcessCapture.<init>(ProcessCapture.kt:33)
        at org.ossreviewtoolkit.utils.CommandLineTool$DefaultImpls.run(CommandLineTool.kt:100)
        at org.ossreviewtoolkit.scanner.LocalScanner.run(LocalScanner.kt:78)
        at org.ossreviewtoolkit.utils.CommandLineTool$DefaultImpls.getVersion(CommandLineTool.kt:106)
        at org.ossreviewtoolkit.scanner.LocalScanner.getVersion(LocalScanner.kt:134)
        at org.ossreviewtoolkit.scanner.LocalScanner.getDetails(LocalScanner.kt:151)
        at org.ossreviewtoolkit.scanner.LocalScanner.scanPackage(LocalScanner.kt:194)
        at org.ossreviewtoolkit.scanner.LocalScanner$scanPackages$2$invokeSuspend$$inlined$map$lambda$1.invokeSuspend(LocalScanner.kt:166)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
        at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
        at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:84)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
        at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
        at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
        at org.ossreviewtoolkit.scanner.Scanner.scanOrtResult(Scanner.kt:105)
        at org.ossreviewtoolkit.commands.ScannerCommand.run(ScannerCommand.kt:165)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:171)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:180)

But I have seem these stack traces, too:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:61)
        at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:348)
        at java.base/sun.nio.cs.StreamDecoder.<init>(StreamDecoder.java:251)
        at java.base/sun.nio.cs.StreamDecoder.<init>(StreamDecoder.java:231)
        at java.base/sun.nio.cs.StreamDecoder.forInputStreamReader(StreamDecoder.java:78)
        at java.base/java.io.InputStreamReader.<init>(InputStreamReader.java:116)
        at kotlin.io.FilesKt__FileReadWriteKt.readText(FileReadWrite.kt:125)
        at kotlin.io.FilesKt__FileReadWriteKt.readText$default(FileReadWrite.kt:125)
        at org.ossreviewtoolkit.utils.ProcessCapture.getStdout(ProcessCapture.kt:67)
        at org.ossreviewtoolkit.utils.CommandLineTool$DefaultImpls.getVersion(CommandLineTool.kt:109)
        at org.ossreviewtoolkit.scanner.LocalScanner.getVersion(LocalScanner.kt:134)
        at org.ossreviewtoolkit.scanner.LocalScanner.getDetails(LocalScanner.kt:151)
        at org.ossreviewtoolkit.scanner.LocalScanner.scanPackage(LocalScanner.kt:194)
        at org.ossreviewtoolkit.scanner.LocalScanner$scanPackages$2$invokeSuspend$$inlined$map$lambda$1.invokeSuspend(LocalScanner.kt:166)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
        at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
        at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:84)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
        at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
        at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
        at org.ossreviewtoolkit.scanner.Scanner.scanOrtResult(Scanner.kt:105)
        at org.ossreviewtoolkit.commands.ScannerCommand.run(ScannerCommand.kt:164)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:171)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:180)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:16)
        at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:395)
        at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:392)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:410)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:435)
        at org.ossreviewtoolkit.OrtMainKt.main(OrtMain.kt:174)

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space                
        at org.ossreviewtoolkit.utils.CommandLineTool$DefaultImpls.getVersion(CommandLineTool.kt:137)
        at org.ossreviewtoolkit.scanner.LocalScanner.getVersion(LocalScanner.kt:134)
        at org.ossreviewtoolkit.scanner.LocalScanner.getDetails(LocalScanner.kt:151)
        at org.ossreviewtoolkit.scanner.LocalScanner.scanPackage(LocalScanner.kt:194)
        at org.ossreviewtoolkit.scanner.LocalScanner$scanPackages$2$invokeSuspend$$inlined$map$lambda$1.invokeSuspend(LocalScanner.kt:166)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
        at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
        at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:84)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
        at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
        at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
        at org.ossreviewtoolkit.scanner.Scanner.scanOrtResult(Scanner.kt:105)
        at org.ossreviewtoolkit.commands.ScannerCommand.run(ScannerCommand.kt:164)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:171)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:180)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:16)
        at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:395)
        at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:392)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:410)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:435)
        at org.ossreviewtoolkit.OrtMainKt.main(OrtMain.kt:174)

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space                        
        at kotlin.io.TextStreamsKt.copyTo(ReadWrite.kt:122)                 
        at kotlin.io.TextStreamsKt.copyTo$default(ReadWrite.kt:120)                                  
        at kotlin.io.TextStreamsKt.readText(ReadWrite.kt:107)                       
        at kotlin.io.FilesKt__FileReadWriteKt.readText(FileReadWrite.kt:125)        
        at kotlin.io.FilesKt__FileReadWriteKt.readText$default(FileReadWrite.kt:125) 
        at org.ossreviewtoolkit.utils.ProcessCapture.getStdout(ProcessCapture.kt:67)                                                                                                                         
        at org.ossreviewtoolkit.utils.ProcessCapture.<init>(ProcessCapture.kt:133)               
        at org.ossreviewtoolkit.utils.ProcessCapture.<init>(ProcessCapture.kt:29)
        at org.ossreviewtoolkit.utils.ProcessCapture.<init>(ProcessCapture.kt:33)        
        at org.ossreviewtoolkit.utils.CommandLineTool$DefaultImpls.run(CommandLineTool.kt:100)
        at org.ossreviewtoolkit.scanner.LocalScanner.run(LocalScanner.kt:78)    
        at org.ossreviewtoolkit.utils.CommandLineTool$DefaultImpls.getVersion(CommandLineTool.kt:106)
        at org.ossreviewtoolkit.scanner.LocalScanner.getVersion(LocalScanner.kt:134)    
        at org.ossreviewtoolkit.scanner.LocalScanner.getDetails(LocalScanner.kt:151)
        at org.ossreviewtoolkit.scanner.LocalScanner.scanPackage(LocalScanner.kt:194)
        at org.ossreviewtoolkit.scanner.LocalScanner$scanPackages$2$invokeSuspend$$inlined$map$lambda$1.invokeSuspend(LocalScanner.kt:166)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
        at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
        at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:84)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
        at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
        at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
        at org.ossreviewtoolkit.scanner.Scanner.scanOrtResult(Scanner.kt:105)
        at org.ossreviewtoolkit.commands.ScannerCommand.run(ScannerCommand.kt:164)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:171)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:180)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:16)
        at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:395)
        at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:392)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:410)

(Sorry for pasting so excessively.)

Based on this and this issue, I put a gradle.properties next to the pom.xml file of the project to be scanned and, to be on the safe side, I tried setting JAVA_OPTS="-Xms1024M -Xmx6g -XX:MaxMetaspaceSize=2g -Dkotlin.daemon.jvm.options='-Xmx2g'" before running the scanner, but without success.

Is there an error somewhere in my memory options? Do I need to allow for even more memory?

I work with the latest ORT code on Ubuntu 20.04 which runs in a Virtual Box environment under Windows. The VM has 10 G available as main memory. A colleague of mine tried to scan the project under Windows and ran into the same problem.

I did not experience this issue when I was scanning the same project with ORT a few months ago. I don’t know what has changed since then.

I’d appreciate any help.

Thanks and keep up the great work.