classgraph: OOM while loading classpath with large JAR files
Hi team, thank you for making this awesome lib!
So I’m working on a project to generate a graph for dependencies downloaded by a specific Gradle build.
I managed to write a plugin for Gradle and get all JARs associated with a project once build is finished, and then call ClassGraph’s scan
function within the plugin, and using overrideClasspath
to point classpath to all JARs.
While these JARs are being necessary to generate a complete graph, the entire size of those JARs could be extremely large (2GB+). Which leads to Gradle to raise OOM exception since the heap is exhausted.
I’m wondering what’s the best practice to deal with this kind of situation where the JARs are very large? Is that possible for ClassGraph to process chunk by chunk?
Thanks in advance!
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 1
- Comments: 57 (27 by maintainers)
Commits related to this issue
- Custom-intern all classfile constant pool strings to save memory (#338) — committed to classgraph/classgraph by lukehutch 5 years ago
- Speed up whitelist checking for large whitelists (#338) — committed to classgraph/classgraph by lukehutch 5 years ago
@lukehutch Wow, thank you so much for all the great suggestions and even their improvements. I’ve read each line of your code and I know exactly what you are trying to accomplish, and they totally make sense and would work!
@laonawuli can you provide a heap dump?