immutables: Significant memory leak in version 2.9.0

When we try to upgrade from version 2.8.8 to version 2.9.0 our medium-sized multi-module Maven build fails with java.lang.OutOfMemoryError: Java heap space. Based on investigation using JFR/JMC and git bisect the issue appears to be introduced in these two commits:

080040703de9ee9a6a842e783be66a3a21ff24bb (#1235).
2b9a79ce359d852c5c697792fd3f9f17b3c8efeb (#1249).

Some numbers:

After the first commit meta space usage increases from ~115MB to ~260MB, while heap memory usage increases from ~270MB to ~620MB.
After the second commit heap memory usage further increases to ~960MB.

(I performed many runs. There is little variance in these runs; on the order of ~5-10MB. After the first commit the classical GC zig-zag/sawtooth pattern all but disappears, hinting at a classical memory leak. See also the screenshots at the bottom of this message.)

JMC’s Memory -> Live Objects view hints at the thread local state of StaticEnvironment being retained. Based on some subsequent debugging the following issues appear to contribute:

AbstractGenerator#init calls StaticEnvironment.preInit, but if AbstractGenerator#process is not subsequently called then StaticEnvironment#shutdown won’t be invoked.
Although we don’t fork, it appears that each invocation of the maven-compiler-plugin loads a new StaticEnvironment class, which means that for each invocation a new thread-local EnvironmentState is created.
These two things combined appear to cause a significant memory leak, (I think) especially due to the accumulation of ProcessingEnvironments.

I don’t yet see a quick fix. Unfortunately javax.annotation.processing.Processor does not provide a “cleanup” or “shutdown” method. The proper fix likely involves getting rid of all this static and thread-local state (as also hinted at here), but that seems like a large task…

That said, as-is version 2.9.0 of Immutables appears to be unusable for us. The build on which I reproduced this has 52 pom.xml files with 90 maven-compiler-plugin invocations (compile and testCompile combined), but we also have a few much-larger builds with 250+ pom.xml files.

Below three screenshots from JMC’s JVM Internals -> Garbage Collections view. The first build uses e31f58d44612f2211679a07229a96d9ddf62aa7a (i.e. the parent of the first problematic commit), the second uses 080040703de9ee9a6a842e783be66a3a21ff24bb, while the third uses 2b9a79ce359d852c5c697792fd3f9f17b3c8efeb. Note the Heap Post GC and Metaspace graphs.

immutables-run-1 immutables-run-2 immutables-run-3

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 2
Comments: 18 (15 by maintainers)

Commits related to this issue

#1355 first attempt to restructure leaking env — committed to immutables/immutables by elucash 2 years ago
#1355 second attempt, less statics, flushing files earlier — committed to immutables/immutables by elucash 2 years ago
some cleanup after CI debug for #1355 — committed to immutables/immutables by elucash 2 years ago
#1355 implementing suggested WeakReference solution — committed to immutables/immutables by elucash 2 years ago

Most upvoted comments

That’s makes sense, thank you for more context/explanation. This change is pending the release for some time now, I just got overwhelmed with other stuff, but also I need to update gpg keys for release and this became a sort of yak shaving. I still hope to just sit and make the release asap.

elucash on Aug 19, 2022

Hello folks. First, thanks a lot for all the research you have been doing. I ran into a similar situation and it was really easy to solve given that you already had a patch available.

I still have no clear understanding how threading / annotation processing instances are managed so that we have such a leak, but if this solution works, this is great!

@elucash I may be able to provide some context here. We are building a large Maven project composed of 450 modules. And I noticed a pretty bad classloader leak (>5.5 GB usage in the Metaspace).

I think what is happening is that every time a module is built, a new ClassLoader is instantiated. Each time, the org.immutables:value annotation processor is loaded in a separate class loader. And since it stores its StaticEnvironment in a ThreadLocal, with a strong reference, the environment is never deleted. Thus, the class definition stays reachable, and so does the class loader itself.

After ~20 modules compiled with the annotation processor, I ended up with as many leaked class loaders (including all their static variables, leading to a memory leak in the heap as well).

See this screenshot, which shows how ThreadLocals hold references to multiple StaticEnvironments, and therefore to their class loaders.

eclipse-mat-classloaders-path-to-gc-root

I cloned this project and built it locally from master. I can confirm that with the fix proposed by @Stephan202, the memory leak is gone.

Would it be possible to cut a release with this change please?

pingtimeout on Aug 19, 2022

Thanks !

Just one clarification, I forgot to mention that in my case, I was running with 2.8.8 when I ran into the leak, not 2.9.0. The whole analysis is still valid as the class loader leak was caused by strong references in thread locals. I just want to make it clear that older versions are affected as well.

So, just in case, for users who run an outdated version: upgrade to ≥2.9.1 to benefit from the WeakReference fix.

pingtimeout on Aug 22, 2022

just a heads up for 2.9.1 released, closing this, hopefully fixed 😃

elucash on Aug 22, 2022

@elucash would it be possible to have a new release with this change? (We’re interested in adopting a release that includes #1156. 😃 )

NB: I noticed that there are no Git tags associated to to the various 2.8.9 and 2.9.0 released found on Maven Central. If they stil exist on your local machine, could you please push those?

Stephan202 on Jun 9, 2022

@Stephan202 thank you! Pushed these changes. If that helps, i’m fully ok with the change. Because EnvironmentState is pinned as a field of the annotation processor, it should stay reachable for as long as the annotation processor instance is reachable and when the processor will be GC, the thread-local weakref will be enqueued. I still have no clear understanding how threading / annotation processing instances are managed so that we have such a leak, but if this solution works, this is great! We will be able to release new version once this fix will be confirmed plus I’ll take a look at couple of new issues added recently (exception in the annotation processor + <any> type again etc), hopefully very soon.

elucash on Apr 19, 2022

Performed a new run based on ca0fa7ca6d1b4533815333c6ad0e4512997f1c3e, and the results look very promising:

Meta space ended up at ~80MB (~30% reduction compared to version 2.8.8!).
Heap space ended up at ~430MB (a ~60% increase compared to version 2.8.8).

immutables-after-ca0fa7c

Stephan202 on Jan 24, 2022

@elucash thanks for jumping on this! I just ran a test against a0010e93e41402c33f45d01befc433662c385c2e and heap space ended up at ~560MB while meta space ended up at 105MB: immutables-after-a0010e9

So your changes appear to at least bring heap space usage back in line with version 2.8.8 levels. Heap usage OTOH still appears to be double of what it was before. (But: nice fix! Happy to see this didn’t involve a major rewrite 😅.)

Stephan202 on Jan 23, 2022