spring-framework: Memory leak with CglibAopProxy$ProxyCallbackFilter
It seems that there is a memory leak with Spring Boot 2.4.1 on my Debian 10.5 with Java 11.0.1 when using @PreAuthorize("hasRole('ROLE')") annotation on controller’s endpoint. As you can see in the first image, there is constant grow in old generation memory of G1 garbage collector when running tests. The grow stops at max heap size which is 512 MB. I prepared a sample project where you can reproduce the problem.
Memory usage 1

Memory usage 2

After importing the project just run this command:
./gradlew clean test
Tests will fail because they assume that HTTP response will be 200, but it is 401, but it shouldn’t matter.
If you remove @PreAuthorize from DummyController or prePostEnabled = true from SecurityConfig then memory usage seems to be ok.
Memory usage 3

Memory usage 4

The project uses dummy controller with one endpoint which has @PreAuthorize("hasRole('ROLE')") annotation. It has also basic SecurityConfig to enable @PreAuthorize annotation. Controller uses some dummy services to make sure that for each test Spring will recreate the context. I also set spring.test.context.cache.maxSize=1 in build.gradle to make sure that the grow of memory does not come from cache.
If you have any more questions about my configuration or anything else, let me know.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 11
- Comments: 30 (16 by maintainers)
I’ve got a promising solution in the works where we reduce the
ProxyCallbackFilterinstance to a plain key role afters its initial use for class generation, stubbing out the Advisors with key stubs (containing advice class and pointcuts just for comparison purposes) once the class has been generated.While the
CGLIB$CALLBACK_FILTERfield remains non-null, its content will not retain any leaky references anymore, independent from the concreteAdvisorimplementations in use there. This should hopefully address the common scenarios for long-lived ClassLoaders as outlined above.Same here. with spring-boot 3.0.4 and spring-framework 6.0.6
Running a few integration test with @DirtiesContext and then I get an OOM after a few tests.
In my case it seems to be the @Validated annotation
After running three simple tests with @DirtiesContext I have3 applicationContexts on the heap. Setting a breakpoint in one test, doing forced GC and getting a heap with YourKit Profiler looks like this:
After adding this workaround to my test context, everything works again: ((thanks to @redhead, code slightly modified)
@jhoeller Unfortunately, it does not appear to have fixed this specific issue. Updating the original sample to use Spring 5.3.10 does not resolve the issue. I’ve put together a branch named spring-5.3.10 that demonstrates it is not fixed. If you run
./gradlew checkyou will see lots of failed tests (expected) and then eventually you will start getting out of memory errors. I haven’t looked into if anything has changed from the previous assessment, but the take away at this point is the tests still produce out of memory errors.Can we please reopen the issue?
This would have been useful information in the original description.
Yes, I believe this is the case. The use of
@PreAuthorizerequiresDummyControllerto be proxied and it’s this proxy creation that is causing the memory leak. Each new context creates a newDummyControllersub-class for the CGLib proxy. There’s nothing we can do about this in Spring Boot as@PreAuthorizeis a Spring Security feature and CGLib proxy creation is part of Spring Framework.I think we should get someone from the Framework team to look at this. It may be that something isn’t behaving as intended or that could be improved in the proxy creation that would enable some reuse.
I tried to write a class loader to collect created proxy classes and clean the static field after the context is closed. However, our setup of contexts is not trivial and I was running into many class-to-class problems (classes not assignable to each other, etc.).
So I went for the BeanPostProcessor instead, which might not work at all times maybe, but works for us so far.
It collects the bean classes if they are Spring proxy classes and clears the static field when the bean post processor is disposed (context is closed).
I reproduced it by just setting
ext['spring-framework.version']='5.3.10'into build.gradle in the original example. But previously I didn’t reproduce it in your. It seems there is a dependency on test order, which can be different. I know that it is unspecified by default, but it seems that gradle eventually orders tests by hash of directory content or modification time.So here is heap dumps here that launch after a minute
Thanks for bringing this to my attention @sbrannen.
It does appear that the class generated by CGLIB is (indirectly) holding a reference to
MethodSecurityMetadataSourceAdvisorwhich in turn holds a reference to theBeanFactory.For reference here is a list of objects retained by
ProxyCallbackFilterHere is the shortest GC Root of
ProxyCalbackFilterI don’t think Spring Security is the correct place to solve this. Any cleanup that the
Advisors perform would just be masking the actual leak which is in Spring Framework + CGLIB. Even if theAdvisorcleans up all of its resources, theAdvisoritself is leaked and so are lower level objects (i.e.AdvisedSupport).I think it would be ideal if Spring Framework could remove the reference to the
ProxyCallbackFilteronCALLBACK_FILTERof the generated class once we know that theCALLBACK_FILTERis no longer needed. It might also mark the generated class asclosedso that if it is used again it reports an error.Removing the refererence to
ProxyCallbackFilterwould ensure that references toAdvisedSupportare cleaned up along with itsmethodCache, all references to itsadvisors, and its remaining members. It’s important to note that cleaning up references toadvisorswill clean up the reference toMethodSecurityMetadataSourceAdvisorand thus theBeanFactoryit refers to. It will also cleanup references to otherAdvisors and any large objects they might refer to. For example,AbstractBeanFactoryPointcutAdvisorhas it’s ownBeanFactoryreference.Obviously this still means the generated classes are still unnecessarily around, but I’m not sure we can drop the references to those given they are held by the jdk.