spring-framework: Memory leak with CglibAopProxy$ProxyCallbackFilter

It seems that there is a memory leak with Spring Boot 2.4.1 on my Debian 10.5 with Java 11.0.1 when using @PreAuthorize("hasRole('ROLE')") annotation on controller’s endpoint. As you can see in the first image, there is constant grow in old generation memory of G1 garbage collector when running tests. The grow stops at max heap size which is 512 MB. I prepared a sample project where you can reproduce the problem.

Memory usage 1

image

Memory usage 2

image

After importing the project just run this command:

./gradlew clean test

Tests will fail because they assume that HTTP response will be 200, but it is 401, but it shouldn’t matter. If you remove @PreAuthorize from DummyController or prePostEnabled = true from SecurityConfig then memory usage seems to be ok.

Memory usage 3

image

Memory usage 4

image

The project uses dummy controller with one endpoint which has @PreAuthorize("hasRole('ROLE')") annotation. It has also basic SecurityConfig to enable @PreAuthorize annotation. Controller uses some dummy services to make sure that for each test Spring will recreate the context. I also set spring.test.context.cache.maxSize=1 in build.gradle to make sure that the grow of memory does not come from cache.

If you have any more questions about my configuration or anything else, let me know.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 11
  • Comments: 30 (16 by maintainers)

Commits related to this issue

Most upvoted comments

I’ve got a promising solution in the works where we reduce the ProxyCallbackFilter instance to a plain key role afters its initial use for class generation, stubbing out the Advisors with key stubs (containing advice class and pointcuts just for comparison purposes) once the class has been generated.

While the CGLIB$CALLBACK_FILTER field remains non-null, its content will not retain any leaky references anymore, independent from the concrete Advisor implementations in use there. This should hopefully address the common scenarios for long-lived ClassLoaders as outlined above.

Same here. with spring-boot 3.0.4 and spring-framework 6.0.6

Running a few integration test with @DirtiesContext and then I get an OOM after a few tests.

In my case it seems to be the @Validated annotation


@Controller
@Validated
@RequiredArgsConstructor
public class WettquoteLigamappingAnlegenServiceGQL {
     // ... injected services fields ...
    @Transactional
    @MutationMapping
    public @NotNull WettquoteLigamappingDTO wettquoteLigamappingAnlegen(
             @Argument @NotNull @Valid WettquoteLigamappingForm wettquoteLigamappingForm) {
            // ....
    }
}

After running three simple tests with @DirtiesContext I have3 applicationContexts on the heap. Setting a breakpoint in one test, doing forced GC and getting a heap with YourKit Profiler looks like this:

Bildschirmfoto vom 2023-03-28 10-41-34

After adding this workaround to my test context, everything works again: ((thanks to @redhead, code slightly modified)

package de.kicktipp.testutils;

import lombok.extern.slf4j.Slf4j;
import org.jetbrains.annotations.NotNull;
import org.springframework.beans.BeansException;
import org.springframework.beans.factory.DisposableBean;
import org.springframework.beans.factory.config.BeanPostProcessor;
import org.springframework.stereotype.Component;
import java.util.HashSet;
import java.util.Set;

//https://github.com/spring-projects/spring-framework/issues/26266
@Component
@Slf4j
public class MemoryLeakFix implements BeanPostProcessor, DisposableBean {

    private static final String CGLIB_PROXY_STATIC_FIELD_WITH_MEMORY_LEAK = "CGLIB$CALLBACK_FILTER";
    private static final String SPRING_CGLIB_PROXY_CLASS_NAME_TAG = "SpringCGLIB";

    private final Set<Class<?>> classesToCleanUp = new HashSet<>();

    @Override
    public Object postProcessAfterInitialization(Object bean, @NotNull String beanName) throws BeansException {
        if (bean.getClass().getName().contains(SPRING_CGLIB_PROXY_CLASS_NAME_TAG)) {
            classesToCleanUp.add(bean.getClass());
        }
        return bean;
    }

    @Override
    public void destroy() {
        classesToCleanUp.forEach(cls -> {
            try {
                var field = cls.getDeclaredField(CGLIB_PROXY_STATIC_FIELD_WITH_MEMORY_LEAK);
                field.setAccessible(true);
                field.set(null, null);
            } catch (NoSuchFieldException e) {
                log.info("No Field", e);
            } catch (IllegalAccessException e) {
                log.error(e.getMessage(), e);
            }
        });
        classesToCleanUp.clear();
    }
}

@jhoeller Unfortunately, it does not appear to have fixed this specific issue. Updating the original sample to use Spring 5.3.10 does not resolve the issue. I’ve put together a branch named spring-5.3.10 that demonstrates it is not fixed. If you run ./gradlew check you will see lots of failed tests (expected) and then eventually you will start getting out of memory errors. I haven’t looked into if anything has changed from the previous assessment, but the take away at this point is the tests still produce out of memory errors.

Can we please reopen the issue?

I didn’t try to reproduce the problem with application’s main method because the problem rather is connected somehow with Spring that fails to clean up old context when tests are running

This would have been useful information in the original description.

Maybe it is somehow connected with @PreAuthorize

Yes, I believe this is the case. The use of @PreAuthorize requires DummyController to be proxied and it’s this proxy creation that is causing the memory leak. Each new context creates a new DummyController sub-class for the CGLib proxy. There’s nothing we can do about this in Spring Boot as @PreAuthorize is a Spring Security feature and CGLib proxy creation is part of Spring Framework.

I think we should get someone from the Framework team to look at this. It may be that something isn’t behaving as intended or that could be improved in the proxy creation that would enable some reuse.

I tried to write a class loader to collect created proxy classes and clean the static field after the context is closed. However, our setup of contexts is not trivial and I was running into many class-to-class problems (classes not assignable to each other, etc.).

So I went for the BeanPostProcessor instead, which might not work at all times maybe, but works for us so far.

public class MemoryLeakFixBeanPostProcessor implements BeanPostProcessor, DisposableBean {

    /**
     * See {@link org.springframework.cglib.proxy.Enhancer}
     */
    private static final String CGLIB_PROXY_STATIC_FIELD_WITH_MEMORY_LEAK = "CGLIB$CALLBACK_FILTER";

    /**
     * See {@link org.springframework.cglib.core.SpringNamingPolicy}
     */
    private static final String SPRING_CGLIB_PROXY_CLASS_NAME_TAG = "BySpringCGLIB";

    private final Set<Class<?>> classesToCleanUp = new HashSet<>();

    @Override
    public Object postProcessAfterInitialization(Object bean, @Nonnull String beanName) throws BeansException {
        if (bean.getClass().getName().contains(SPRING_CGLIB_PROXY_CLASS_NAME_TAG)) {
            classesToCleanUp.add(cls);
        }
        return bean;
    }

    @Override
    public void destroy() throws Exception {
        classesToCleanUp.forEach(cls -> {
            try {
                var field = cls.getDeclaredField(CGLIB_PROXY_STATIC_FIELD_WITH_MEMORY_LEAK);
                field.setAccessible(true);
                field.set(null, null);
            } catch (Exception e) {
                // we can't do anything..
            }
        });
        classesToCleanUp.clear();
    }
}

It collects the bean classes if they are Spring proxy classes and clears the static field when the bean post processor is disposed (context is closed).

I reproduced it by just setting ext['spring-framework.version']='5.3.10' into build.gradle in the original example. But previously I didn’t reproduce it in your. It seems there is a dependency on test order, which can be different. I know that it is unspecified by default, but it seems that gradle eventually orders tests by hash of directory content or modification time.

Снимок экрана_2021-10-12_06-55-57

So here is heap dumps here that launch after a minute

Thanks for bringing this to my attention @sbrannen.

It does appear that the class generated by CGLIB is (indirectly) holding a reference to MethodSecurityMetadataSourceAdvisor which in turn holds a reference to the BeanFactory.

For reference here is a list of objects retained by ProxyCallbackFilter

image

Here is the shortest GC Root of ProxyCalbackFilter

image

I don’t think Spring Security is the correct place to solve this. Any cleanup that the Advisors perform would just be masking the actual leak which is in Spring Framework + CGLIB. Even if the Advisor cleans up all of its resources, the Advisor itself is leaked and so are lower level objects (i.e. AdvisedSupport).

I think it would be ideal if Spring Framework could remove the reference to the ProxyCallbackFilter on CALLBACK_FILTER of the generated class once we know that the CALLBACK_FILTER is no longer needed. It might also mark the generated class as closed so that if it is used again it reports an error.

Removing the refererence to ProxyCallbackFilter would ensure that references to AdvisedSupport are cleaned up along with its methodCache, all references to its advisors, and its remaining members. It’s important to note that cleaning up references to advisors will clean up the reference to MethodSecurityMetadataSourceAdvisor and thus the BeanFactory it refers to. It will also cleanup references to other Advisors and any large objects they might refer to. For example, AbstractBeanFactoryPointcutAdvisor has it’s own BeanFactory reference.

Obviously this still means the generated classes are still unnecessarily around, but I’m not sure we can drop the references to those given they are held by the jdk.