spring-cloud-openfeign: Feign client lazy initialization fails when using CompletableFuture.supplyAsync() to call the Feign client initially

Describe the bug Feign client initialization is always lazy. As such, the first calling thread triggers the initialization, which may be another Spring component (e.g. REST controller) using CompletableFuture.supplyAsync() to do so. This uses ForkJoinWorkerThread underneath, resulting in a ClassNotFoundException, if and only if running in a dockerized environment based on Spring Boot Build Image plugin.

Using other JVM Executors(except newWorkStealingExecutor), e.g. fixedSizeThreadExecutor(), does not result in this behavior. Also eagerly initializing the Feign client by calling it synchronously, e.g. from within a CommandLineRunner, circumvents this. Once, the Feign client is properly initialized, there is no problem, however, Feign does not yet support eager initialization.

Sample A sample project with instructions to reproduce is provided here: https://github.com/maverick1601/openfeign-build-image-error

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 3
Comments: 29

Links to this issue

spring - Feign Client : use jar running (CompletableFuture.supplyAsync call Feign Client throw java.lang.IllegalArgumentException) - Stack Overflow

Commits related to this issue

Inject the class loader from the parent context into the child context more aggressively. `ConditionEvaluator` extracts the class loader from the bean factory at instantiation, and then uses said cla... — committed to tommyk-gears/spring-cloud-commons by tommyk-gears 2 years ago
Inject the class loader from the parent context into the child context more aggressively. `ConditionEvaluator` extracts the class loader from the bean factory at instantiation, and then uses said cla... — committed to tommyk-gears/spring-cloud-commons by tommyk-gears 2 years ago

Most upvoted comments

Spring boot 3.1.0 spring-cloud 2022.0.3 Java 17

The previous workaround is working with little changes

 @Bean
  @ConditionalOnMissingBean
  public LoadBalancerClientFactory loadBalancerClientFactory(LoadBalancerClientsProperties properties) {
    return new LoadBalancerClientFactory(properties) {
      @Override public AnnotationConfigApplicationContext createContext(String name) {
        ClassLoader originalClassLoader = Thread.currentThread().getContextClassLoader();
        Thread.currentThread().setContextClassLoader(this.getClass().getClassLoader());
        AnnotationConfigApplicationContext context = (AnnotationConfigApplicationContext) super.createContext(name);
        Thread.currentThread().setContextClassLoader(originalClassLoader);
        return context;
      }
    };
  }

badrkacimi on Jun 26, 2023

The previous workaround didn’t work for me as the constructor of LoadBalancerClientFactory without arguments is deprecated, as a result that solution causes NPE.

Here is the modified version:

    @Bean
    @ConditionalOnMissingBean
    public LoadBalancerClientFactory loadBalancerClientFactory(LoadBalancerClientsProperties properties) {
        return new LoadBalancerClientFactory(properties) {
            @Override
            protected AnnotationConfigApplicationContext createContext(String name) {
                // FIXME: temporary switch classloader to use the correct one when creating the context
                ClassLoader originalClassLoader = Thread.currentThread().getContextClassLoader();
                Thread.currentThread().setContextClassLoader(this.getClass().getClassLoader());
                AnnotationConfigApplicationContext context = super.createContext(name);
                Thread.currentThread().setContextClassLoader(originalClassLoader);
                return context;
            }
        };
    }

DenisKorolev on Apr 28, 2022

We got similar issue in one of our applications. After investigation, we found this is caused by https://bugs.openjdk.org/browse/JDK-8172726.

Start with Java 9, ForkJoinPool creates thread using system class loader as context class loader, refer to https://github.com/openjdk/jdk11u/blob/master/src/java.base/share/classes/java/util/concurrent/ForkJoinPool.java#L721.

If we run applications in IDE, the system class loader has been used to load Spring and application classes, so it’s not easy to reproduce this issue. If we run applications from command line using fat JAR, org.springframework.boot.loader.LaunchedURLClassLoader will be used to load those classes instead. If threads switch to use system class loader, it will not be able to load Spring and other classes inside the fat JAR.

crmky on Jul 7, 2023

An alternative solution is to use rxjava,

mvn dep

<dependency> <groupId>io.reactivex</groupId> <artifactId>rxjava</artifactId> </dependency>

mgrigis on Mar 1, 2022

This issue is not within the team’s priorities right now, but is open to community contributions.

OlgaMaciaszek on Oct 28, 2021

@maverick1601, thanks for reporting this. Will discuss it with the team and get back to you.

OlgaMaciaszek on Jan 29, 2021

Our applications use a different workaround. We created ApplicationContextInitializer implementation (and added to spring.factories) to manually set the class loader as the context class loader who starts the application.

public class ClassLoaderApplicationContextInitializer
        implements ApplicationContextInitializer<ConfigurableApplicationContext> {

    @Override
    public void initialize(ConfigurableApplicationContext applicationContext) {
        applicationContext.setClassLoader(applicationContext.getClassLoader());
    }
}

crmky on Jul 7, 2023

The fix in https://github.com/spring-cloud/spring-cloud-commons/pull/1098 tries to use the class loader of parent ApplicationContext in child ApplicationContext:

context = new AnnotationConfigApplicationContext(beanFactory); context.setClassLoader(this.parent.getClassLoader());

However typically the parent ApplicationContext doesn’t set class loader explicitly, it will use ClassUtils.getDefaultClassLoader() to return the default class loader. Unfortunately that implementation will return context class loader if that has been set.

crmky on Jul 7, 2023

I’m facing the exact same problem after an update from Spring 2.3 to Spring 2.7 + Java update from 14 to 17. We are not using any @Async or CompletableFuture, but the feign client is called by a Grizzly Thread (started as part of a Spring Boot app) that is part of our core infrastructure and needs to call another running Spring Boot app via Feign. This used to work without any problems before the update. Disappointed way too often to see such an issue open for such a long time with no prospect of a solution

HJK181 on Oct 5, 2022

Hi, is there any news about this issue and the associated PR? I encountered a similar issue when using openfeign client called within a method annotated as @Async that uses a ForkJoinPool executor (spring-boot: 2.5.5, spring-cloud: 2020.0.4). I have yet to find a solution or workaround.

P.S. I think it’s related to #600

P.S.S. I’ve tried the solution described in #600 and I managed to make it working. I hope that it can be fixed asap because it’s not only a openfeign issue but it does apply on LoadBalanced RestTemplates too

davgia on Oct 6, 2021

We experience the same issue with k8s resource limits. After we removed the CPU limit of 1, our application failed to execute Feign requests as the POD where the application runs now had more CPUs visible to the JVM. 4 instead of 1. I’m wondering why the Spring team does not see this as a serious issue as the usage of CompletableFuture in a project is very likely and we have numerous comments here with lots of related issues that were closed …

HJK181 on Apr 17, 2023

The previous workaround didn’t work for me as the constructor of LoadBalancerClientFactory without arguments is deprecated, as a result that solution causes NPE.

@DenisKorolev is there any particular reason why you have omitted the following line from the previous workaround?
clientFactory.setConfigurations(configurations.getIfAvailable(Collections::emptyList));
This line still appears in the org.springframework.cloud.loadbalancer.config.LoadBalancerAutoConfiguration.

If you have a custom loadbalancer configure, like @LoadBalancerClients(defaultConfiguration = CustomLoadBalancerConfiguration.class) you should add it loadBalancerClientFactory.setConfigurations(configurations.getIfAvailable(Collections::emptyList));

    @Bean
    @ConditionalOnMissingBean
    public LoadBalancerClientFactory loadBalancerClientFactory(LoadBalancerClientsProperties properties, ObjectProvider<List<LoadBalancerClientSpecification>> configurations) {
        LoadBalancerClientFactory loadBalancerClientFactory = new LoadBalancerClientFactory(properties) {
            @Override
            protected AnnotationConfigApplicationContext createContext(String name) {
                // FIXME: temporary switch classloader to use the correct one when creating the context
                ClassLoader originalClassLoader = Thread.currentThread().getContextClassLoader();
                Thread.currentThread().setContextClassLoader(this.getClass().getClassLoader());
                AnnotationConfigApplicationContext context = super.createContext(name);
                Thread.currentThread().setContextClassLoader(originalClassLoader);
                return context;
            }
        };
        loadBalancerClientFactory.setConfigurations(configurations.getIfAvailable(Collections::emptyList));
        return loadBalancerClientFactory;
    }

kisick on Apr 15, 2023

I wanted to share some information about some similar bug.

We have one spring service working with docker, k8, java 17 and the use of CompletableFuture, everything was working ok, until we decided to add more memory and cpu

We pass from

resources:
  requests:
    memory: "64Mi"
    cpu: "48m"
  limits:
    memory: "512Mi"
    cpu: "1000m"

  resources:
    requests:
      memory: "512Mi"
      cpu: "60m"
    limits:
      memory: "512Mi"

And the same bug appear Could not find class [org.springframework.boot.autoconfigure.condition.OnBeanCondition] the classLoader was almost empty, like we pass from 15000 class to 50 in the classLoader.

But the the code was related to StreamBridge and not feign client (even if we use it in the code, but not directly in the code called), but for me is the same JVM/spring bug. Is was working ok with the IDE, but the Jar was failing, etc

The solution put in place was add

@Configuration
public class TaskExecutorConfig {

  @Bean("asyncPool")
  public Executor asyncPool() {

    return Executors.newFixedThreadPool(10);
  }
}

And pass it when we call the CompletableFuture

CompletableFuture.runAsync(
  () -> {
                your code here...
            },
  asyncPool
);

nicolasArmando on Feb 28, 2023

Made another sample project that reproduces a problem. It also contains a submodule with proposed workaround by @DenisKorolev. There are also a test with boot 2.3 and ribbon that works fine and a test for when eager-loading enabled for Spring’s loadbalancer (available in boot 3+, but you can implement the same logic for yourself if it fits).

Psimage on Oct 30, 2022

My problem like it, #1139

But my problem is deadlock, Maybe it’s the same thing with different consequences

Found one Java-level deadlock:
=============================
"ForkJoinPool.commonPool-worker-2":
  waiting to lock monitor 0x000000002c456c18 (object 0x00000005c2284aa8, a java.util.concurrent.ConcurrentHashMap),
  which is held by "main"
"main":
  waiting to lock monitor 0x000000002bda1db8 (object 0x000000071649d0e0, a java.util.concurrent.ConcurrentHashMap),
  which is held by "ForkJoinPool.commonPool-worker-2"

Java stack information for the threads listed above:
===================================================
"ForkJoinPool.commonPool-worker-2":
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:179)
        - waiting to lock <0x00000005c2284aa8> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:493)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:520)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:491)
        at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:227)
        at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:231)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1419)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1218)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1175)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostPro
cessor.java:595)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:90)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.
java:376)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1404)      
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:592)       
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
        at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$162/2081000371.getObject(Unknown Source)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        - locked <0x000000071a98c5a8> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:847)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
        - locked <0x000000071a98c200> (a java.lang.Object)
        at org.springframework.cloud.context.named.NamedContextFactory.createContext(NamedContextFactory.java:136)
        at org.springframework.cloud.context.named.NamedContextFactory.getContext(NamedContextFactory.java:101)
        - locked <0x000000071649d0e0> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getContext(SpringClientFactory.java:131)
        at org.springframework.cloud.context.named.NamedContextFactory.getInstance(NamedContextFactory.java:145)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getInstance(SpringClientFactory.java:121)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getClientConfig(SpringClientFactory.java:75)
        at org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient.getClientConfig(LoadBalancerFeignClient.java:97)
        at org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:81)
        at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:110)
        at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:80)
        at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103)
        at com.sun.proxy.$Proxy154.getPersonOne(Unknown Source)
        at com.siiri.cts.open.biz.config.TestDeadLockConfig.lambda$initBean$0(TestDeadLockConfig.java:28)
        at com.siiri.cts.open.biz.config.TestDeadLockConfig$$Lambda$565/2000449863.run(Unknown Source)
        at java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1626)
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java)
        at java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1618)
        at java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
"main":
        at org.springframework.cloud.context.named.NamedContextFactory.getContext(NamedContextFactory.java:100)
        - waiting to lock <0x000000071649d0e0> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getContext(SpringClientFactory.java:131)
        at org.springframework.cloud.context.named.NamedContextFactory.getInstance(NamedContextFactory.java:145)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getInstance(SpringClientFactory.java:121)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getClientConfig(SpringClientFactory.java:75)
        at org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient.getClientConfig(LoadBalancerFeignClient.java:97)
        at org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:81)
        at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:110)
        at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:80)
        at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103)
        at com.sun.proxy.$Proxy153.getByAccountName(Unknown Source)
        at com.siiri.cts.open.biz.config.TestDeadLockConfig.initBean(TestDeadLockConfig.java:33)
        at com.siiri.cts.open.biz.config.TestDeadLockConfig$$EnhancerBySpringCGLIB$$3776b5f.CGLIB$initBean$0(<generated>)
        at com.siiri.cts.open.biz.config.TestDeadLockConfig$$EnhancerBySpringCGLIB$$3776b5f$$FastClassBySpringCGLIB$$bc6c3884.invoke(<generated>)       
        at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363)       
        at com.siiri.cts.open.biz.config.TestDeadLockConfig$$EnhancerBySpringCGLIB$$3776b5f.initBean(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:622)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:456)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory
.java:1320)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1159)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)       
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
        at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$162/2081000371.getObject(Unknown Source)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        - locked <0x00000005c2284aa8> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:847)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
        - locked <0x00000005c23cb290> (a java.lang.Object)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1204)
        at com.siiri.cts.open.web.OpenServiceWebApplication.main(OpenServiceWebApplication.java:42)

Found 1 deadlock.

DDAaTao on Sep 2, 2022

The previous workaround didn’t work for me as the constructor of LoadBalancerClientFactory without arguments is deprecated, as a result that solution causes NPE.

@DenisKorolev is there any particular reason why you have omitted the following line from the previous workaround?

clientFactory.setConfigurations(configurations.getIfAvailable(Collections::emptyList));

This line still appears in the org.springframework.cloud.loadbalancer.config.LoadBalancerAutoConfiguration.

timosmit on May 10, 2022