spring-cloud-netflix: EurekaClient shutdown causing BeanCreationNotAllowedException which prevents unregistration of the client from the registry (RefreshScope)

BeanCreationNotAllowedException is thrown during application context shutdown by the DiscoveryClient.

This is issue has already been reported since long in different scenarios and different (apparent) causes. There have been many attempts at solving the issue so far, but unfortunately, although each attempt solved one cause, there are still cases where the exception is thrown.

A short summary first before a more detailed description below:

• the problem occurs during shutdown of the application after the eureka client has successfully registered with the registry
• the problem does NOT occur if the SpringCloud RefreshScope feature is disabled via spring.cloud.refresh.enabled=false. RefreshScope is the root cause in this scenario.
• it can be easily reproduced with Boot 2.0.4 or 1.5.14 (ie Spring Cloud Edgware.SR4 or Finchley.SR1)
• the exception prevents the client from being unregistered from the registry during shutdon

The problem can be easily reproduced with a simple Spring Initilizr bootstrap project with only the Eureka Discovery dependency selected. This is reproductible with Boot 2.0.4 or 1.5.14 (ie Spring Cloud Edgware.SR4 or Finchley.SR1). You need a running Eureka server so the sample client can be successfully registered. Start the application with the standard configuration (nothing special needed here) and make sure it is registered with the Eureka registry. Then stop the application properly so the shutdown sequence is executed (Ctrl-C should do it). You should see the following log lines during the shutdown:

2018-09-03 13:50:26.443  INFO 9494 --- [on(2)-127.0.0.1] inMXBeanRegistrar$SpringApplicationAdmin : Application shutdown requested.
2018-09-03 13:50:26.443  INFO 9494 --- [on(2)-127.0.0.1] ationConfigEmbeddedWebApplicationContext : Closing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@7a362b6b: startup date [Mon Sep 03 13:49:45 CEST 2018]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@5c30a9b0
2018-09-03 13:50:26.444  INFO 9494 --- [on(2)-127.0.0.1] o.s.c.n.e.s.EurekaServiceRegistry        : Unregistering application unknown with eureka with status DOWN
2018-09-03 13:50:26.444  WARN 9494 --- [on(2)-127.0.0.1] com.netflix.discovery.DiscoveryClient    : Saw local status change event StatusChangeEvent [timestamp=1535975426444, current=DOWN, previous=UP]
2018-09-03 13:50:26.444  INFO 9494 --- [nfoReplicator-0] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_UNKNOWN/macfury.lan: registering service...
2018-09-03 13:50:26.445  INFO 9494 --- [on(2)-127.0.0.1] o.s.c.support.DefaultLifecycleProcessor  : Stopping beans in phase 0
2018-09-03 13:50:26.446  INFO 9494 --- [on(2)-127.0.0.1] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans on shutdown
2018-09-03 13:50:26.446  INFO 9494 --- [on(2)-127.0.0.1] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans
2018-09-03 13:50:26.448  INFO 9494 --- [on(2)-127.0.0.1] com.netflix.discovery.DiscoveryClient    : Shutting down DiscoveryClient ...
2018-09-03 13:50:26.450  WARN 9494 --- [on(2)-127.0.0.1] .s.c.a.CommonAnnotationBeanPostProcessor : Invocation of destroy method failed on bean with name 'scopedTarget.eurekaClient': org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'eurekaInstanceConfigBean': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
2018-09-03 13:50:26.455  INFO 9494 --- [nfoReplicator-0] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_UNKNOWN/macfury.lan - registration status: 204
2018-09-03 13:50:26.505  INFO 9494 --- [on(2)-127.0.0.1] o.apache.catalina.core.StandardService   : Stopping service [Tomcat]

As you can see in the last few lines, a BeanCreationNotAllowedException exception is thrown when destroying the bean scopedTarget.eurekaClient. The remaining part of the message gives us some hints about what’s going on:

Error creating bean with name ‘eurekaInstanceConfigBean’: Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)

The DiscoveryClient#shutdown() method starts with the following statements:

(1)    public synchronized void shutdown() {
(2)        if (isShutdown.compareAndSet(false, true)) {
(3)            logger.info("Shutting down DiscoveryClient ...");
(4)
(5)            if (statusChangeListener != null && applicationInfoManager != null) {
(6)                applicationInfoManager.unregisterStatusChangeListener(statusChangeListener.getId());
(7)            }

The exception is thrown when the execution hits line (6). You should remember both the DiscoveryClient and ApplicationInfoManager are RefreshScope beans (unless the Refresh feature is disabled with spring.cloud.refresh.enabled=false, it is enabled by default). Consequence is the applicationInfoManager is a refresh-scope proxy whose target must be resolved when the unregisterStatusChangeListener is invoked - this is done via a getBean(..) call on the application context. The shutdown method is invoked by the application context during its shutdown procedure. At this point the underlying BeanFactory is in a closing state and actively refuses any getBean(...) call made by other components, throwing BeanCreationNotAllowedException at any attempt.

Let’s recap:

• DiscoveryClient.shutdown() is invoked during the application context shutdown/close process
• DiscoveryClient needs to unregister from the ApplicationInfoManager
• applicationInfoManager is refresh-scoped, its target must be resolved for every method invocation
• the resolution fails with a BeanCreationNotAllowedException because the BeanFactory is in closing state.

At the end, we do actually request a bean from a BeanFactory in a destroy method implementation!, which is, as per the exception message, not allowed.

The problem does NOT appear if the refresh feature is disabled with spring.cloud.refresh.enabled=false.