spring-cloud-sleuth: reactor actuator invalid response when too much actuator request on decorate-on-each set true

Describe the bug Please provide details of the problem, including the version of Spring Cloud that you are using.

set: webflux + sleuth + actuator version: set(‘springCloudVersion’, “Hoxton.SR1”) and id ‘org.springframework.boot’ version ‘2.2.4.RELEASE’

In decorate-on-each: true setting, actuator not work after 100 time’s call. but, In decorate-on-each: false setting, then, actuator work after 100 time’s call.

this log about In decorate-on-each: true setting, after 100 time’s call, and call /actuator/health then error log

2020-02-11 14:53:50.628 ERROR [,8baf0e4c1ad5671a,8baf0e4c1ad5671a,false] 40736 --- [ctor-http-nio-7] a.w.r.e.AbstractErrorWebExceptionHandler : [da7e3300]  500 Server Error for HTTP GET "/actuator/health"

reactor.core.Exceptions$ReactorRejectedExecutionException: hard cap on deferred tasks reached for boundedElastic("boundedElastic",maxThreads=2,maxTaskQueued=2,ttl=60s)
	at reactor.core.Exceptions.failWithRejected(Exceptions.java:275) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	|_ checkpoint ⇢ Handler org.springframework.boot.actuate.endpoint.web.reactive.AbstractWebFluxEndpointHandlerMapping$ReadOperationHandler#handle(ServerWebExchange) [DispatcherHandler]
	|_ checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.cloud.sleuth.instrument.web.TraceWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ HTTP GET "/actuator/health" [ExceptionHandlingWebHandler]
Stack trace:
		at reactor.core.Exceptions.failWithRejected(Exceptions.java:275) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.scheduler.BoundedElasticScheduler.schedule(BoundedElasticScheduler.java:242) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.scheduler.Schedulers$CachedScheduler.schedule(Schedulers.java:1040) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoSubscribeOnCallable.subscribe(MonoSubscribeOnCallable.java:52) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4105) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoZip.subscribe(MonoZip.java:128) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4105) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:418) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:243) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:201) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:363) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4105) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoZip.subscribe(MonoZip.java:128) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4105) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:418) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:243) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:201) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:363) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2199) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:162) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:103) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1637) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:203) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:203) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1637) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoIgnoreThen$ThenAcceptInner.onNext(MonoIgnoreThen.java:296) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1637) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:144) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1637) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:247) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:329) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96) ~[spring-cloud-sleuth-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:173) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2199) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:132) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.request(ScopePassingSpanSubscriber.java:79) ~[spring-cloud-sleuth-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
		at reactor.core.publisher.MonoZip$ZipInner.onSubscribe(MonoZip.java:318) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onSubscribe(ScopePassingSpanSubscriber.java:71) ~[spring-cloud-sleuth-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onSubscribe(MonoPeekTerminal.java:145) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4105) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoZip.subscribe(MonoZip.java:128) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:153) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:274) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:851) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96) ~[spring-cloud-sleuth-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2199) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:162) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.request(ScopePassingSpanSubscriber.java:79) ~[spring-cloud-sleuth-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2007) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:1881) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onSubscribe(ScopePassingSpanSubscriber.java:71) ~[spring-cloud-sleuth-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4105) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:441) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:211) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at org.springframework.cloud.sleuth.instrument.web.TraceWebFilter$MonoWebFilterTrace.subscribe(TraceWebFilter.java:180) ~[spring-cloud-sleuth-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4105) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.netty.http.server.HttpServerHandle.onStateChange(HttpServerHandle.java:64) ~[reactor-netty-0.9.4.RELEASE.jar:0.9.4.RELEASE]
		at reactor.netty.tcp.TcpServerBind$ChildObserver.onStateChange(TcpServerBind.java:228) ~[reactor-netty-0.9.4.RELEASE.jar:0.9.4.RELEASE]
		at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:465) ~[reactor-netty-0.9.4.RELEASE.jar:0.9.4.RELEASE]
		at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:90) ~[reactor-netty-0.9.4.RELEASE.jar:0.9.4.RELEASE]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:167) ~[reactor-netty-0.9.4.RELEASE.jar:0.9.4.RELEASE]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) ~[netty-codec-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) ~[netty-codec-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.45.Final.jar:4.1.45.Final]
		at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_191]

Sample If possible, please provide a test case or sample application that reproduces the problem. This makes it much easier for us to diagnose the problem and to verify that we have fixed it.

https://github.com/creatorKoo/error-report-actuator

start with this option -Dreactor.schedulers.defaultBoundedElasticSize=2 -Dreactor.schedulers.defaultBoundedElasticQueueSize=2

and

#!/bin/bash
# actuator request 100 times
curl http://127.0.0.1:11140/actuator/health &                                                                                                                                                                        curl http://127.0.0.1:11140/actuator/health &                                                                                                                                                                        
curl http://127.0.0.1:11140/actuator/health &       
curl http://127.0.0.1:11140/actuator/health &       
... (more than 100 lines) ...
curl http://127.0.0.1:11140/actuator/health

run this script and call /actuator/health and not error

and remove that info from bootstrap.yaml (set default)

spring:
  sleuth:
    reactor:
#      decorate-on-each: true
      decorate-on-each: false # this option fix error but lose useful functions.

run 100time’s script again and call /actuator/health and then error repeat.

I want to use decorate-on-each: true option with high load actuator call.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 18 (8 by maintainers)

Commits related to this issue

update reactor-core 3.3.3.BUILD-SNAPSHOT fix https://github.com/spring-cloud/spring-cloud-sleuth/issues/1556 — committed to creatorKoo/error-report-actuator by creatorKoo 4 years ago

Most upvoted comments

In the meantime, as a workaround that should be removed as soon as the reactor-core fix is usable, you can set the task queue limit to ‘unbounded’ (Integer.MAX_VALUE):

-Dreactor.schedulers.defaultBoundedElasticQueueSize=2147483647

simonbasle on Feb 12, 2020

actually I don’t agree with your findings that it is ok without sleuth. Having sleuth added to the mix just seems to accelerate the problem, probably because a bunch of blocking health indicators get registered by spring-cloud, which will occupy more spots in the bounded queue…

In any case, there is an issue with boundedElasticScheduler(), obviously. We’re exploring an alternative implementation that should not have these recovery issues in reactor/reactor-core#2040

simonbasle on Feb 12, 2020

Ah I get it, yeah. The boundedElasticScheduler never seems to recover from the error / clear its queue of deferred tasks. Not sure how the spring cloud health indicators play a role in that 🤔

simonbasle on Feb 12, 2020