reactor-netty: WebClient fails with Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms after 33000 request in loop
I’ve tried write a simple load test with WebClient. One thread with simple loop, 100k iterations, but after 2000 or 3000 cycles I receive a message:
Apr 09, 2020 6:00:38 AM io.netty.util.ResourceLeakDetector reportTracedLeak
SEVERE: LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
Created at:
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:363)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:139)
io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:147)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
After all 100k iterations, I’ve received only 33k successful responses, and no one was failed. My code written to wait for all responses and prints success/failed requests after one second of sleep. After 45 seconds, I’ve received other 67k failed requests with exception:
reactor.netty.internal.shaded.reactor.pool.PoolAcquireTimeoutException: Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms
at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.run(AbstractPool.java:334)
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
|_ checkpoint ⇢ Request to GET http://localhost:8080/users/50725/5362 [DefaultWebClient]
Stack trace:
at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.run(AbstractPool.java:334)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
And my stats looks like
test results, success: 33,085, errors: 66,914
Note that 33085 + 66914 = 99999
Steps to Reproduce
I wrote a simple app with spring-webflux:5.2.5 and reactor-netty:0.9.6:
public class WebClientTest {
public static void main(String... args) throws InterruptedException {
WebClient webClient = WebClient.builder()
.baseUrl("http://localhost:8080")
.build();
AtomicLong successCount = new AtomicLong();
AtomicLong errorCount = new AtomicLong();
int requestCount = 100000;
System.out.println("test start");
for (int i = 0; i < requestCount; i++) {
if (i % (requestCount / 100) == 0)
System.out.printf("sent: %d requests\n", i);
webClient.get()
.uri("/users/{id}", i)
.retrieve()
.toBodilessEntity()
.subscribe(response -> successCount.incrementAndGet(), throwable -> {
throwable.printStackTrace();
errorCount.incrementAndGet();
});
}
while (true) {
long success = successCount.get();
long error = errorCount.get();
System.out.printf("test results, success: %,d, errors: %,d\n", success, error);
if (success + error == requestCount)
break;
Thread.sleep(1000);
}
}
}
My build.gradle
apply plugin: 'java'
repositories {
mavenLocal()
mavenCentral()
}
dependencies {
compile 'org.springframework:spring-webflux:5.2.5.RELEASE'
compile 'io.projectreactor.netty:reactor-netty:0.9.6.RELEASE'
}
Your Environment
Mac OS Mojave,
- Reactor version(s) used: 0.9.6.RELEASE
- Other relevant libraries versions (eg.
netty, …):4.1.48.Final - JVM version (
java -version):java version "1.8.0_192" - OS and version (eg
uname -a):Darwin ahesofmbr 18.7.0 Darwin Kernel Version 18.7.0: Sun Dec 1 18:59:03 PST 2019; root:xnu-4903.278.19~1/RELEASE_X86_64 x86_64
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 5
- Comments: 15 (7 by maintainers)
@echoesbr the fix for this one should be available already with 5.2.6 and above (see the linked issue). The other one in Reactor Netty is in 0.9.5 and above.
@vioao This was closed with a fix in Spring Framework (see the linked issue). If you experience a similar issue please open a new issue with a reproducible example.