reactor-netty: WebClient fails with Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms after 33000 request in loop

I’ve tried write a simple load test with WebClient. One thread with simple loop, 100k iterations, but after 2000 or 3000 cycles I receive a message:

Apr 09, 2020 6:00:38 AM io.netty.util.ResourceLeakDetector reportTracedLeak
SEVERE: LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
    io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:363)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
    io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:139)
    io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:147)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.lang.Thread.run(Thread.java:748)

After all 100k iterations, I’ve received only 33k successful responses, and no one was failed. My code written to wait for all responses and prints success/failed requests after one second of sleep. After 45 seconds, I’ve received other 67k failed requests with exception:

reactor.netty.internal.shaded.reactor.pool.PoolAcquireTimeoutException: Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms
    at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.run(AbstractPool.java:334)
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
    |_ checkpoint ⇢ Request to GET http://localhost:8080/users/50725/5362 [DefaultWebClient]
Stack trace:
        at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.run(AbstractPool.java:334)
        at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
        at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

And my stats looks like

test results, success: 33,085, errors: 66,914 

Note that 33085 + 66914 = 99999

Steps to Reproduce

I wrote a simple app with spring-webflux:5.2.5 and reactor-netty:0.9.6:

public class WebClientTest {

    public static void main(String... args) throws InterruptedException {
        WebClient webClient = WebClient.builder()
                .baseUrl("http://localhost:8080")
                .build();

        AtomicLong successCount = new AtomicLong();
        AtomicLong errorCount = new AtomicLong();
        int requestCount = 100000;
        System.out.println("test start");
        for (int i = 0; i < requestCount; i++) {
            if (i % (requestCount / 100) == 0)
                System.out.printf("sent: %d requests\n", i);

            webClient.get()
                    .uri("/users/{id}", i)
                    .retrieve()
                    .toBodilessEntity()
                    .subscribe(response -> successCount.incrementAndGet(), throwable -> {
                        throwable.printStackTrace();
                        errorCount.incrementAndGet();
                    });
        }


        while (true) {
            long success = successCount.get();
            long error = errorCount.get();
            System.out.printf("test results, success: %,d, errors: %,d\n", success, error);
            if (success + error == requestCount)
                break;

            Thread.sleep(1000);
        }
    }
}

My build.gradle

apply plugin: 'java'

repositories {
    mavenLocal()
    mavenCentral()
}

dependencies {
    compile 'org.springframework:spring-webflux:5.2.5.RELEASE'
    compile 'io.projectreactor.netty:reactor-netty:0.9.6.RELEASE'
}

Your Environment

Mac OS Mojave,

  • Reactor version(s) used: 0.9.6.RELEASE
  • Other relevant libraries versions (eg. netty, …): 4.1.48.Final
  • JVM version (java -version): java version "1.8.0_192"
  • OS and version (eg uname -a): Darwin ahesofmbr 18.7.0 Darwin Kernel Version 18.7.0: Sun Dec 1 18:59:03 PST 2019; root:xnu-4903.278.19~1/RELEASE_X86_64 x86_64

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 5
  • Comments: 15 (7 by maintainers)

Most upvoted comments

@echoesbr the fix for this one should be available already with 5.2.6 and above (see the linked issue). The other one in Reactor Netty is in 0.9.5 and above.

@vioao This was closed with a fix in Spring Framework (see the linked issue). If you experience a similar issue please open a new issue with a reproducible example.