netty: netty thread was infinite loop in BaseMpscLinkedArrayQueue.poll

Expected behavior

spring cloud gateway shoud comsume new requests.

Actual behavior

i’m using spring cloud bulding a project,the gateway was spring cloud gateway which using [webflux ](web-application-type: reactive),and it’s runing on netty. after about in 48 hours, the gateway didn’t work. the process is runing,but it didn’t accept new request anymore. after did some analysises,i got some clues below: accept queue overflows: two days later,the queue is always full. accept_queue also,i killed some tcp link manually,but the process didn’t consume.

top threads is using cpu: top accoding to the cpu usage by the threads that appears reactor threads is in the infinite loop(the details can find in the thread file below): “reactor-http-nio-1” #42 daemon prio=5 os_prio=0 tid=0x0000ffff7ce24000 nid=0x4015 runnable [0x0000ffff48bbd000] java.lang.Thread.State: RUNNABLE at io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueue.poll(BaseMpscLinkedArrayQueue.java:340) at io.netty.util.internal.shaded.org.jctools.queues.MpscUnboundedArrayQueue.poll(MpscUnboundedArrayQueue.java:23) at io.netty.util.concurrent.SingleThreadEventExecutor.pollTaskFrom(SingleThreadEventExecutor.java:215) at io.netty.util.concurrent.SingleThreadEventExecutor.pollTask(SingleThreadEventExecutor.java:210) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:459) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) image

DirectMemory usage is in an appropriate,about 83M: image

jvm heap was fine: image

this is thread file: gateway_thread.txt

Steps to reproduce

Couldn’t reproduce

Minimal yet complete reproducer code (or URL to code)

Netty version

4.1.74

JVM version (e.g. java -version)

Java HotSpot™ 64-Bit Server 1.8.0_341

OS version (e.g. uname -a)

CentOS Linux release 7.5.1804 (AltArch) Linux minio1 4.14.0-49.el7a.aarch64

Could anyone give me some advice? thanks

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 19 (10 by maintainers)

Most upvoted comments

The problem did not happen anymore after i upgraded jdk to openjdk11, thanks all of you.

@yellowsand1 I take all potential bugs in JCTools seriously. What version of the JDK are you on? I understand moving from 8 to 11 is a large effort, but updating to the latest update of the major version you are on is a reasonable ask.

I agree with @franz1981 here… I think the most likely is a JDK bug.