netty: netty thread was infinite loop in BaseMpscLinkedArrayQueue.poll
Expected behavior
spring cloud gateway shoud comsume new requests.
Actual behavior
i’m using spring cloud bulding a project,the gateway was spring cloud gateway which using [webflux ](web-application-type: reactive),and it’s runing on netty. after about in 48 hours, the gateway didn’t work. the process is runing,but it didn’t accept new request anymore.
after did some analysises,i got some clues below:
accept queue overflows:
two days later,the queue is always full.
also,i killed some tcp link manually,but the process didn’t consume.
top threads is using cpu:
accoding to the cpu usage by the threads that appears reactor threads is in the infinite loop(the details can find in the thread file below):
“reactor-http-nio-1” #42 daemon prio=5 os_prio=0 tid=0x0000ffff7ce24000 nid=0x4015 runnable [0x0000ffff48bbd000]
java.lang.Thread.State: RUNNABLE
at io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueue.poll(BaseMpscLinkedArrayQueue.java:340)
at io.netty.util.internal.shaded.org.jctools.queues.MpscUnboundedArrayQueue.poll(MpscUnboundedArrayQueue.java:23)
at io.netty.util.concurrent.SingleThreadEventExecutor.pollTaskFrom(SingleThreadEventExecutor.java:215)
at io.netty.util.concurrent.SingleThreadEventExecutor.pollTask(SingleThreadEventExecutor.java:210)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:459)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:750)

DirectMemory usage is in an appropriate,about 83M:

jvm heap was fine:

this is thread file: gateway_thread.txt
Steps to reproduce
Couldn’t reproduce
Minimal yet complete reproducer code (or URL to code)
Netty version
4.1.74
JVM version (e.g. java -version)
Java HotSpot™ 64-Bit Server 1.8.0_341
OS version (e.g. uname -a)
CentOS Linux release 7.5.1804 (AltArch) Linux minio1 4.14.0-49.el7a.aarch64
Could anyone give me some advice? thanks
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 19 (10 by maintainers)
The problem did not happen anymore after i upgraded jdk to openjdk11, thanks all of you.
@yellowsand1 I take all potential bugs in JCTools seriously. What version of the JDK are you on? I understand moving from 8 to 11 is a large effort, but updating to the latest update of the major version you are on is a reasonable ask.
I agree with @franz1981 here… I think the most likely is a JDK bug.
@nitsanw FYI…