reactor-netty: reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response

I use spring cloud gateway as our gateway, this exception occurs in our production environment randomly when the request arrived. I search the related issue on the github and found it had been fixed, but my reactor-netty is the latest release(0.9.7).

Steps to Reproduce

It is very difficult to reproduce，which only occurs in production environment randomly. This is the log when the exception occurs:

2020-05-12 17:31:22.948 DEBUG [reactor-http-epoll-11] [PooledConnectionProvider] [id: 0xcbc4c890, L:/172.18.206.6:23016 - R:172.18.207.3/172.18.207.3:8151] Channel acquired, now 1 active connections and 0 inactive connections 2020-05-12 17:31:22.948 DEBUG [reactor-http-epoll-11] [HttpClientConnect] [id: 0xcbc4c890, L:/172.18.206.6:23016 - R:172.18.207.3/172.18.207.3:8151] Handler is being applied: {uri=http://172.18.207.3:8151/route, method=POST} 2020-05-12 17:31:22.948 DEBUG [reactor-http-epoll-11] [PooledConnectionProvider] [id: 0xcbc4c890, L:/172.18.206.6:23016 - R:172.18.207.3/172.18.207.3:8151] onStateChange(POST{uri=/route, connection=PooledConnection{channel=[id: 0xcbc4c890, L:/172.18.206.6:23016 - R:172.18.207.3/172.18.207.3:8151]}}, [request_prepared]) 2020-05-12 17:31:22.948 DEBUG [reactor-http-epoll-11] [PooledConnectionProvider] [id: 0xcbc4c890, L:/172.18.206.6:23016 - R:172.18.207.3/172.18.207.3:8151] onStateChange(POST{uri=/route, connection=PooledConnection{channel=[id: 0xcbc4c890, L:/172.18.206.6:23016 - R:172.18.207.3/172.18.207.3:8151]}}, [request_sent]) 2020-05-12 17:31:22.948 DEBUG [reactor-http-epoll-11] [PooledConnectionProvider] [id: 0xcbc4c890, L:/172.18.206.6:23016 ! R:172.18.207.3/172.18.207.3:8151] Channel closed, now 0 active connections and 0 inactive connections 2020-05-12 17:31:22.948 DEBUG [reactor-http-epoll-11] [PooledConnectionProvider] [id: 0xcbc4c890, L:/172.18.206.6:23016 ! R:172.18.207.3/172.18.207.3:8151] onStateChange(POST{uri=/route, connection=PooledConnection{channel=[id: 0xcbc4c890, L:/172.18.206.6:23016 ! R:172.18.207.3/172.18.207.3:8151]}}, [response_incomplete]) 2020-05-12 17:31:22.949 WARN [reactor-http-epoll-11] [HttpClientConnect] [id: 0xcbc4c890, L:/172.18.206.6:23016 ! R:172.18.207.3/172.18.207.3:8151] The connection observed an error reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response

The configuration of spring cloud gateway:

spring.cloud.gateway.httpclient.pool.type=fixed spring.cloud.gateway.httpclient.pool.max-connections=1000 spring.cloud.gateway.httpclient.pool.acquire-timeout=7500 spring.cloud.gateway.httpclient.pool.max-idle-time=90s spring.cloud.gateway.httpclient.pool.max-life-time=90s

Your Environment

reactor-netty: 0.9.7.RELEASE
netty: 4.1.49.Final
spring cloud gateway: 2.2.2.RELEASE
JVM version (javar -version): jdk 1.8
OS and version (eg uname -a): centos 6.7

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 21 (9 by maintainers)

Most upvoted comments

From the linked issue https://github.com/spring-cloud/spring-cloud-gateway/issues/1148

It appears that the server is Tomcat with configurations

connectionTimeout | The number of milliseconds this Connector will wait, after accepting a connection, for the request URI line to be presented. Use a value of -1 to indicate no (i.e. infinite) timeout. The default value is 60000 (i.e. 60 seconds) but note that the standard server.xml that ships with Tomcat sets this to 20000 (i.e. 20 seconds). Unless disableUploadTimeout is set to false, this timeout will also be used when reading the request body (if any).

and

keepAliveTimeout | The number of milliseconds this Connector will wait for another HTTP request before closing the connection. The default value is to use the value that has been set for the connectionTimeout attribute. Use a value of -1 to indicate no (i.e. infinite) timeout.

These two configurations might be a cause for a connection close.

I would recommend to configure maxIdleTime on the client side having in mind the keepAliveTimeout on Tomcat. Without such configuration Reactor Netty can receive the close event at any time between acquiring the connection from the pool and before actual sending of the request. Also you might want to switch to LIFO leasing strategy so that you will use always the most recently used connection.

violetagg on Jun 24, 2020

@violetagg There are no logs on the server. I have a question, why sending a request to the server on the closed connection. Could we check the connection state before sending the request?

yantaowu on Jun 4, 2020

when I made poor server that send response header only(not response body) the result is Connection prematurely closed DURING response cf ) https://github.com/reactor/reactor-netty/issues/1037#issuecomment-602444231

so I think that PrematureCloseException is not always a reactor-netty problem.

yangbongsoo on Jun 4, 2020