spring-framework: Spring 5 Webclient throws ReadTimeout Exception [SPR-17610]

Viswanathan Manickam opened SPR-17610 and commented

I am getting readtimeout exception when I try to load test 1000 requests per second for 30 mins. Also, when the server sits idle for sometime and when I send request,  same error occurs the very first time but further requests process fine.

The response time for some requests is around 5 sec but other requests are in  250ms.

Please find the configuration below:

@Bean
public WebClient webClient() {
 TcpClient tcpClient = TcpClient.create()
 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 30000)
 .option(ChannelOption.SO_KEEPALIVE, true)
 .doOnConnected { connection ->
 connection.addHandlerLast(new ReadTimeoutHandler(5))
 .addHandlerLast(new WriteTimeoutHandler(30))
 }

 ReactorClientHttpConnector httpConnector = new ReactorClientHttpConnector(HttpClient.from(tcpClient));

 return WebClient.builder()
 .clientConnector(httpConnector)
 .build();
}

 

public Mono<String> retryPost(String url, JSONObject body) {
 Mono<String> result = webClient.post().uri(url)
 .contentType(MediaType.APPLICATION_JSON)
 .accept(MediaType.APPLICATION_JSON_UTF8)
 .body(BodyInserters.fromObject(body))
 .exchange()
 .log()
 .flatMap { clientResponse ->
 return handleResponse(clientResponse)
 }
 return result;
}

 

private Mono<String> handleResponse(ClientResponse clientResponse) {
 if (clientResponse.statusCode().is4xxClientError() || clientResponse.statusCode().is5xxServerError()) {
 return clientResponse.bodyToMono(String.class)
 .flatMap { errorBody ->
 return Mono.error(new CustomException(errorBody, clientResponse.statusCode().value()))
 }
 } else {
 return clientResponse.bodyToMono(String.class).log();
 }
}

Please find the log I received:

14:39:33.092 [http-nio-8080-exec-4] INFO  reactor.Mono.SwitchIfEmpty.11 - onSubscribe(FluxSwitchIfEmpty.SwitchIfEmptySubscriber)
14:39:33.093 [http-nio-8080-exec-4] INFO  reactor.Mono.SwitchIfEmpty.11 - request(unbounded)
14:39:38.128 [reactor-http-nio-4] ERROR r.n.http.client.HttpClientConnect - [id: 0x05f49b52, L::60850 - R:] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
14:39:38.131 [reactor-http-nio-4] ERROR reactor.Mono.SwitchIfEmpty.11 - onError(io.netty.handler.timeout.ReadTimeoutException)
14:39:38.131 [reactor-http-nio-4] ERROR reactor.Mono.SwitchIfEmpty.11 - 
io.netty.handler.timeout.ReadTimeoutException: null
14:39:38.179 [http-nio-8080-exec-5] ERROR c.f.l.a.GlobalControllerExceptionHandler - null
io.netty.handler.timeout.ReadTimeoutException: null

Let me know if I miss anything


Affects: 5.1.3

Reference URL: https://stackoverflow.com/questions/53786424/springboot-webclient-throws-readtimeout-errors

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Reactions: 1
  • Comments: 16

Most upvoted comments

@bclozel i found this issue may relates to the Reactor Connection Pool, i reproduced this problem in my dev envrionment,I did 4 requests at 2019-09-06 00:10:14.739(t1)、2019-09-06 00:19:51.274(t2)、2019-09-06 00:30:37.992(t3)、2019-09-06 00:33:13.314(t4), the request1(at t1), request2(at t2) webclient didn’t idle too long time, the requests were processed fine, but request3(at t3), webclient was idle 10minutes, webclient got Read Timeout, then i did request4 , it was processed fine as well. and i noticed that request1、request2、request3,they used same local port: 59838 but request4 used local port:60928

so i guess that: the connection(or channel in netty) will be cached in the connection pool, reactor netty will manage and reuse it, but when the connection idle too long, the connection may be closed or cut down by remote side(F5 loadbalance or Nginx?) or some other reason, it became disconnected on the link layer, but the connection pool did not realize what happened actually, the connection pool still thinks this connection is connected and provide it to webclient, and boom: read time out.

        HttpClient httpClient = HttpClient.create()
                .tcpConfiguration(tcpClient -> {
                    tcpClient = tcpClient.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 5000);
                    tcpClient = tcpClient.doOnConnected(conn -> conn
                            .addHandlerLast(new ReadTimeoutHandler(15000, TimeUnit.MILLISECONDS)));
                    return tcpClient;
                });

request1:

2019-09-06 00:10:14.768 [reactor-http-kqueue-5] DEBUG reactor.netty.http.client.HttpClientOperations - [id: 0x0759376f, L:/10.49.3.9:59838 - R:****.com/****:****] Received response (auto-read:false) : [Server=nginx, Date=Thu, 05 Sep 2019 16:10:15 GMT, Content-Type=application/json, Content-Length=64, Connection=keep-alive, Access-Control-Allow-Headers=Accept, Accept-Encoding, Authorization, Content-Type, Origin, Access-Control-Allow-Methods=GET, OPTIONS, Access-Control-Expose-Headers=Date, Vary=Accept-Encoding, Access-Control-Allow-Credentials=true]
2019-09-06 00:10:14.768 [reactor-http-kqueue-5] DEBUG reactor.netty.resources.PooledConnectionProvider - [id: 0x0759376f, L:/10.49.3.9:59838 - R:****.com/****:****] onStateChange(GET{uri=/api/v1/query?time=1567439463&query****), connection=PooledConnection{channel=[id: 0x0759376f, L:/10.49.3.9:59838 - R:****/****:****]}}, [response_received])
2019-09-06 00:10:14.768 [reactor-http-kqueue-5] DEBUG o.s.web.reactive.function.client.ExchangeFunctions - [1807c047] Response 200 OK

request2:

2019-09-06 00:19:51.272 [reactor-http-kqueue-5] DEBUG reactor.netty.http.client.HttpClientOperations - [id: 0x0759376f, L:/10.49.3.9:59838 - R:****.com/****:****] Received response (auto-read:false) : [Server=nginx, Date=Thu, 05 Sep 2019 16:19:51 GMT, Content-Type=application/json, Content-Length=64, Connection=keep-alive, Access-Control-Allow-Headers=Accept, Accept-Encoding, Authorization, Content-Type, Origin, Access-Control-Allow-Methods=GET, OPTIONS, Access-Control-Expose-Headers=Date, Vary=Accept-Encoding, Access-Control-Allow-Credentials=true]
2019-09-06 00:19:51.272 [reactor-http-kqueue-5] DEBUG reactor.netty.resources.PooledConnectionProvider - [id: 0x0759376f, L:/10.49.3.9:59838 - R:****.com/****:****] onStateChange(GET{uri=/api/v1/query?time=1567439463&query=****), connection=PooledConnection{channel=[id: 0x0759376f, L:/10.49.3.9:59838 - R:****.com/***:***]}}, [response_received])
2019-09-06 00:19:51.272 [reactor-http-kqueue-5] DEBUG o.s.web.reactive.function.client.ExchangeFunctions - [1a576a1] Response 200 OK

request3:

2019-09-06 00:30:52.998 [reactor-http-kqueue-5] WARN  reactor.netty.http.client.HttpClientConnect - [id: 0x0759376f, L:/10.49.3.9:59838 - R:njyh.data.promes.cloudytrace.com/10.104.232.131:9801] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
	Suppressed: java.lang.Exception: #block terminated with an error

request4:

2019-09-06 00:33:13.338 [reactor-http-kqueue-3] DEBUG reactor.netty.http.client.HttpClientOperations - [id: 0x47c1f6c3, L:/10.49.3.9:60928 - R:****.com/****:****] Received response (auto-read:false) : [Server=nginx, Date=Thu, 05 Sep 2019 16:33:13 GMT, Content-Type=application/json, Content-Length=64, Connection=keep-alive, Access-Control-Allow-Headers=Accept, Accept-Encoding, Authorization, Content-Type, Origin, Access-Control-Allow-Methods=GET, OPTIONS, Access-Control-Expose-Headers=Date, Vary=Accept-Encoding, Access-Control-Allow-Credentials=true]
2019-09-06 00:33:13.338 [reactor-http-kqueue-3] DEBUG reactor.netty.resources.PooledConnectionProvider - [id: 0x47c1f6c3, L:/10.49.3.9:60928 - R:****.com/****:***] onStateChange(GET{uri=/api/v1/query?time=1567439463&query=****), connection=PooledConnection{channel=[id: 0x47c1f6c3, L:/10.49.3.9:60928 - R:****.com/***:****]}}, [response_received])
2019-09-06 00:33:13.338 [reactor-http-kqueue-3] DEBUG o.s.web.reactive.function.client.ExchangeFunctions - [11c0862d] Response 200 OK

sorry for my poor english