reactor-netty: Reactor netty client doesn't return response/fail after CloudFoundry gorouter request timeout

I am from SAP and we develop product which is working in CloudFoundry environment and make requests to CC api using official cf java client. The issue is related with http requests made by reactor-netty client, sometimes requests pass global set limit(15 minutes) of gorouter and return response with error code 502 due to timeout. Gorouter is main component in CloudFoundry which is used as reverse proxy for almost everything in CF and it used also as load balancer. Unfortunately such requests made by reactor-netty client doesn’t return or fail any response from gorouter.

Expected Behavior

It should be return or fail with errors from CF gorouter and don’t hang indefinitely.

Actual Behavior

Actual behavior right now is when this CF gorouter timeout occurs, reactor-netty client doesn’t return or fails with any errors and if you don’t set responseTimeout configuration, it hangs indefinitely.

Steps to Reproduce

Push into CF simple web application which return response with big delay (in this case more than 15 minutes). Make http request to the CF application with reactor-netty client, you can take a look at example code for client

@Test
public void repoCase() {
 HttpClient client = HttpClient.create()
                                      .wiretap(true);
        long startTime = System.currentTimeMillis();
        Mono<Foo> response = client.get()
                                   .uri("https://<route-to-your-cf-application>")
                                   .responseSingle((responses, bytes) -> bytes.asString()
                                                                              .map(it -> new Foo(responses.status()
                                                                                                          .code(),
                                                                                                 it)));

        Thread t1 = new Thread(() -> {
            try {
                Thread.sleep(TimeUnit.MINUTES.toMillis(15));
            } catch (InterruptedException e) {
                System.out.println("Finished before 15 minutes");
                return;
            }
            System.out.println("15 minutes already passed out");
        });
        t1.start();

        Foo responseObject = response.block();

        long endTime = System.currentTimeMillis();
        System.out.println(MessageFormat.format("Response status {0} body:{1} for time: {2} ms", responseObject.status,
                                                responseObject.response, endTime - startTime));
        t1.interrupt();
}

static class Foo {
        int status;
        String response;

        public Foo(int status, String response) {
            this.status = status;
            this.response = response;
        }
}

When I try to execute the same request to my CF application from terminal using curl, I receive timeout response from gorouter:

time curl 'https://<route-to-your-cf-application>' -v
.....
> GET /configurations/1/v2/service_instances/aa2c9957-2577-4666-af70-cdc604eebd6d HTTP/1.1
> Host: <route-to-your-cf-application>
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 502 Bad Gateway
< Content-Type: text/plain; charset=utf-8
< X-Cf-Routererror: endpoint_failure
< X-Content-Type-Options: nosniff
< Date: Thu, 11 Feb 2021 16:04:25 GMT
< Content-Length: 67
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload;
<
502 Bad Gateway: Registered endpoint failed to handle the request.
* Connection #0 to host <route-to-your-cf-application> left intact

real    15m0.528s
user    0m0.000s
sys     0m0.125s

Possible Solution

Your Environment

• Reactor version(s) used: 0.9.12.RELEASE
• Other relevant libraries versions (eg. netty, …): netty-codec-http: 4.1.52.RELEASE - other netty jars are with the same version
• JVM version (java -version): SAPMachine based on openjdk version 11.0.9.1
• OS and version (eg. uname -a): Windows 10 1909

Also I did enabled debug logs of reactor-netty client and I paste them here - https://gist.github.com/theghost5800/cf4cfedf812121a2f2b874dfbd0396f9

Overall I am understanding that I have to configure responseTimeout but results between execution of curl command and reactor-netty are different.