keycloak: ProcessingException: RESTEASY004655: Unable to invoke request: java.net.SocketException: Connection reset

I was able to reliable reproduce the issue using the latest keycloak and keycloak-admin-client (each 22.0.5).

Reproducer setup

I am running on Ubuntu 20.04 (WSL2 on Windows) and OpenJDK 17.

• Start a keycloak server: docker run -p 8081:8080 -e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=admin quay.io/keycloak/keycloak:22.0.5 start-dev
• Have tcpkill send RST packets to connections on port 8081 (you may need to sudo apt install dsniff): sudo tcpkill -1 -i lo port 8081
  • This is to simulate a firewall or other network infrastructure closing the connection. In our case we suspect a firewall to kill connections that are open for too long (10+ hours)
• Set up a java (e.g. maven) project that uses the keycloak-admin-client and run this reproducer:

  public class App {
      public static void main(String[] args) throws InterruptedException {
          Keycloak kc = KeycloakBuilder.builder()
                  .serverUrl("http://localhost:8081")
                  .realm("master")
                  .username("admin")
                  .password("admin")
                  .clientId("admin-cli")
                  .build();
          while (true) {
              tryTalkToKeycloakWithExpiredToken(kc);
              Thread.sleep(100);
          }
      }
    
      private static void tryTalkToKeycloakWithExpiredToken(Keycloak kc) {
          // force the next request to refresh the token
          kc.tokenManager().invalidate(kc.tokenManager().getAccessToken().getToken());
          kc.realm("master").getAdminEvents().size(); // trigger any request
      }
  }
  

The reproducer may take a few iterations until it hits this exception and exits:

Exception in thread "main" jakarta.ws.rs.ProcessingException: RESTEASY004655: Unable to invoke request: java.net.SocketException: Connection reset
	at org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine.invoke(ManualClosingApacheHttpClient43Engine.java:361)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:427)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:134)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:103)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:61)
	at jdk.proxy2/jdk.proxy2.$Proxy16.refreshToken(Unknown Source)
	at org.keycloak.admin.client.token.TokenManager.refreshToken(TokenManager.java:121)
	at org.keycloak.admin.client.token.TokenManager.getAccessToken(TokenManager.java:77)
	at org.keycloak.admin.client.token.TokenManager.getAccessTokenString(TokenManager.java:70)
	at org.keycloak.admin.client.resource.BearerAuthFilter.filter(BearerAuthFilter.java:52)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.filterRequest(ClientInvocation.java:644)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:424)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:134)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:103)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:61)
	at jdk.proxy2/jdk.proxy2.$Proxy29.getAdminEvents(Unknown Source)
	at org.example.App.tryTalkToKeycloakWithExpiredToken(App.java:33)
	at org.example.App.main(App.java:25)
Caused by: java.net.SocketException: Connection reset
	at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:323)
	at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)
	at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)
	at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine.invoke(ManualClosingApacheHttpClient43Engine.java:344)
	... 17 more

Some iterations may fully succeed, some may only print this message:

Nov 07, 2023 12:51:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://localhost:8081: Connection reset by peer
Nov 07, 2023 12:51:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://localhost:8081

I believe the differences are due to tcpkill working on a best-effort and timing based approach to inject RST packets, which may arrive at the application sooner or later. The desicionmaking whether a request is retried lives inside the DefaultHttpRequestRetryHandler. I observed that retryRequest (the method that decides whether to retry a request) ends with this snippet of code:

        if (!clientContext.isRequestSent() || this.requestSentRetryEnabled) {
            // Retry if the request has not been sent fully or
            // if it's OK to retry methods that have been sent
            return true;
        }
        // otherwise do not retry
        return false;

The code always reaches that statement. I saw that this.requestSentRetryEnabled is always false, and the deciding factor is clientContext.isRequestSent(). My conclusion is that the POST request to refresh the token has been fully sent before a RST packet was received, resulting in the retry mechanism not retrying the request because it might have been processed by the server and can not be assumed to be idempotent.

Possible solutions

Assuming the troublemaker is some network component closing idle connections, the problem can be avoided by auto-closing idle connections and/or testing pooled connections before use. This is what @tramiaczek suggested in his previous comment https://github.com/keycloak/keycloak/issues/8917#issuecomment-1712895984

A minimal configuration to enable auto-closing of connections may look like this:

Keycloak kc = KeycloakBuilder.builder()
        .serverUrl("http://localhost:8081")
        .realm("master")
        .username("admin")
        .password("admin")
        .clientId("admin-cli")
        .resteasyClient(((ResteasyClientBuilder) ClientBuilder.newBuilder())
                .connectionPoolSize(3)
                .connectionTTL(10, TimeUnit.SECONDS)
                .build())
        .build();

For this specific usecase, namely the keycloak-admin-client transparently refreshing its token, it might be useful for it to automatically retry this too, since other connection failures are typically retried as well. Though I am unsure how this would be done.

Alternatively, application-side retry mechanisms may be considered. Given the keycloak admin client may throw connection errors at any time anyway, applications should probably be robust against this.

@tramiaczek thanks for clarification. Do I understand correctly that when passing the custom client the error did not occur again?

Yes, it currently works almost 2 months and the error was not reopened by the client.

We have the problem reported from the client’s environment - random java.net.SocketException: Connection reset while executing get user operation (in most cases). Our scenario was" create a user (through Keycloak API) and generate some OTP for it (OTP generation code was not related to the Keycloak API, but we needed to get user Id from Keycloak to put it as a reference in our database). Keycloak and the service that invoked the API were deployed as separate pod’s in the same Kubernetes namespace - the client’s TEST/UAT env and our DEV environments were pretty the same (client followed our instructions to prepare env).

Of course as we could not reproduce the issue, we tried to seek for the solution in a blind manner - we tried to use the latest possible version of keycloak-admin-client library, but it did not help. So we thought that we should have as wide control over the HTTP client as possible and prepared the workaround described above. Our first idea was to turn on some DEBUG logs and observe how the connection pool or timeouts behave on client’s env, especially when the problem occurs again. But it did not occur… and the client closed the issue in Jira 😃 As client swears that he did not change enything in the environment, we suppose that some idle connection eviction/validation mechanims resolved the problem.

I have the same issue, but possibly found the workaround (it is under tests now). KeycloakBuilder has a method: .resteasyClient(..) to provide custom implementation of javax.ws.rs.client.Client and that implementation could have some settings about connection/read timeouts, evicting idle connections policy and retry handler implementation. Spring configuration for Keycloak bean could look like the following:

    @Bean
    public Client customizedResteasyClient() {
        CloseableHttpClient httpClient = createHttpClient();

        ApacheHttpClient43Engine engine = new ApacheHttpClient43Engine(httpClient);
        return ((ResteasyClientBuilder) ClientBuilder.newBuilder())
                .httpEngine(engine)
                .connectTimeout(10000, TimeUnit.MILLISECONDS)
                .readTimeout(7000, TimeUnit.MILLISECONDS)
                .connectionTTL(-1, TimeUnit.MILLISECONDS)
                .disableTrustManager()
                .build();
    }

    private CloseableHttpClient createHttpClient() {
        PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
        cm.setValidateAfterInactivity(1000);
        cm.setMaxTotal(200);
        cm.setDefaultMaxPerRoute(20);
        return HttpClients.custom()
                .setConnectionManager(cm)
                .evictExpiredConnections()
                .evictIdleConnections(10000, TimeUnit.MILLISECONDS)
                .setRetryHandler(DefaultHttpRequestRetryHandler.INSTANCE)
                .build();
    }

    @Bean
     public Keycloak customizedKeycloakAdminClient() {
        return KeycloakBuilder.builder()
                .grantType(OAuth2Constants.CLIENT_CREDENTIALS)
                .realm("admin")
                .clientId("clientId")
                .clientSecret("clientSecret")
                .serverUrl("http://127.0.0.1:8080")
                .resteasyClient(customizedResteasyClient())
                .build();
    }

I think that pooling settings like retry handler and idle connections eviction mechanism could cause that sockets used to connect through HTTP protocol should be tested before use and connections closed by some external mechanisms like firewall would be rejected by KeycloakAdmin client.