okhttp: OkHttp and Retrofit throws javax.net.ssl.SSLException until restart

I have an android app in kotlin that talks to an express nodeJS api via mutual authentication. The app is designed to keep collecting data after the screen is off/on, survive reboots, etc. This is done with the user’s consent.

This is occuring on an Android 8.1 Samsung tablet that has no other apps installed (other than the standard samsung installs)

After some time of the screen being off, (Time until the error shows up ranges from 20 minutes to 3 hours) we get the below exception until we restart (We left it running in this error state for >8 hours, and it never recovered) the app or recreate retrofit object. The foreground service is still running and collecting data without an issue.

The issue is not a server problem, I had a shell script curl the server with the same parameters and that worked for ~16 hours.

We have tried:

• Setting client header ‘Connection’, ‘close’
• Setting server header ‘Connection’, ‘close’
• both of the above
• Setting client header ‘Connection’, ‘Keep-Alive’
• Setting server header ‘Connection’, ‘Keep-Alive’
• Both of the above
• Combinations of all of the above
• Checking for a server performance issue
• Making body smaller (Current limit is 100 items, which translates to a ~120kb body)
• Some other things I am forgetting

Workaround: When this exception is thrown, we create a new retrofit service, drain all connectionss, then resume posting to the API.

Versions used:

implementation 'com.squareup.retrofit2:retrofit:2.6.1'
implementation 'com.squareup.retrofit2:converter-gson:2.6.1'
implementation 'com.squareup.okhttp3:okhttp:4.2.0'
implementation 'com.squareup.okhttp3:logging-interceptor:4.2.0'
implementation 'com.squareup.okhttp3:okhttp-brotli:4.2.0'

Stack Trace:

 javax.net.ssl.SSLException: Read error: ssl=0xa65d7940: I/O error during system call, Connection reset by peer
        at com.android.org.conscrypt.NativeCrypto.SSL_read(Native Method)
        at com.android.org.conscrypt.SslWrapper.read(SslWrapper.java:391)
        at com.android.org.conscrypt.ConscryptFileDescriptorSocket$SSLInputStream.read(ConscryptFileDescriptorSocket.java:567)
        at okio.InputStreamSource.read(Okio.kt:102)
        at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:159)
        at okio.RealBufferedSource.indexOf(RealBufferedSource.kt:349)
        at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:222)
        at okhttp3.internal.http1.Http1ExchangeCodec.readHeaderLine(Http1ExchangeCodec.kt:210)
        at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:181)
        at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:105)
        at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:82)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:37)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:82)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:84)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:71)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
        at com.someCompany.someApp.api.RetrofitFactory$logoutInterceptor$$inlined$invoke$1.intercept(Interceptor.kt:75)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
        at okhttp3.brotli.BrotliInterceptor.intercept(BrotliInterceptor.kt:39)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
        at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.kt:215)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
        at com.someCompany.someApp.api.RetrofitFactory$headersInterceptor$$inlined$invoke$1.intercept(Interceptor.kt:75)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.kt:184)
        at okhttp3.RealCall$AsyncCall.run(RealCall.kt:136)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
        at java.lang.Thread.run(Thread.java:764)

What the server sees:

BadRequestError: request aborted 
 at IncomingMessage.onAborted (/app/node_modules/raw-body/index.js:231:10) 
 at emitNone (events.js:106:13) 
 at IncomingMessage.emit (events.js:208:7)
 at abortIncoming (_http_server.js:445:9) 
 at socketOnClose (_http_server.js:438:3) 
 at emitOne (events.js:121:20) 
 at TLSSocket.emit (events.js:211:7) 
 at _handle.close (net.js:561:12) 
 at Socket.done (_tls_wrap.js:360:7) 
 at Object.onceWrapper (events.js:315:30) 
 at emitOne (events.js:116:13) 
 at Socket.emit (events.js:211:7) 
 at TCP._handle.close [as _onclose] (net.js:561:12) 

RetrofitFactory:

object RetrofitFactory {
    fun makeRetrofitService(context: Context, isAuthenticated: Boolean = false): RetrofitService {
        val baseUrl: String = if (BuildConfig.DEBUG) {
            "https://api.staging.someCompany.com/"
        } else {
            "https://api-api.someCompany.com/"
        }
        return Retrofit.Builder().baseUrl(baseUrl)
            .addConverterFactory(GsonConverterFactory.create(makeGson()))
            .client(makeClient(context, isAuthenticated))
            .build().create(RetrofitService::class.java)
    }

    private fun makeGson(): Gson {
        return GsonBuilder().excludeFieldsWithModifiers(Modifier.TRANSIENT).create()
    }
    fun makeClient(context: Context, isAuthenticated: Boolean): OkHttpClient {
        val hostnameVerifier = HostnameVerifier { hostname, _ ->
            HttpsURLConnection.getDefaultHostnameVerifier().run {
                if (BuildConfig.DEBUG) {
                    hostname == "relay-api.staging.someCompany.com"
                } else {
                    hostname == "relay-api.someCompany.com"
                }
            }
        }

        val sslAndMgr = if (isAuthenticated) {
            clientAuthSslContext(context)
        } else {
            basicSslContext(context)
        }

        return OkHttpClient.Builder()
            .connectTimeout(15, TimeUnit.SECONDS)
            .readTimeout(15, TimeUnit.SECONDS)
            .sslSocketFactory(sslAndMgr.first, sslAndMgr.second)
            .addInterceptor(headersInterceptor()).addInterceptor(loggingInterceptor())
            .addInterceptor(BrotliInterceptor)
            .addInterceptor(logoutInterceptor())
            .hostnameVerifier(hostnameVerifier)
            .retryOnConnectionFailure(true)
            .build()
    }
    private fun loggingInterceptor() = HttpLoggingInterceptor().apply {
        level =
            if (BuildConfig.DEBUG) HttpLoggingInterceptor.Level.BODY else HttpLoggingInterceptor.Level.NONE
    }


    private fun logoutInterceptor() = Interceptor { chain ->
        val mainResponse = chain.proceed(chain.request())

        if (mainResponse.code == 401) {
            Certificates(someApp().applicationContext).logout()
        }
        mainResponse
    }
    private fun headersInterceptor() = Interceptor { chain ->
        chain.proceed(
            (chain.request().newBuilder()
                .addHeader("Accept", "application/json")
                .addHeader("Accept-Language", "en")
                .addHeader("Content-Type", "application/json").build()
                    )
        )
    }

    private fun basicSslContext(context: Context): Pair<SSLSocketFactory, X509TrustManager> {
        val certMgr = Certificates(context)
        val keyStore = KeyStore.getInstance(KeyStore.getDefaultType())
        keyStore.load(null, null)

        keyStore.setCertificateEntry("serverCert", certMgr.loadServerChain())
        val trustMgrFactory =
            TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm())
        trustMgrFactory.init(keyStore)

        val sslContext = SSLContext.getInstance("TLS")
        sslContext.init(null, trustMgrFactory.trustManagers, null)
        return Pair(sslContext.socketFactory, trustMgrFactory.trustManagers[0] as X509TrustManager)
    }

    private fun clientAuthSslContext(context: Context): Pair<SSLSocketFactory, X509TrustManager> {
        val certMgr = Certificates(context) //This is a helper that loads certificates/PKs as needed

        val keyStore = KeyStore.getInstance(KeyStore.getDefaultType())
        keyStore.load(null, null)
        keyStore.setCertificateEntry("ca", certMgr.loadServerChain())
        keyStore.setCertificateEntry("client", certMgr.loadClientCert())
        keyStore.setKeyEntry(
            "private",
            certMgr.loadPrivateKey(),
            null,
            arrayOf(certMgr.loadClientCert(), certMgr.loadCA())
        )

        val kmf: KeyManagerFactory = KeyManagerFactory.getInstance("X509")

        kmf.init(keyStore, null)

        val trustMgrFactory =
            TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm())
        trustMgrFactory.init(keyStore)
        val sslContext = SSLContext.getInstance("TLS")
        sslContext.init(kmf.keyManagers, trustMgrFactory.trustManagers, null)

        return Pair(object : DelegatingSSLSocketFactory(sslContext.socketFactory) { //DelegatingSSLSocketFactory from https://github.com/square/okhttp/blob/master/okhttp/src/test/java/okhttp3/DelegatingSSLSocketFactory.java 
            @Throws(
                IOException::class
            )
            override fun configureSocket(sslSocket: SSLSocket): SSLSocket {
                sslSocket.sslParameters.needClientAuth = true
                sslSocket.needClientAuth = true
                return super.configureSocket(sslSocket)
            }
        }, trustMgrFactory.trustManagers[0] as X509TrustManager)
    }
}

Where we actually call the api:

val res = CoroutineScope(Dispatchers.IO).launch {
            try {

                //call Room DB and get list of items out

                val body = BulkRelayRequest(items.toTypedArray())
                val response = apiService.postBulkRelayLogs(body)

                if (response.isSuccessful) {
                   //update room DB
                    Log.i(TAG, "Posted ***** with ids ${response.body()!!.payload.ids}")
                } else {
                    //failure :(
                    val eParams = Bundle()
                    firebaseAnalytics.logEvent("API_SEND_FAILED", eParams)
                    Log.e(TAG, "Error: failed to post ****** to API")
                }
            } catch(e: SSLException)
            {
                isDraining = true
                apiService = RetrofitFactory.makeRetrofitService(context, true)
                Log.e(TAG, "SSL Error occurred. Draining connections and restarting retrofit")
            }
            catch (e: Throwable) {
                val eParams = Bundle()
                firebaseAnalytics.logEvent("API_SEND_FAILED", eParams)
                Log.e(TAG, "Error: failed to post *** to API and threw an exception", e)
            }
            finally {
                connectionsActive--
            }
        }
    res.ensureActive()