okhttp: Potential failure on CallServerInterceptor if writeRequestHeader fails

Unfortunately this bug is hard to replicate and to create a failing test. But I hope it still gets attention.

We’ve noticed that there is a big number of crashes in our customer base after upgrading from OkHttp 4.9.3 to version 4.12.0, as per stacktrace below.

We’ve also noticed a change in 4.11 could be related to that:

Read the response even if writing the request fails (https://github.com/square/okhttp/pull/6295) (https://github.com/square/okhttp/pull/7453)

https://github.com/square/okhttp/commit/e46a200fceaa5fbd9c9278eebe9c978399bd329c

It seems that a change has been introduced in CallServerInterceptor which moved the exchange.write* operations into a try/catch block (I suppose by design, to fix the issue as referenced in the commit).

However, the effect of that could be that the following block, the exchange.readResponseHeaders might be executed with state set to 0 (IDLE, not initalised), if the previous exchange.writeRequestHeaders(request) operation failed for any reason apart from ConnectionShutdownException. The readResponseHeaders has a guard statement which would throw an exception if state == 0.

I couldn’t find a way which could cause the exchange.writeRequestHeaders call to raise an exception, but we are observing many cases (it is our top crash, affecting thousands of customers in a couple of days after rollout).

It is important to note that we also updated from Okio 3.0.0 to 3.7.0 in the same release.

Stacktrace:

Exception java.lang.IllegalStateException: state: 0
  at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders (Http1ExchangeCodec.kt)
  at okhttp3.internal.connection.Exchange.readResponseHeaders (Exchange.kt)
  at okhttp3.internal.http.CallServerInterceptor.intercept (CallServerInterceptor.kt)
  at okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.kt)
  at okhttp3.internal.connection.ConnectInterceptor.intercept (ConnectInterceptor.kt)
  at okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.kt)
  at okhttp3.internal.cache.CacheInterceptor.intercept (CacheInterceptor.kt)
  at okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.kt)
  at okhttp3.internal.http.BridgeInterceptor.intercept (BridgeInterceptor.kt)
  at okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.kt)
  at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept (RetryAndFollowUpInterceptor.kt)
  at okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.kt)
  at my.company.SomeInterceptor 
  (...)
  at okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.kt)
  at okhttp3.internal.connection.RealCall.getREsponseWithInterceptorChain$okhttp (RealCall.kt)
  at okhttp3.internal.connection.RealCall$AsyncCall.run (RealCall.kt)
  at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:644)
  at java.lang.Thread.run (Thread.java:1012)

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 4
  • Comments: 15

Most upvoted comments

We haven’t been trying to catch the IllegalStateException so far, rather avoid it happening. Are you able to repro that specific condition?

My worry with catching the ISE, is that we don’t know what state we are in. Since we haven’t fixed this so far, we may have to apply this bandaid and remove it once we have fixed the ISE root cause.

I was able to reproduce this condition by:

  1. Starting web servers A and B
  2. Make the server A do a http GET request to the server B, which completes successfully
  3. Restart the server B
  4. Repeat the step 2, which will fail with the ISE

After the second step, server A keeps the connection open to server B, which is reset by server B after restart, thus throwing an IOException, which is caught by okhttp client and kept for rethrow until headers are read, but reading the headers fail due to state being 0. From what I noticed is that this only happens when GET requests are made, because when request has a body, even during connection reset by peer, the state is not 0 which succeeds at passing the state check, but when request doesn’t have a body, it fails on state check. I assume this is due to okhttp3.internal.http1.Http1ExchangeCodec.writeRequest changing the state from STATE_IDLE to STATE_OPEN_REQUEST_BODY at the end of the function, because during connection reset by peer, if the request has a body, it succeeds at changing the state, but when request doesn’t have a body, it fails at changing the state, thus throwing ISE at later state check for state still being STATE_IDLE.

Thanks, I’ll try to make a repro in the code. Really appreciate your pointers.

Any solution, we are experiencing this as well.

The same for me. I also receive this error. How I can fix this?

Probably a dupe of https://github.com/square/okhttp/issues/6853

I’ll take a look again on the weekend.