okhttp: NoSuchElementException coming from RouteSelector after upgrading 3.12 -> 3.14

After upgrading our android app from okhttp 3.12.2 to 3.14.0, we started seeing this error (stacktrace below) coming through Crashlytics in production at a very high rate. Because we upgraded directly from 3.12 to 3.14, I’m not sure if this error was introduced in 3.13 or 3.14.

With an admittedly small sample size, the crash was affecting over 10% of sessions and over 20% of users, so we were forced to roll back. It appeared across Android versions 7, 8, 9, and Q.

Unfortunately, I wasn’t able to reproduce it locally. However, it seems very similar to this issue from 2 years ago, which appears to have been re-introduced to the library: https://github.com/square/okhttp/issues/3308

I am not sure if this is related but we are compiling our app with R8 version 1.4.72

Our app uses Java 8, min SDK 21, and the latest version of retrofit (2.5.0).

Hopefully this information and the stack trace below is enough to be helpful, apologies for not being able to include a reproducible case outside of production.

Thank you for the amazing library and everything you do for android open source!

Fatal Exception: java.util.NoSuchElementException
       at okhttp3.internal.connection.RouteSelector.next(RouteSelector.java:75)
       at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:187)
       at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:107)
       at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:87)
       at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:162)
       at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
       at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
       at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
       at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
       at com.redacted.myapp.MyNetworkingModule.doRequestWithExceptionCatching(MyNetworkingModule.java:150)
       at com.redacted.myapp.MyNetworkingModule.lambda$provideOkhttpClient$0(MyNetworkingModule.java:99)
       at com.redacted.myapp.-$$Lambda$MyNetworkingModule$f9YhFuxMEYBUFTv_jyu3YSQWZX8.intercept(-.java:2)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
       at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:221)
       at okhttp3.RealCall$AsyncCall.execute(RealCall.java:172)
       at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
       at java.lang.Thread.run(Thread.java:764)

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 13
Comments: 36 (11 by maintainers)

Commits related to this issue

Don't crash on retry when there are no more routes (3.x branch) https://github.com/square/okhttp/issues/4875 — committed to square/okhttp by swankjesse 5 years ago
Don't crash on retry when there are no more routes (3.x branch) https://github.com/square/okhttp/issues/4875 — committed to square/okhttp by swankjesse 5 years ago
Don't crash on retry when there are no more routes (4.x branch) https://github.com/square/okhttp/issues/4875 — committed to square/okhttp by swankjesse 5 years ago
* http: revert okHTTP to 3.12.2 due to https://github.com/square/okhttp/issues/4875 — committed to neowu/core-ng-project by neowu 5 years ago
Reproduce NoSuchElement exception in RouteSelector.next() in a test I was unable to write a test that reproduces https://github.com/square/okhttp/issues/4875 consistently, but this is the closest I g... — committed to amirlivneh/okhttp by amirlivneh 5 years ago
Reproduce NoSuchElement exception in RouteSelector.next() in a test I was unable to write a test that reproduces https://github.com/square/okhttp/issues/4875 consistently, but this is the closest I g... — committed to amirlivneh/okhttp by amirlivneh 5 years ago
Reproduce NoSuchElementException in RouteSelector.next() in a test I was unable to write a test that reproduces https://github.com/square/okhttp/issues/4875 consistently, but this is the closest I go... — committed to amirlivneh/okhttp by amirlivneh 5 years ago
Reproduce NoSuchElementException in RouteSelector.next() in a test I was unable to write a test that reproduces https://github.com/square/okhttp/issues/4875 consistently, but this is the closest I go... — committed to amirlivneh/okhttp by amirlivneh 5 years ago
Reproduce NoSuchElementException in RouteSelector.next() in a test I was unable to write a test that reproduces https://github.com/square/okhttp/issues/4875 consistently, but this is the closest I go... — committed to square/okhttp by amirlivneh 5 years ago
Lock in a route in hasRouteToTry() Otherwise we end up making multiple calls to retryCurrentRoute() and if it returns true then false we can fail to find a route on a retry. Closes: https://github.c... — committed to square/okhttp by swankjesse 5 years ago
Lock in a route in hasRouteToTry() (3.14.x branch) Otherwise we end up making multiple calls to retryCurrentRoute() and if it returns true then false we can fail to find a route on a retry. Closes: ... — committed to square/okhttp by swankjesse 5 years ago
Lock in a route in hasRouteToTry() Otherwise we end up making multiple calls to retryCurrentRoute() and if it returns true then false we can fail to find a route on a retry. Closes: https://github.c... — committed to square/okhttp by swankjesse 5 years ago
Lock in a route in hasRouteToTry() Otherwise we end up making multiple calls to retryCurrentRoute() and if it returns true then false we can fail to find a route on a retry. Closes: https://github.c... — committed to square/okhttp by swankjesse 5 years ago
Reproduce NoSuchElementException in RouteSelector.next() in a test I was unable to write a test that reproduces https://github.com/square/okhttp/issues/4875 consistently, but this is the closest I go... — committed to SeniorZhai/okhttp by amirlivneh 5 years ago
Lock in a route in hasRouteToTry() (3.14.x branch) Otherwise we end up making multiple calls to retryCurrentRoute() and if it returns true then false we can fail to find a route on a retry. Closes: ... — committed to SeniorZhai/okhttp by swankjesse 5 years ago

Most upvoted comments

3.14.2 is out and has this fix. Go get it!

swankjesse on May 19, 2019

Just saw this in 3.14.2

java.util.NoSuchElementException: 
	at okhttp3.internal.connection.RouteSelector.next()(RouteSelector.java:15)
	at okhttp3.internal.connection.ExchangeFinder.findConnection()(ExchangeFinder.java:21)
	at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection()(ExchangeFinder.java:1)
	at okhttp3.internal.connection.ExchangeFinder.find()(ExchangeFinder.java:6)
	at okhttp3.internal.connection.Transmitter.newExchange()(Transmitter.java:5)
	at okhttp3.internal.connection.ConnectInterceptor.intercept()(ConnectInterceptor.java:5)
	at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:10)
	at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:1)
	at okhttp3.internal.cache.CacheInterceptor.intercept()(CacheInterceptor.java:22)
	at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:10)
	at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:1)
	at okhttp3.internal.http.BridgeInterceptor.intercept()(BridgeInterceptor.java:22)
	at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:10)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept()(RetryAndFollowUpInterceptor.java:6)
	at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:10)
	at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:1)
	at com.redacted.myapp.networking.CloudinaryWebPInterceptor.intercept()(CloudinaryWebPInterceptor.java:21)
	at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:10)
	at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:1)
	at com.redacted.myapp.interceptors.DaliRedirectInterceptor.intercept()(DaliRedirectInterceptor.java:26)
	at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:10)
	at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:1)
	at com.redacted.myapp.shared.features.common.utils.image.DaliCompatInterceptor.intercept()(DaliCompatInterceptor.java:1)
	at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:10)
	at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:1)
	at com.redacted.myapp.core.interceptors.UserAgentInterceptor.intercept()(UserAgentInterceptor.java:22)
	at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:10)
	at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:1)
	at okhttp3.RealCall.getResponseWithInterceptorChain()(RealCall.java:13)
	at okhttp3.RealCall$AsyncCall.execute()(RealCall.java:2)
	at okhttp3.internal.NamedRunnable.run()(NamedRunnable.java:3)
	at java.util.concurrent.ThreadPoolExecutor.runWorker()(ThreadPoolExecutor.java:1133)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run()(ThreadPoolExecutor.java:607)
	at java.lang.Thread.run()(Thread.java:761)

AndrewWestberg on Jun 10, 2019

We are still having the same issue with 3.14.1, retrofit 2.5.0, minSdk 21

Fatal Exception: java.util.NoSuchElementException
       at okhttp3.internal.connection.RouteSelector.next(RouteSelector.java:75)
       at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:187)
       at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:107)
       at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:87)
       at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
       at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
       at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
       at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
       at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
       at it.subito.networking.utils.TimingLoggerInterceptor.intercept(TimingLoggerInterceptor.java:26)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
       at it.subito.networking.okhttpclient.interceptors.HeaderInterceptor.intercept(HeaderInterceptor.java:49)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
       at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:221)
       at okhttp3.RealCall$AsyncCall.execute(RealCall.java:172)
       at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
       at java.lang.Thread.run(Thread.java:818)

danielebart on Apr 23, 2019

@leofirespotter we were able to nail down an instance with an executable test case and made a fix in 4.1.0. Please give it a shot if you can.

dave-r12 on Aug 16, 2019

Yep, I see how we made this mistake.

@yschimke good idea. This plus the other Exchange regression motivates us to make an okhttp_3.14.1 release. Folks who are (unreasonably!) grumpy about Kotlin should still have a rock-solid 3.x branch to ship.

swankjesse on Apr 6, 2019

Ignore: misunderstood the point of the check

The hasNext() and next() are on different objects, which looks weird.


    boolean newRouteSelection = false;
    if (selectedRoute == null && (routeSelection == null || !routeSelection.hasNext())) {
      newRouteSelection = true;
      routeSelection = routeSelector.next();
    }

FWIW - I’m not actively fixing this, so just flagging for swankjesse or whoever picks it up

yschimke on Apr 5, 2019

I reproduced the exception in an improvised test where a connection is shutdown prematurely:

After a SocketException is thrown, RetryAndFollowUpInterceptor decides to retry because ExchangeFinder.retryCurrentRoute() and ExchangeFinder.hasRouteToTry() return true: https://github.com/square/okhttp/blob/master/okhttp/src/main/java/okhttp3/internal/connection/Transmitter.java#L313
Immediately after that, before ExchangeFinder.findConnection() is reached, a ConnectionShutdownException is tracked on the Connection and increments Connection.routeFailureCount.
ExchangeFinder.findConnection() decides to try a new route because retryCurrentRoute() now returns false due to Connection.routeFailureCount > 0, but there is no other route: https://github.com/square/okhttp/blob/master/okhttp/src/main/java/okhttp3/internal/connection/ExchangeFinder.java#L143

amirlivneh on May 3, 2019

Sorry, i have no idea. All i can say is that this is the top crash in our latest build and we didn’t have this crash before updating to the 3.14.1 (from 3.12). I’ll keep you posted if something comes up, thank you!

danielebart on Apr 29, 2019