kubernetes-client: kubernetes-client could not work with java 8u252

@manusa I have verified it could work after export HTTP2_DISABLE=true. Great job.

I can confirm as well that the workaround seems to work for me both locally on Kube 1.18.2 and on OCP 4.2. Thanks for looking into this @manusa and finding the solution … great help for us!

Patched release with cherry- picked backport done: https://github.com/fabric8io/kubernetes-client/releases/tag/v4.9.2

Maven Central: https://repo1.maven.org/maven2/io/fabric8/kubernetes-client/4.9.2/

Patched release with cherry- picked backport done: https://github.com/fabric8io/kubernetes-client/releases/tag/v4.9.2

Maven Central: https://repo1.maven.org/maven2/io/fabric8/kubernetes-client/4.9.2/

Thank you @manusa !

Just started release process for v4.9.2

@manusa Thank for your commit #2227, solve my problme!

@wangyang0918 Hello, I can still see the problem when I execute the following command. $ ./bin/flink list --target kubernetes-application -Dkubernetes.cluster-id=my-first-application-cluster

Is this normal? Thanks for your help. I use the following Kubernetes configuration.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.10", GitCommit:"a84e568eeb56c4e3966314fc2d58374febd12ed7", GitTreeState:"clean", BuildDate:"2021-03-09T14:35:22Z", GoVersion:"go1.13.15 BoringCrypto", Compiler:"gc", Platform:"linux/amd64"}

I use Java 11.0.8 on my laptop. I am using flink:1.12.1-scala_2.11-java11 as base image.

------------------------------------------------------------
 The program finished with the following exception:

io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [Service]  with name: [my-first-application-cluster-rest]  in namespace: [default]  failed.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:225)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:168)
	at org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.getRestService(Fabric8FlinkKubeClient.java:225)
	at org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.getRestEndpoint(Fabric8FlinkKubeClient.java:166)
	at org.apache.flink.kubernetes.KubernetesClusterDescriptor.lambda$createClusterClientProvider$0(KubernetesClusterDescriptor.java:96)
	at org.apache.flink.kubernetes.KubernetesClusterDescriptor.retrieve(KubernetesClusterDescriptor.java:133)
	at org.apache.flink.kubernetes.KubernetesClusterDescriptor.retrieve(KubernetesClusterDescriptor.java:66)
	at org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:1005)
	at org.apache.flink.client.cli.CliFrontend.list(CliFrontend.java:431)
	at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1064)
	at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1136)
	at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1136)
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
	at java.base/java.net.Socket.connect(Socket.java:608)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.platform.Platform.connectSocket(Platform.java:129)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:246)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.RealConnection.connect(RealConnection.java:166)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:134)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$3(HttpClientUtils.java:109)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at org.apache.flink.kubernetes.shaded.okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:254)
	at org.apache.flink.kubernetes.shaded.okhttp3.RealCall.execute(RealCall.java:92)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:469)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:395)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:376)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:845)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:214)
	... 12 more

is there a release planned with this #2227 fix? Upgrade to okhttp 3.12.11 didn’t help me.

4.10.2

Cause

The issue is related with the auto-selected Protocol used for each platform.

When using JDK 8u252 the selected Protocol (in RealConnection) is http/2 which as it seems is not compatible with this Platform (exception is thrown in the invoked startHttp2(pingIntervalMillis) method).

When using JDK 8u233 (<u252) the selected Protocol (in RealConnection) is http/1.1which works fine.

When using JDK 11 the selected Protocol is http/2 but startHttp2(pingIntervalMillis) actually works fine.

Description

When using > JDK 8u251, OkHttp “wrongly” detects the Platform as Jdk9Platform. This in turn enables Application-Layer Protocol Negotiation (ALPN) support which is not available out-of-the-box for JDK 8, thus enabling http/2protocol which OkHttp then selects (as it is the most efficient).

In order to prevent this we need to force http/1.0 protocol whenever JDK 8 is detected during runtime (which would be the default anyway). I will submit a PR with the fix.

Workaround for <= v4.10.1

Set Config#http2Disable (http2.disable system property // HTTP2_DISABLE environment variable) true.

So do we have a concrete plan for v4.9.2?

We’re finishing with the fixes for bugs reported in 4.10.1, we’ll probably release 4.10.2 in the next few days. This patched version should take care of any bugs and should be a direct replacement for 4.9.1.

If after 4.10.2 release there are still some issues/bug, or if you find some strong reason (besides our previous recommendation) to not upgrade version, we can then release a back-ported patched 4.9.2.

So do we have a concrete plan for v4.9.2?

Thx for your feedback.

We are using the latest releases of OkHttp with your fixes ❤️.

The problem is that sun.security.ssl.SSLSocketImpl is never throwing the UnsupportedOperationException in SSLSocketImpl.java, so that is why HTTP/2 becomes available.

You can check the below screenshot form a debugging session in OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_252-b09)

Anyway, we managed to fix the issue by forcing http/1.1 from our side when JDK 8 is detected (in a rudimentary way), but I’m not sure that other users of OkHttp will face a similar issue.

This was fixed in 3.12.11 and 4.6.0 a couple of weeks ago.

https://square.github.io/okhttp/changelog/ https://square.github.io/okhttp/changelog_3x/

Fix: Don’t crash on Java 8u252 which introduces an API previously found only on Java 9 and above. See Jetty’s overview of the API change and its consequences.

@manusa Thanks a lot for the investigation and fix. Could we also backport this to branch 4.9? We are going to upgrade the fabric8 kubernetes-client version to 4.9.x as @rohanKanojia 's suggestion.

ah, We recently split the Kubernetes Model into smaller jars per apiVersion in #2137 . But it has introduced some regressions #2195, #2201 and #2205

If you’re not affected by these bugs, you can upgrade. We have plans on fixing them and then cutting a stable release.