kubernetes-client: fabric8.kubernetes.client.KubernetesClientException - Detection of Kubernetes version failed.

K8S: v1.18

Strimzi Kafka Operator was v0.18 - operational Kafka Cluster v2.5.0 - operational

In order for us to get to Kafka cluster 2.7.0, we first need to upgrade operator to v0.22 though came across problem.

Upgraded operator from 0.18 to 0.22 and got following:

io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.ClusterOperationsImpl.fetchVersion(ClusterOperationsImpl.java:54) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
        at io.fabric8.kubernetes.client.DefaultKubernetesClient.getVersion(DefaultKubernetesClient.java:489) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
        at io.strimzi.operator.PlatformFeaturesAvailability.lambda$getVersionInfoFromKubernetes$5(PlatformFeaturesAvailability.java:150) ~[io.strimzi.operator-common-0.22.0.jar:0.22.0]
        at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
        at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: okhttp3.internal.http2.ConnectionShutdownException

This could have been caused by different number of reasons (not necessarily client release) though most obvious ones we ruled out for following reasons:

  1. We had successfully upgraded operator from 0.15 to 0.18 and did not experience a problem with K8S client is not able to reach K8S api;
  2. We did some manual tests / checks to ensure there was nothing preventing client from reaching K8S api
  3. [most critical one]: we set HTTP2_DISASBLE: “true” and strimzi operator passed through the point it used to fail. POD actually comes up and before it would cyclic crash/loop. (there are other errors in the logs but related to kafka clusters and objects we have - nothing related to the issue herein reported).

Could this be a problem with lib/java version used by the K8S client? Asking that because there were 2-3 previously reported issues (java8, if not mistaken) with similar symptom and also “addressed” by this very same work-around.

If you need additional logs/information, please let us know. All we need to do to reproduce the problem is remove env variable passed (HTTP2_DISABLE) as mentioned above. Just wanted some help as to further troubleshoot this problem.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 15 (7 by maintainers)

Most upvoted comments

@manusa i know what is going on. Check this out…

  1. Based on -> https://www.ibm.com/docs/en/zos/2.3.0?topic=programming-cipher-suite-definitions , 9d => TLS_RSA_WITH_AES_256_GCM_SHA384 which based on -> https://httpwg.org/specs/rfc7540.html#BadCipherSuites is considered weak and should not be proposed.
  2. We have “forced” our K8S 1.18/API-Server to accept this specific cipher and problem disappeared as expected. In our clusters, for security reasons, we do not allow weak ciphers (TLS v1.2) that have been blacklisted.

Conclusions:

  • client (fabric8io) shall not propose, by default, such cipher - the default option should be any cipher that is not weak/blacklisted like the currently used 9d - see above link.
  • the Strimzi/operator needs to either handle that somehow (we opened bug report with them as well) OR place a requirement to have client properly handle that. Ideally client should pick a cipher that by default has not been blacklisted.
  • Anyone that has such combination strimzi kafka operator v0.22/K8S 1.18 and works is because api-server is not properly configured [that is -> allows a weak/black listed cipher to be used]

for reference: have opened this on Strimzi/Kafka: https://github.com/strimzi/strimzi-kafka-operator/issues/5044