strimzi-kafka-operator: PKIX errors due to certs not renewing

Discussed in https://github.com/strimzi/strimzi-kafka-operator/discussions/6103

<div type='discussions-op-text'>

Originally posted by talonx December 24, 2021 We are running several Kafka clusters managed by operator (v 0.20.0). We don’t have any custom TLS certs. A few days ago we started seeing multiple active controllers in one of our clusters, and the problem seems to boil down to the fact that two brokers (it’s a 3 broker cluster) are not able to talk to each other. The error is

{"exception":{"stacktrace":"org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed\nCaused by: javax.net.ssl.SSLProtocolException: Unexpected handshake message: server_hello\n\tat java.base\/sun.security.ssl.Alert.createSSLException(Alert.java:129)\n\tat java.base\/sun.security.ssl.Alert.createSSLException(Alert.java:117)\n\tat java.base\/sun.security.ssl.TransportContext.fatal(TransportContext.java:313)\n\tat java.base\/sun.security.ssl.TransportContext.fatal(TransportContext.java:269)\n\tat java.base\/sun.security.ssl.TransportContext.fatal(TransportContext.java:260)\n\tat java.base\/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:438)\n\tat java.base\/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)\n\tat java.base\/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)\n\tat java.base\/java.security.AccessController.doPrivileged(Native Method)\n\tat java.base\/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)\n\tat org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:425)\n\tat org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:509)\n\tat org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:363)\n\tat org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:286)\n\tat org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:174)\n\tat org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547)\n\tat org.apache.kafka.common.network.Selector.poll(Selector.java:485)\n\tat org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:549)\n\tat org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:74)\n\tat kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:288)\n\tat kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:242)\n\tat kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)","exception_class":"org.apache.kafka.common.errors.SslAuthenticationException","exception_message":"SSL handshake failed"},"source_host":"x-platform-kafka-0","method":"warn","level":"WARN","message":"[RequestSendThread controllerId=0] Controller 0's connection to broker x-platform-kafka-1.x-platform-kafka-brokers.kafka.svc:9091 (id: 1 rack: null) was unsuccessful","mdc":{},"@timestamp":"2021-12-23T18:39:14.654Z","file":"Logging.scala","line_number":"72","thread_name":"Controller-0-to-broker-1-send-thread","@version":1,"logger_name":"kafka.controller.RequestSendThread","class":"kafka.utils.Logging"}

I am assuming the operator should be updating the certificates automatically. What could be going wrong here?</div>

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 25 (10 by maintainers)

Most upvoted comments

@scholzj you may find this entertaining, we have somewhat done the impossible and got the certs to generate through a really atypical approach, compile the thing ourselves and re-order the reconcile loop. For anyone running into this issue and you are on 0.20.0+, we can confidently say that cert generation works JUST fine and you are most likely running into the operator not running because of a Kubernetes incompatibility or something along those lines. CHECK your operator logs FIRST.

For anyone else in this situation, I recommend you follow the approach below, we had about 20hrs of downtime and this is basically the only solution. I’m going to open a new issue with my thoughts on the operator, specifically with its order of its reconcile loop (I think it could be better, put critical reconcile loop things like cert-generation FIRST.)

Clone the repo, switch to the broken version

  1. We cloned the repo git clone git@github.com:strimzi/strimzi-kafka-operator.git
  2. We switched to the tag git checkout 0.20.0 OR specifically your version

Modify the reconcile loop

The reconcile loop determines what the operator is going to do next and in our case, the PodDisruption part failed to run which means all other steps in the reconcile loop fail to run (including the cert generation)

  1. OPEN /cluster-operator/src/main/java/io/strimzi/operator/cluster/operator/assembly/KafkaAssemblyOperator.java
  2. Modify the loop, here is the order of ours, we commented out .compose(state -> state.zkPodDisruptionBudget()) and moved all the cert generation/secret generation HIGHER than Ingress and PodDisruptionBudget which BOTH use v1beta1 making the operator fail
    Future<Void> reconcile(ReconciliationState reconcileState)  {
        Promise<Void> chainPromise = Promise.promise();

        reconcileState.initialStatus()
                .compose(state -> state.reconcileCas(this::dateSupplier))
                .compose(state -> state.clusterOperatorSecret(this::dateSupplier))
                .compose(state -> state.getKafkaClusterDescription())
                // Roll everything if a new CA is added to the trust store.
                .compose(state -> state.rollingUpdateForNewCaKey())
                .compose(state -> state.getZookeeperDescription())
                .compose(state -> state.zkModelWarnings())
                .compose(state -> state.zkManualPodCleaning())
                .compose(state -> state.zkNetPolicy())
                .compose(state -> state.zkManualRollingUpdate())
                .compose(state -> state.zkVersionChange())
                .compose(state -> state.zookeeperServiceAccount())
                .compose(state -> state.zkPvcs())
                .compose(state -> state.zkService())
                .compose(state -> state.zkHeadlessService())
                .compose(state -> state.zkAncillaryCm())
                .compose(state -> state.zkNodesSecret(this::dateSupplier))
                // .compose(state -> state.zkPodDisruptionBudget())
                .compose(state -> state.zkStatefulSet())
                .compose(state -> state.zkScalingDown())
                .compose(state -> state.zkRollingUpdate())
                .compose(state -> state.zkPodsReady())
                .compose(state -> state.zkScalingUp())
                .compose(state -> state.zkScalingCheck())
                .compose(state -> state.zkServiceEndpointReadiness())
                .compose(state -> state.zkHeadlessServiceEndpointReadiness())
                .compose(state -> state.zkPersistentClaimDeletion())

                .compose(state -> state.checkKafkaSpec())
                .compose(state -> state.kafkaModelWarnings())
                .compose(state -> state.kafkaManualPodCleaning())
                .compose(state -> state.kafkaNetPolicy())
                .compose(state -> state.kafkaManualRollingUpdate())
                .compose(state -> state.kafkaVersionChangeCheck())
                .compose(state -> state.kafkaVersionChange(false))
                .compose(state -> state.kafkaPvcs())
                .compose(state -> state.kafkaInitServiceAccount())
                .compose(state -> state.kafkaInitClusterRoleBinding())
                .compose(state -> state.kafkaScaleDown())
                .compose(state -> state.kafkaServices())
                .compose(state -> state.kafkaRoutes())
                .compose(state -> state.kafkaGenerateCertificates(this::dateSupplier))
                .compose(state -> state.kafkaBrokersSecret())
                .compose(state -> state.entityOperatorSecret(this::dateSupplier))
                .compose(state -> state.customListenerCertificates())
                .compose(state -> state.kafkaIngresses())
                .compose(state -> state.kafkaInternalServicesReady())
                .compose(state -> state.kafkaLoadBalancerServicesReady())
                .compose(state -> state.kafkaNodePortServicesReady())
                .compose(state -> state.kafkaRoutesReady())
                .compose(state -> state.kafkaIngressesReady())
                .compose(state -> state.kafkaAncillaryCm())
                .compose(state -> state.kafkaJmxSecret())
                .compose(state -> state.kafkaPodDisruptionBudget())
                .compose(state -> state.kafkaStatefulSet())
                .compose(state -> state.kafkaVersionChange(true))
                .compose(state -> state.kafkaRollingUpdate())
                .compose(state -> state.kafkaScaleUp())
                .compose(state -> state.kafkaPodsReady())
                .compose(state -> state.kafkaServiceEndpointReady())
                .compose(state -> state.kafkaHeadlessServiceEndpointReady())
                .compose(state -> state.kafkaPersistentClaimDeletion())
                // This has to run after all possible rolling updates which might move the pods to different nodes
                .compose(state -> state.kafkaNodePortExternalListenerStatus())
                .compose(state -> state.kafkaCustomCertificatesToStatus())

                .compose(state -> state.checkUnsupportedTopicOperator())

                .compose(state -> state.getEntityOperatorDescription())
                .compose(state -> state.entityOperatorServiceAccount())
                .compose(state -> state.entityOperatorTopicOpRoleBinding())
                .compose(state -> state.entityOperatorUserOpRoleBinding())
                .compose(state -> state.entityOperatorTopicOpAncillaryCm())
                .compose(state -> state.entityOperatorUserOpAncillaryCm())
                .compose(state -> state.entityOperatorDeployment())
                .compose(state -> state.entityOperatorReady())

                .compose(state -> state.getCruiseControlDescription())
                .compose(state -> state.cruiseControlNetPolicy())
                .compose(state -> state.cruiseControlServiceAccount())
                .compose(state -> state.cruiseControlAncillaryCm())
                .compose(state -> state.cruiseControlSecret(this::dateSupplier))
                .compose(state -> state.cruiseControlDeployment())
                .compose(state -> state.cruiseControlService())
                .compose(state -> state.cruiseControlReady())

                .compose(state -> state.getKafkaExporterDescription())
                .compose(state -> state.kafkaExporterServiceAccount())
                .compose(state -> state.kafkaExporterSecret(this::dateSupplier))
                .compose(state -> state.kafkaExporterDeployment())
                .compose(state -> state.kafkaExporterReady())

                .compose(state -> state.getJmxTransDescription())
                .compose(state -> state.jmxTransServiceAccount())
                .compose(state -> state.jmxTransConfigMap())
                .compose(state -> state.jmxTransDeployment())
                .compose(state -> state.jmxTransDeploymentReady())

                .map((Void) null)
                .onComplete(chainPromise);

        return chainPromise.future();
    }

Add logs everywhere

Add logs below, you can sprinkle them wherever cert generation happens. You can scroll ABOVE to follow the code paths that are responsible for cert generation

  1. /cluster-operator/src/main/java/io/strimzi/operator/cluster/model/ClusterCa.java
  2. /cluster-operator/src/main/java/io/strimzi/operator/cluster/operator/assembly/KafkaAssemblyOperator.java
  3. /operator-common/src/main/java/io/strimzi/operator/cluster/model/Ca.java

Building

  1. Follow the guide https://github.com/strimzi/strimzi-kafka-operator/blob/main/development-docs/DEV_GUIDE.md to building the operator.

BUT there are edge cases:

  1. api/pom.xml uses an old version of org.projectlombok which fails to compile, open api/pom.xml and update:
         <dependency>
             <groupId>org.projectlombok</groupId>
             <artifactId>lombok</artifactId>
-            <version>1.18.4</version>
+            <version>1.18.26</version>
  1. The yq version that 0.20.0 and maybe other builds expect is NOT 4.x and will cause the build to fail. You need yq version 3.3.2

Get the version here https://github.com/mikefarah/yq/releases (search to 3.3.2)

  1. If your prerequisites-check.sh fails on yq, make sure that yq is using 3.3.2

  2. Update Makefile.docker to use a linux arch:

 docker_build_default:
        # Build Docker image ...
-       $(DOCKER_CMD) build $(DOCKER_BUILD_ARGS) --build-arg strimzi_version=$(RELEASE_VERSION) -t strimzi/$(PROJECT_NAME):latest $(DOCKERFILE_DIR)
+       $(DOCKER_CMD) buildx build $(DOCKER_BUILD_ARGS) --platform=linux/amd64 --build-arg strimzi_version=$(RELEASE_VERSION) -t strimzi/$(PROJECT_NAME):latest $(DOCKERFILE_DIR)
  1. FOR BUILDING, here is what we did, below steps are done under BASH 5.x which is required by this repo:(we just installing via homebrew then ran ./opt/homebrew/Cellar/bash/5.2.21/bin/bash to enter bash 5:

/path/to/yq below should point to the yq 3.3.2 that you downloaded and we put it first in the path so other yq versions are not used:

# first run /opt/homebrew/Cellar/bash/5.2.21/bin/bash
export DOCKER_ORG=YOUR-ORG
export DOCKER_REGISTRY=YOUR-REGISTRY
export JAVA_HOME=/Library/Java/JavaVirtualMachines/openjdk-17.jdk/Contents/Home
export PATH="/path/to/yq:$PATH"
make MVN_ARGS="-DskipTests -Dmaven.javadoc.skip=true -e" all

Deploying

kubectl edit deployment strimzi-cluster-operator -n kafka-operator

Change the image to your image and CHECK for logs. The logs you add everywhere should help, here were our logs:

2023-11-27 18:40:16 INFO  Main:154 - Cluster Operator verticle started in namespace
2023-11-27 18:40:18 ERROR Ca:361 - CERTIFICATE LOOP
2023-11-27 18:40:18 ERROR Ca:388 - IS EXPIRING? 
2023-11-27 18:40:18 ERROR Ca:361 - CERTIFICATE LOOP
2023-11-27 18:40:18 ERROR Ca:388 - IS EXPIRING? 
2023-11-27 18:40:18 ERROR Ca:361 - CERTIFICATE LOOP
2023-11-27 18:40:18 ERROR Ca:388 - IS EXPIRING? 
2023-11-27 18:40:20 ERROR KafkaAssemblyOperator:2278 - generating kafka certificates
2023-11-27 18:40:20 ERROR KafkaAssemblyOperator:2282 - calling generateCertificates
2023-11-27 18:40:20 ERROR AbstractModel:693 - generating certs!
2023-11-27 18:40:20 ERROR Ca:157 - generateBrokerCerts
2023-11-27 18:40:20 ERROR Ca:207 - maybeCopyOrGenerateCerts
2023-11-27 18:40:20 ERROR Ca:361 - CERTIFICATE LOOP
2023-11-27 18:40:20 ERROR Ca:388 - IS EXPIRING? 
2023-11-27 18:40:20 ERROR Ca:391 - EXPIRING! REASON 
2023-11-27 18:40:20 ERROR Ca:401 - REGENRATING CERTS!!
2023-11-27 18:40:21 ERROR Ca:361 - CERTIFICATE LOOP
2023-11-27 18:40:21 ERROR Ca:388 - IS EXPIRING? 
2023-11-27 18:40:21 ERROR Ca:391 - EXPIRING! REASON 
2023-11-27 18:40:21 ERROR Ca:401 - REGENRATING CERTS!!
2023-11-27 18:40:21 ERROR Ca:361 - CERTIFICATE LOOP
2023-11-27 18:40:21 ERROR Ca:388 - IS EXPIRING? 
2023-11-27 18:40:21 ERROR Ca:391 - EXPIRING! REASON 
2023-11-27 18:40:21 ERROR Ca:401 - REGENRATING CERTS!!

The certs and secrets were successfully generated and our Kafka cluster is back online.

Export your Kafka instance

  1. Export the data somewhere to be imported later
  2. Install the latest strimzi operator and migrate the data back in. I would NOT bother upgrading the operator if you have this level of incompatibility like we did.

All upgrade paths are documented in our documentation. However, if you run Strimzi 0.20 on Kube 1.25, you are outside of any working and supported configurations - also with regard to the CRDs upgrades that happened in between these versions. And any proper upgrade might not be possible anymore. Again, you should check the old discussions on GitHub and on Slack to see how to best deal with it. It is not possible to give individual guidance again and again to everyone who did not bother to monitor or upgrade the clusters for years. Sorry.