strimzi-kafka-operator: PKIX errors due to certs not renewing
Discussed in https://github.com/strimzi/strimzi-kafka-operator/discussions/6103
<div type='discussions-op-text'>Originally posted by talonx December 24, 2021 We are running several Kafka clusters managed by operator (v 0.20.0). We don’t have any custom TLS certs. A few days ago we started seeing multiple active controllers in one of our clusters, and the problem seems to boil down to the fact that two brokers (it’s a 3 broker cluster) are not able to talk to each other. The error is
{"exception":{"stacktrace":"org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed\nCaused by: javax.net.ssl.SSLProtocolException: Unexpected handshake message: server_hello\n\tat java.base\/sun.security.ssl.Alert.createSSLException(Alert.java:129)\n\tat java.base\/sun.security.ssl.Alert.createSSLException(Alert.java:117)\n\tat java.base\/sun.security.ssl.TransportContext.fatal(TransportContext.java:313)\n\tat java.base\/sun.security.ssl.TransportContext.fatal(TransportContext.java:269)\n\tat java.base\/sun.security.ssl.TransportContext.fatal(TransportContext.java:260)\n\tat java.base\/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:438)\n\tat java.base\/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)\n\tat java.base\/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)\n\tat java.base\/java.security.AccessController.doPrivileged(Native Method)\n\tat java.base\/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)\n\tat org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:425)\n\tat org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:509)\n\tat org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:363)\n\tat org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:286)\n\tat org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:174)\n\tat org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547)\n\tat org.apache.kafka.common.network.Selector.poll(Selector.java:485)\n\tat org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:549)\n\tat org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:74)\n\tat kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:288)\n\tat kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:242)\n\tat kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)","exception_class":"org.apache.kafka.common.errors.SslAuthenticationException","exception_message":"SSL handshake failed"},"source_host":"x-platform-kafka-0","method":"warn","level":"WARN","message":"[RequestSendThread controllerId=0] Controller 0's connection to broker x-platform-kafka-1.x-platform-kafka-brokers.kafka.svc:9091 (id: 1 rack: null) was unsuccessful","mdc":{},"@timestamp":"2021-12-23T18:39:14.654Z","file":"Logging.scala","line_number":"72","thread_name":"Controller-0-to-broker-1-send-thread","@version":1,"logger_name":"kafka.controller.RequestSendThread","class":"kafka.utils.Logging"}
I am assuming the operator should be updating the certificates automatically. What could be going wrong here?</div>
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 25 (10 by maintainers)
@scholzj you may find this entertaining, we have somewhat done the impossible and got the certs to generate through a really atypical approach, compile the thing ourselves and re-order the reconcile loop. For anyone running into this issue and you are on 0.20.0+, we can confidently say that cert generation works JUST fine and you are most likely running into the operator not running because of a Kubernetes incompatibility or something along those lines. CHECK your operator logs FIRST.
For anyone else in this situation, I recommend you follow the approach below, we had about 20hrs of downtime and this is basically the only solution. I’m going to open a new issue with my thoughts on the operator, specifically with its order of its reconcile loop (I think it could be better, put critical reconcile loop things like cert-generation FIRST.)
Clone the repo, switch to the broken version
git clone git@github.com:strimzi/strimzi-kafka-operator.gitgit checkout 0.20.0OR specifically your versionModify the reconcile loop
The reconcile loop determines what the operator is going to do next and in our case, the PodDisruption part failed to run which means all other steps in the reconcile loop fail to run (including the cert generation)
.compose(state -> state.zkPodDisruptionBudget())and moved all the cert generation/secret generation HIGHER than Ingress and PodDisruptionBudget which BOTH use v1beta1 making the operator failAdd logs everywhere
Add logs below, you can sprinkle them wherever cert generation happens. You can scroll ABOVE to follow the code paths that are responsible for cert generation
Building
BUT there are edge cases:
api/pom.xmland update:Get the version here https://github.com/mikefarah/yq/releases (search to 3.3.2)
If your prerequisites-check.sh fails on yq, make sure that yq is using 3.3.2
Update Makefile.docker to use a linux arch:
/opt/homebrew/Cellar/bash/5.2.21/bin/bashto enter bash 5:/path/to/yq below should point to the yq 3.3.2 that you downloaded and we put it first in the path so other yq versions are not used:
Deploying
Change the image to your image and CHECK for logs. The logs you add everywhere should help, here were our logs:
The certs and secrets were successfully generated and our Kafka cluster is back online.
Export your Kafka instance
All upgrade paths are documented in our documentation. However, if you run Strimzi 0.20 on Kube 1.25, you are outside of any working and supported configurations - also with regard to the CRDs upgrades that happened in between these versions. And any proper upgrade might not be possible anymore. Again, you should check the old discussions on GitHub and on Slack to see how to best deal with it. It is not possible to give individual guidance again and again to everyone who did not bother to monitor or upgrade the clusters for years. Sorry.