strimzi-kafka-operator: Cluster operator failed to renew auto-generated certs leading to broker failure
Describe the bug
We are using a strimzi controlled kafka cluster for more than 3 years in production. Yesterday, the operator generated certificates expired (they were created on 06-04-2019) and this caused the cluster to stop working. Also we were using all prometheus alerting rules with kafka-exporter & kafka metrics defined in documentation and no alert went off. This deployment was first created using operator version 0.14.0 which I guess first generated the certificates. Then it was upgraded step by step up to 0.26.1.
Expected behavior We were expecting those certificates to auto renew since they are auto generated by the operator.
Environment (please complete the following information):
- Strimzi version: 0.26.1
- Installation method: YAML files
- Kubernetes cluster: GKE 1.20.15-gke.1000
- Infrastructure: GCP
YAML files and logs
Cluster yaml file:
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: kf
namespace: infrastructure
spec:
kafka:
version: 2.8.0
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
config:
delete.topic.enable: true
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
log.message.format.version: "2.7"
inter.broker.protocol.version: "2.8"
storage:
type: persistent-claim
size: 200Gi
deleteClaim: false
jvmOptions:
gcLoggingEnabled: false
"-Xmx": 1g
"-Xms": 1g
readinessProbe:
initialDelaySeconds: 120
timeoutSeconds: 5
livenessProbe:
initialDelaySeconds: 120
timeoutSeconds: 5
resources:
requests:
memory: "5Gi"
cpu: "2"
limits:
memory: "5Gi"
cpu: "2"
metricsConfig:
type: jmxPrometheusExporter
valueFrom:
configMapKeyRef:
name: kf-kafka-metrics
key: kf-kafka-metrics.yml
zookeeper:
replicas: 3
storage:
type: persistent-claim
size: 8Gi
deleteClaim: false
resources:
requests:
memory: "1Gi"
cpu: "400m"
limits:
memory: "1Gi"
cpu: "400m"
metricsConfig:
type: jmxPrometheusExporter
valueFrom:
configMapKeyRef:
name: kf-zookeeper-metrics
key: kf-zookeeper-metrics.yml
entityOperator:
topicOperator: {}
userOperator: {}
kafkaExporter:
groupRegex: ".*"
topicRegex: ".*"
resources:
requests:
cpu: 200m
memory: 64Mi
limits:
cpu: 500m
memory: 128Mi
logging: info
enableSaramaLogging: false
readinessProbe:
initialDelaySeconds: 15
timeoutSeconds: 5
livenessProbe:
initialDelaySeconds: 15
timeoutSeconds: 5
Kafka logs:
2022-04-06 12:18:13,124 INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /10.56.25.25 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-1-ListenerName(REPLICATION-9091)-SSL-3]
Zookeeper logs after restarting zk pod:
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkClientCerts(CertificateMessage.java:700)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:411)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:375)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1565)
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1579)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1463)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1290)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1337)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
... 17 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369)
at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:263)
at java.base/sun.security.validator.Validator.validate(Validator.java:264)
at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:135)
at org.apache.zookeeper.common.ZKTrustManager.checkClientTrusted(ZKTrustManager.java:88)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkClientCerts(CertificateMessage.java:682)
... 32 more
Caused by: java.security.cert.CertPathValidatorException: validity check failed
at java.base/sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)
at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:224)
at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:144)
at java.base/sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83)
at java.base/java.security.cert.CertPathValidator.validate(CertPathValidator.java:309)
at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:364)
... 39 more
Caused by: java.security.cert.CertificateExpiredException: NotAfter: Wed Apr 06 12:21:35 GMT 2022
at java.base/sun.security.x509.CertificateValidity.valid(CertificateValidity.java:277)
at java.base/sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:675)
at java.base/sun.security.provider.certpath.BasicChecker.verifyValidity(BasicChecker.java:190)
at java.base/sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:144)
at java.base/sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
... 44 more
Additional context I believe that certificates were never renewed because they expired exactly 3 years after cluster was created.
There is nothing in the logs of the operator that resembles anything other than normal operation. I know renewal interval is by default 30 days before expiration now. So I checked 30 days prior and nothing out of the ordinary.
Is there anything that can be done to monitor the certificate expiration ? If it fails auto-renewal at least we can try to do it manually. Thanks
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 19 (8 by maintainers)
from looking at the metrics, it was the active controller that failed to update the cert and didn’t fail over (split-brain?). there seems to be some intermittent network issues in the cluster; most likely the root cause. I’ve annotated the strimzipodset for a rolling restart which it is working through now. we also restarted the operator to be sure.
the scenario unfortunately makes sense, not sure if there’s any way to catch it before it happens though. 1. strimzi rolls the CA + certs. 2. one of the brokers doesn’t receive the new certs/CA. 3. somehow becomes the active controller for the cluster. 4. certs/CA expires (31d). 5. active controller can’t talk to the other brokers and thinks its the only active broker – though ZK should disagree on this. 6. ?? clients are not able to read or write (could see write block from under-min-isr, but not sure about the read block though that is what we experienced).
is there any way to catch the cert expiry mismatch before it becomes a problem?