strimzi-kafka-operator: Cluster operator failed to renew auto-generated certs leading to broker failure

Describe the bug

We are using a strimzi controlled kafka cluster for more than 3 years in production. Yesterday, the operator generated certificates expired (they were created on 06-04-2019) and this caused the cluster to stop working. Also we were using all prometheus alerting rules with kafka-exporter & kafka metrics defined in documentation and no alert went off. This deployment was first created using operator version 0.14.0 which I guess first generated the certificates. Then it was upgraded step by step up to 0.26.1.

Expected behavior We were expecting those certificates to auto renew since they are auto generated by the operator.

Environment (please complete the following information):

  • Strimzi version: 0.26.1
  • Installation method: YAML files
  • Kubernetes cluster: GKE 1.20.15-gke.1000
  • Infrastructure: GCP

YAML files and logs

Cluster yaml file:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kf
  namespace: infrastructure
spec:
  kafka:
    version: 2.8.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      delete.topic.enable: true
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "2.7"
      inter.broker.protocol.version: "2.8"
    storage:
      type: persistent-claim
      size: 200Gi
      deleteClaim: false
    jvmOptions:
      gcLoggingEnabled: false
      "-Xmx": 1g
      "-Xms": 1g
    readinessProbe:
      initialDelaySeconds: 120
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 120
      timeoutSeconds: 5
    resources:
      requests:
        memory: "5Gi"
        cpu: "2"
      limits:
        memory: "5Gi"
        cpu: "2"
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kf-kafka-metrics
          key: kf-kafka-metrics.yml
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 8Gi
      deleteClaim: false
    resources:
      requests:
        memory: "1Gi"
        cpu: "400m"
      limits:
        memory: "1Gi"
        cpu: "400m"
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kf-zookeeper-metrics
          key: kf-zookeeper-metrics.yml
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafkaExporter:
    groupRegex: ".*"
    topicRegex: ".*"
    resources:
      requests:
        cpu: 200m
        memory: 64Mi
      limits:
        cpu: 500m
        memory: 128Mi
    logging: info
    enableSaramaLogging: false
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5

Kafka logs:

2022-04-06 12:18:13,124 INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /10.56.25.25 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-1-ListenerName(REPLICATION-9091)-SSL-3]

Zookeeper logs after restarting zk pod:

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkClientCerts(CertificateMessage.java:700)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:411)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:375)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
	at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1565)
	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1579)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1463)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1290)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1337)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
	... 17 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
	at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369)
	at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:263)
	at java.base/sun.security.validator.Validator.validate(Validator.java:264)
	at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:135)
	at org.apache.zookeeper.common.ZKTrustManager.checkClientTrusted(ZKTrustManager.java:88)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkClientCerts(CertificateMessage.java:682)
	... 32 more
Caused by: java.security.cert.CertPathValidatorException: validity check failed
	at java.base/sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)
	at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:224)
	at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:144)
	at java.base/sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83)
	at java.base/java.security.cert.CertPathValidator.validate(CertPathValidator.java:309)
	at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:364)
	... 39 more
Caused by: java.security.cert.CertificateExpiredException: NotAfter: Wed Apr 06 12:21:35 GMT 2022
	at java.base/sun.security.x509.CertificateValidity.valid(CertificateValidity.java:277)
	at java.base/sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:675)
	at java.base/sun.security.provider.certpath.BasicChecker.verifyValidity(BasicChecker.java:190)
	at java.base/sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:144)
	at java.base/sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
	... 44 more

Additional context I believe that certificates were never renewed because they expired exactly 3 years after cluster was created.

There is nothing in the logs of the operator that resembles anything other than normal operation. I know renewal interval is by default 30 days before expiration now. So I checked 30 days prior and nothing out of the ordinary.

Is there anything that can be done to monitor the certificate expiration ? If it fails auto-renewal at least we can try to do it manually. Thanks

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 19 (8 by maintainers)

Most upvoted comments

from looking at the metrics, it was the active controller that failed to update the cert and didn’t fail over (split-brain?). there seems to be some intermittent network issues in the cluster; most likely the root cause. I’ve annotated the strimzipodset for a rolling restart which it is working through now. we also restarted the operator to be sure.

the scenario unfortunately makes sense, not sure if there’s any way to catch it before it happens though. 1. strimzi rolls the CA + certs. 2. one of the brokers doesn’t receive the new certs/CA. 3. somehow becomes the active controller for the cluster. 4. certs/CA expires (31d). 5. active controller can’t talk to the other brokers and thinks its the only active broker – though ZK should disagree on this. 6. ?? clients are not able to read or write (could see write block from under-min-isr, but not sure about the read block though that is what we experienced).

is there any way to catch the cert expiry mismatch before it becomes a problem?