strimzi-kafka-operator: Kafka-Connect Readiness and Liveness failed error

Hello Everyone, I am trying to deploy kafka-connect on strimzi operator but getting Readiness and Liveness failed error.looked for this in issues,but did not found any working solution.

Strimzi operator version : 0.19.0

Kafka version : 2.5.0

strimzi/kafka-connect:0.11.4-kafka-2.1.0

Kafka is also running in the same namespace with following bootstrap data-kafka-bootstrap ClusterIP 172.20.203.132 <none> 9091/TCP,9092/TCP,9093/TCP,9404/TCP 24d

Following is my kafka-connect config.I have not configured any readiness or liveness in this config. With Readiness and Liveness probe Disabled

versionNumber: 2.5.0
replicaCount: 1
namespace: kafka-cluster
bootstrapServersName: "data-kafka-bootstrap:9092"

image:
  repository: strimzi/kafka-connect
  tag: 0.11.4-kafka-2.1.0
  pullPolicy: Always

tls:
  certname : data-cluster-ca-cert

imagePullSecrets: []
nameOverride: "data"
fullnameOverride: ""

config_storage_replication_factor: 3
offset_storage_replication_factor: 3
status_storage_replication_factor: 3

Here is the describe output

4s         Normal    Scheduled           pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Successfully assigned kafka-cluster/data-kafka-connect-cluster-connect-74d98646d8-d825l to ip-10-0-1-15.us-east-2.compute.internal
13s         Normal    Pulling             pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Pulling image "strimzi/kafka-connect:0.11.4-kafka-2.1.0"
12s         Normal    Pulled              pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Successfully pulled image "strimzi/kafka-connect:0.11.4-kafka-2.1.0"
12s         Normal    Created             pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Created container data-kafka-connect-cluster-connect
11s         Normal    Started             pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Started container data-kafka-connect-cluster-connect
14s         Normal    SuccessfulCreate    replicaset/data-kafka-connect-cluster-connect-74d98646d8   Created pod: data-kafka-connect-cluster-connect-74d98646d8-d825l
14s         Normal    NoPods              poddisruptionbudget/data-kafka-connect-cluster-connect     No matching pods found
14s         Normal    ScalingReplicaSet   deployment/data-kafka-connect-cluster-connect              Scaled up replica set data-kafka-connect-cluster-connect-74d98646d8 to 1
6m59s       Warning   BackOff             pod/kafka-connect-cluster-connect-596b456dbf-5s2qq         Back-off restarting failed container
0s          Warning   Unhealthy           pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Liveness probe failed: Get http://10.0.1.51:8083/: dial tcp 10.0.1.51:8083: connect: connection refused
0s          Warning   Unhealthy           pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Readiness probe failed: Get http://10.0.1.51:8083/: dial tcp 10.0.1.51:8083: connect: connection refused
0s          Warning   Unhealthy           pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Liveness probe failed: Get http://10.0.1.51:8083/: dial tcp 10.0.1.51:8083: connect: connection refused
0s          Warning   Unhealthy           pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Readiness probe failed: Get http://10.0.1.51:8083/: dial tcp 10.0.1.51:8083: connect: connection refused
0s          Warning   Unhealthy           pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Liveness probe failed: Get http://10.0.1.51:8083/: dial tcp 10.0.1.51:8083: connect: connection refused

Although strimzi Operator has by default this configured.

  livenessProbe:
    initialDelaySeconds: 10
    periodSeconds: 30
  readinessProbe:
    initialDelaySeconds: 10
    periodSeconds: 30

I have also tried configuring kafka-connect with Readiness and Liveness probe with following config.But still same result.

With Readiness and Liveness probe Enabled

    livenessProbe:
      httpGet:
          path: /connectors
          port: 8083
      initialDelaySeconds: 60
      periodSeconds: 5
      timeoutSeconds: 5
      successThreshold: 1
      failureThreshold: 10
    readinessProbe:
      httpGet:
          path: /connectors
          port: 8083
      initialDelaySeconds: 40
      periodSeconds: 10
      timeoutSeconds: 5
      successThreshold: 1
      failureThreshold: 3

Please let me know if anything else required.

Thanks in advance.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 20 (10 by maintainers)

Most upvoted comments

Well, it is your choice really (consider security, versus performance etc.) - I would normally recommend using TLS unless you have some actual reasons against it.

But in any case, you can:

Keep using the port 9092 and jut disable the TLS
Keep the TLS but use port 9093 in the bootstrap address which is used in the Kafka cluster for TLS

Both should work.

scholzj on Oct 14, 2020