dapr: pubsub kafka can not be used

I’m trying to use kafka as pubsub backend according to the dmo:

https://github.com/azure-octo/dapr-kafka-csharp/blob/master/deploy/kafka-pubsub.yaml

I changed the brokers section and here is my yaml:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: dapr-kafka-pubsub
  namespace: my-namespace
spec:
  type: pubsub.kafka
  metadata:
    - name: brokers
      value: "10.230.205.8:9092,10.230.205.9:9092"
    - name: authRequired
      value: "false"

I can ping and telnet the kafka broker successfully, and connect to it with kafka’s official java SDK.

However, that yaml keeps my daprd from starting, and logs as follow:

error intializing pub sub pubsub.kafka/v1: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

and then daprd exits.

It’s ok to use redis as pubsub backend, I can’t even start the sidecar using kafka as pubsub backend.

Any idea how to solve this question？

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 16 (5 by maintainers)

Most upvoted comments

@johnseed can you help have a try with this pubsub yaml?

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: order_pub_sub
spec:
  type: pubsub.kafka
  version: v1
  metadata:
  - name: brokers # Required. Kafka broker connection setting
    value: "kafka:9092"
  - name: authType # Required.
    value: "none"
  - name: authRequired
    value: "false"
  - name: disableTls
    value: "true"
scopes:
  - orderprocessing
  - checkout

Taction on Mar 9, 2022

P.S.

dapr subscription needs kafka v0.10.2.0 at least, so in our test with kafka 0.10.0.1, only pulisher is available.

darren-wang on Mar 11, 2022

@johnseed can you help have a try with this pubsub yaml?

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: order_pub_sub
spec:
  type: pubsub.kafka
  version: v1
  metadata:
  - name: brokers # Required. Kafka broker connection setting
    value: "kafka:9092"
  - name: authType # Required.
    value: "none"
  - name: authRequired
    value: "false"
  - name: disableTls
    value: "true"
scopes:
  - orderprocessing
  - checkout

It worked! Thank you so much.

johnseed on Mar 9, 2022

The default version is 2.0.0.0, and you can define it with version. I am currently not sure the supported range but you can have a try by adding:

- name: version
  value: "0.10.0.1"

Taction on Mar 9, 2022

Will take a look

yaron2 on Mar 8, 2022