milvus: [Bug]: Kafka topics should be explicitly created with externally configured kafka
Is there an existing issue for this?
- I have searched the existing issues
Environment
- Milvus version: milvusdb/milvus-dev tag - 2.1.0
- Deployment mode(standalone or cluster): cluster
- SDK version(e.g. pymilvus v2.0.0rc2): go v2
- OS(Ubuntu or CentOS): ubuntu
- CPU/Memory: xx
- GPU: No
- Others:
External kafka enabled. We are using managed kafka cluster through Confluent where we use a "Basic" or "Standard" cluster [1].
[1]
Current Behavior
With external kafka configured, Topics required for inter-server communication are not created, but are delegated to the kafka consumers which are configured with allow.auto.create.topics
- https://github.com/milvus-io/milvus/blob/master/internal/mq/msgstream/mqwrapper/kafka/kafka_client.go#L121
Expected Behavior
With confluent managed kafka clusters, configuring behavior is currently not supported for the “Basic” and “Standard” clusters [1]. Also it seems that configuring this, may not result in intended behavior, see - https://github.com/confluentinc/confluent-kafka-go/issues/615
It would be great if these topics are created as part of the producer (this operation possibly could be made idempotent?) creation.
Steps To Reproduce
1. Setting up milvus cluster mode with the following changes:
"image": {
"all": {
"repository": "milvusdb/milvus-dev",
"tag": "2.1.0-latest"
}
},
"externalKafka": {
"enabled": true,
"brokerList": "brokerlist",
"securityProtocol": "SASL_SSL",
"sasl":
"mechanisms": "PLAIN",
"username": "username",
"password": "password",
}
- Deploying milvus on a kubernetes cluster
- Using go sdk to create a collection using
CreateCollection
method. This fails with:
CreateCollection failed: send dd create collection req failed, error = Broker: Unknown topic or partition
- In the logs of
datacoord
service/pod, we see:
[2022/06/30 23:58:54.152 +00:00] [ERROR] [kafka_consumer.go:143] ["consume msg failed"] [topic=by-dev-datacoord-timetick-channel] [groupID=by-dev-dataCoord] [error="by-dev-datacoord-timetick-channel [0]: topic does not exist (Broker: Unknown topic or partition)"] [stack="github.com/milvus-io/milvus/internal/mq/msgstream/mqwrapper/kafka.(*Consumer).Chan.func1.1\n\t/go/src/github.com/milvus-io/milvus/internal/mq/msgstream/mqwrapper/kafka/kafka_consumer.go:143"]
### Milvus Log
_No response_
### Anything else?
_No response_
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 28 (10 by maintainers)
Still relevant, please reopen.