kafka-go: Unknown Topic Or Partition error, even though the topic should be created automatically
Describe the bug
When publishing a message, I get Unknown Topic Or Partition: the request is for a topic or partition that does not exist on this broker, event though it should create the message by default, as far as I understand the README.md.
It worked before, when I was using the exact same setup on NixOS (until I switched to Arch Linux yesterday).
Kafka Version
Latest docker-image from confluentinc (should be 2.7.1 if it’s the latest official release from their GitHub, as of today)
To Reproduce
Start Kafka using the following docker-compose.yaml:
version: "3.3"
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ports:
- 22181:2181
kafka:
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
ports:
- 29092:29092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
execute the following commands:
docker-compose -f "FILE UNDER WHICH YOU'VE STORED THE ABOVE" up -d
mkdir tmp
cd tmp
go mod init example.com/tmp
go get -u github.com/segmentio/kafka-go
vim main.go
Fill main.go with the following lines:
package main
import (
"context"
"github.com/segmentio/kafka-go"
)
func main() {
w := &kafka.Writer{
Addr: kafka.TCP("localhost:29092"),
Topic: "hello",
}
err := w.WriteMessages(context.Background(), kafka.Message{
Key: []byte("hello"),
Value: []byte("hello!"),
})
if err != nil {
panic(err)
}
}
Run main.go
go run main.go
The output was the following for me:
panic: [3] Unknown Topic Or Partition: the request is for a topic or partition that does not exist on this broker
goroutine 1 [running]:
main.main()
/home/tim/tmp/main.go:21 +0x265
exit status 2
Expected behavior
The message should be published, even though the topic doesn’t exist yet.
Additional context
❯ uname -a
Linux t14s 5.12.10-arch1-1 #1 SMP PREEMPT Thu, 10 Jun 2021 16:34:50 +0000 x86_64 GNU/Linux
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 15
- Comments: 16 (5 by maintainers)
Commits related to this issue
- #683 WIP allow Writer to auto-create topics — committed to segmentio/kafka-go by deleted user 5 years ago
- Issue #683 WIP allow Writer to auto-create topics — committed to segmentio/kafka-go by deleted user 5 years ago
- Issue #683 WIP allow Writer to auto-create topics — committed to segmentio/kafka-go by deleted user 5 years ago
- Issue #683 WIP allow Writer to auto-create topics — committed to segmentio/kafka-go by deleted user 5 years ago
- Issue #683 WIP allow Writer to auto-create topics — committed to segmentio/kafka-go by deleted user 5 years ago
- Issue #683 WIP allow Writer to auto-create topics — committed to segmentio/kafka-go by deleted user 5 years ago
- #683 Use `Writer` literal to avoid need to inject mocking (fixes test failure) — committed to segmentio/kafka-go by deleted user 5 years ago
- #683 Add Writer literal to writer_test.go — committed to segmentio/kafka-go by deleted user 5 years ago
- Fix tests more #683 — committed to segmentio/kafka-go by deleted user 5 years ago
- Fix more tests #683 — committed to segmentio/kafka-go by deleted user 5 years ago
- Increase timeouts #683 — committed to segmentio/kafka-go by deleted user 5 years ago
- Fix tests more #683 — committed to segmentio/kafka-go by deleted user 5 years ago
- Fix more tests #683 — committed to segmentio/kafka-go by deleted user 5 years ago
- Increase timeouts #683 — committed to segmentio/kafka-go by deleted user 5 years ago
- Fix tests more #683 — committed to segmentio/kafka-go by deleted user 5 years ago
- Fix more tests #683 — committed to segmentio/kafka-go by deleted user 5 years ago
- Increase timeouts #683 — committed to segmentio/kafka-go by deleted user 5 years ago
I just tested this and I noticed that the topic is created if
kafka.DialLeaderis used to initialize the connection instead of directly usingkafka.Writer. Adding the following code to the beginning of yourmain.go, just to initialize the topic, “fixes” the issue:This is not a good solution, I wouldn’t even consider it as a solution to be honest, but can be used as a workaround if needed.
@uloureiro: @rhansen2 will be taking this issue over. Thanks for your patience!
Hi @uloureiro, I haven’t had a chance to continue driving this, but going to see if someone else can pick it up.
Thanks for the reply @dtjm. I’m glad someone’s looking into this. At first I also wasn’t sure if I misconfigured something, but then other people in my team started experiencing the same behavior. I’m thrilled to know what went wrong there, as kafkacat seems to suffer from the same problem.