kafka-go: Unknown Topic Or Partition error, even though the topic should be created automatically

Describe the bug

When publishing a message, I get Unknown Topic Or Partition: the request is for a topic or partition that does not exist on this broker, event though it should create the message by default, as far as I understand the README.md.

It worked before, when I was using the exact same setup on NixOS (until I switched to Arch Linux yesterday).

Kafka Version

Latest docker-image from confluentinc (should be 2.7.1 if it’s the latest official release from their GitHub, as of today)

To Reproduce

Start Kafka using the following docker-compose.yaml:

version: "3.3"
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 22181:2181
  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

execute the following commands:

docker-compose -f "FILE UNDER WHICH YOU'VE STORED THE ABOVE" up -d
mkdir tmp
cd tmp
go mod init example.com/tmp
go get -u github.com/segmentio/kafka-go
vim main.go

Fill main.go with the following lines:

package main

import (
	"context"

	"github.com/segmentio/kafka-go"
)

func main() {
	w := &kafka.Writer{
		Addr:  kafka.TCP("localhost:29092"),
		Topic: "hello",
	}

	err := w.WriteMessages(context.Background(), kafka.Message{
		Key:   []byte("hello"),
		Value: []byte("hello!"),
	})

	if err != nil {
		panic(err)
	}
}

Run main.go

go run main.go

The output was the following for me:

panic: [3] Unknown Topic Or Partition: the request is for a topic or partition that does not exist on this broker

goroutine 1 [running]:
main.main()
        /home/tim/tmp/main.go:21 +0x265
exit status 2

Expected behavior

The message should be published, even though the topic doesn’t exist yet.

Additional context

❯ uname -a
Linux t14s 5.12.10-arch1-1 #1 SMP PREEMPT Thu, 10 Jun 2021 16:34:50 +0000 x86_64 GNU/Linux

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 15
  • Comments: 16 (5 by maintainers)

Commits related to this issue

Most upvoted comments

I just tested this and I noticed that the topic is created if kafka.DialLeader is used to initialize the connection instead of directly using kafka.Writer. Adding the following code to the beginning of your main.go, just to initialize the topic, “fixes” the issue:

	conn, err := kafka.DialLeader(context.Background(), "tcp", "localhost:29092", "hello", 0)
	if err != nil {
		panic(err)
	}
	// close the connection because we won't be using it
	conn.Close()

This is not a good solution, I wouldn’t even consider it as a solution to be honest, but can be used as a workaround if needed.

@uloureiro: @rhansen2 will be taking this issue over. Thanks for your patience!

Hi @uloureiro, I haven’t had a chance to continue driving this, but going to see if someone else can pick it up.

Thanks for the reply @dtjm. I’m glad someone’s looking into this. At first I also wasn’t sure if I misconfigured something, but then other people in my team started experiencing the same behavior. I’m thrilled to know what went wrong there, as kafkacat seems to suffer from the same problem.