kafka-docker: Error when sending message to topic twitter with key: null

I am trying to run the this on my local machine. I have docker native installed (no docker-machine) on OSX.

My docker-compose.yml looks like

➜  kafka-docker git:(master) ✗ cat docker-compose.yml 
version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    build: .
    ports:
      - "9002:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

When I do docker-compose up -d, it runs them and I can confirm it as

➜  kafka-docker git:(master) ✗ docker ps
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                                NAMES
b2fa3f4e131d        wurstmeister/zookeeper   "/bin/sh -c '/usr/sbi"   2 minutes ago       Up 2 minutes        22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   kafkadocker_zookeeper_1
5f64ce15a589        kafkadocker_kafka        "start-kafka.sh"         2 minutes ago       Up 2 minutes        0.0.0.0:9002->9092/tcp                               kafkadocker_kafka_1

As see, I have mapping to localhost port 9092. I also see the following in kafka server logs

[2016-11-20 04:44:50,430] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: PLAINTEXT -> EndPoint(localhost,9002,PLAINTEXT) (kafka.utils.ZkUtils)

Now on my localhost (no inside docker container), I try to do

➜  kafka_2.11-0.10.1.0 bin/kafka-console-producer.sh --broker-list localhost:9092 --topic twitter
Hey

and it gives me error as

[2016-11-19 20:46:49,896] ERROR Error when sending message to topic twitter with key: null, value: 3 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

Questions

  • What have I done wrong here?
  • The kafka should be running inside docker container and I can run client from my localhost to send messages, right?
  • What can I do to fix the issue?

Thanks a lot

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Reactions: 21
  • Comments: 18

Most upvoted comments

Send some messages: bin/kafka-console-producer.sh --broker-list localhost:9092 --topic parker001

Get messages from consumer: bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic parker001 --from-beginning

notes: don’t mistake the port.

yea and also change the KAFKA_ADVERTISED_HOST_NAME=localhost to KAFKA_ADVERTISED_HOST_NAME=kafka or KAFKA_ADVERTISED_HOST_NAME=<ip-address-of-your-mac>