kafka-docker: Cannot consume nor produce from/to Kafka with default compose file

Hello, Replacing the KAFKA_ADVERTISED_HOST_NAME with 0.0.0.0 (I’m trying to run it locally), I’m able to successfully run the compose file. Here’s my docker ps output:

CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                                NAMES
67ddc55ff896        kafkadocker_kafka        "start-kafka.sh"         11 minutes ago      Up 11 minutes       0.0.0.0:32769->9092/tcp                              kafkadocker_kafka_1
cffa6466c04d        wurstmeister/zookeeper   "/bin/sh -c '/usr/..."   11 minutes ago      Up 11 minutes       22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   kafkadocker_zookeeper_1

However, when running: ./kafka-console-consumer.sh --topic test --bootstrap-server 0.0.0.0:32769 or ./kafka-console-producer.sh --broker-list 0.0.0.0:32769 --topic test

from a local kafka installation folder, I only get this kind of log lines:

[2017-02-14 15:01:03,970] WARN Error while fetching metadata with correlation id X : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

Modifying the compose file to expose kafka on my host’s 9092 port does the trick but is not an option since I would like to spawn several instances.

I’m using Docker 1.13.1, kafka-docker at tag 0.10.1.1 and binaries and scripts from kafka_2.11-0.10.1.1 .

Thanks!

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 5
Comments: 20

Most upvoted comments

This is my working configuration for this

#!/usr/bin/env bash

export DOCKER_KAFKA_HOST=$(ipconfig getifaddr en0)

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: ${DOCKER_KAFKA_HOST}
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

You should source the env before running the docker-compose, it should work

+43

hung-phan on May 12, 2017

This is my working configuration for this:

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka1:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://:9092
      KAFKA_ADVERTISED_LISTENERS: INSIDE://192.168.47.131:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
    depends_on:
      - zookeeper
  kafka2:
    image: wurstmeister/kafka
    ports:
      - "9096:9096"
    environment:
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://:9096
      KAFKA_ADVERTISED_LISTENERS: INSIDE://192.168.47.131:9096
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
    depends_on:
      - zookeeper
  kafka3:
    image: wurstmeister/kafka
    ports:
      - "9010:9010"
    environment:
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://:9010
      KAFKA_ADVERTISED_LISTENERS: INSIDE://192.168.47.131:9010
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
    depends_on:
      - zookeeper

barrer on Dec 19, 2018

Any update on this? It appears the 2 most popular images for kafka on Docker do not work with Docker for Mac, or Docker for Windows.

It’s time to write my own I suppose.

arianitu on Apr 19, 2017

Shikkic on Mar 8, 2017

This is my working configuration for this

#!/usr/bin/env bash

export DOCKER_KAFKA_HOST=$(ipconfig getifaddr en0)

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: ${DOCKER_KAFKA_HOST}
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

You should source the env before running the docker-compose, it should work

Got this working, thanks a lot. I was having trouble as my advertised host was set to localhost even though my clients were also running on the same box. How come that doesn’t work?

henryjcee on Dec 13, 2018

Using @hung-phan 's configuration as well as manually creating the topic using the Kafka quickstart docs (i.e. bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test) worked for me using the Node client.

JesterXL on Jan 31, 2018

I have the same problem

mach-kernel on May 12, 2017

This sounds like a configuration issue. Please check the wiki page: https://github.com/wurstmeister/kafka-docker/wiki/Connectivity. Hopefully that will help.

sscaling on Mar 29, 2018

I figured it out too. For me, I was forwarding this externally on a different port, and did not set KAFKA_ADVERTISED_PORT! Many thanks @hung-phan for the additional info, this is sure to help someone out 😸

mach-kernel on May 12, 2017