kafka-docker: Internal host wrong

Hi everyone

In conclusion, What is the KAFKA_ADVERTISED_HOST_NAME for mac os x? The problem occurs when in OSX there is no docker-machine by default.

Thank you.

If you want to connect from different components within your compose file only, you can follow this example (irrespective of whether you are running docker-machine or docker natively):

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: "kafka"
      KAFKA_ADVERTISED_PORT: "9092"
      KAFKA_CREATE_TOPICS: "test:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  consumer:
    image: wurstmeister/kafka
    depends_on:
      - kafka
    command: [ "sh", "-c", "sleep 10 && $$KAFKA_HOME/bin/kafka-console-consumer.sh --topic=test --zookeeper=zookeeper:2181" ]
  producer:
    image: wurstmeister/kafka
    depends_on:
      - kafka
    command: [ "sh", "-c", "sleep 15 && echo 'hello kafka' | $$KAFKA_HOME/bin/kafka-console-producer.sh --topic=test --broker-list=kafka:9092" ]

if you want to be able to connect from external components and run docker natively, you can set KAFKA_ADVERTISED_HOST_NAME to the IP of your host (e.g. the output of ifconfig en0) if en0 is your main interface.

i.e. if your IP address is 192.168.0.19

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: "192.168.0.19"
      KAFKA_ADVERTISED_PORT: "9092"
      KAFKA_CREATE_TOPICS: "test:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  consumer:
    image: wurstmeister/kafka
    depends_on:
      - kafka
    command: [ "sh", "-c", "sleep 10 && $$KAFKA_HOME/bin/kafka-console-consumer.sh --topic=test --zookeeper=zookeeper:2181" ]
  producer:
    image: wurstmeister/kafka
    depends_on:
      - kafka
    command: [ "sh", "-c", "sleep 15 && echo 'hello kafka' | $$KAFKA_HOME/bin/kafka-console-producer.sh --topic=test --broker-list=192.168.0.19:9092" ]

if you want to be able to connect from external components and run docker in docker-machine, you can set KAFKA_ADVERTISED_HOST_NAME to the IP of your docker-machine (e.g. the output of docker-machine ip local) if local is your docker-machine name.

i.e. if your docker-machine IP address is 192.168.99.100

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: "192.168.99.100"
      KAFKA_ADVERTISED_PORT: "9092"
      KAFKA_CREATE_TOPICS: "test:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  consumer:
    image: wurstmeister/kafka
    depends_on:
      - kafka
    command: [ "sh", "-c", "sleep 10 && $$KAFKA_HOME/bin/kafka-console-consumer.sh --topic=test --zookeeper=zookeeper:2181" ]
  producer:
    image: wurstmeister/kafka
    depends_on:
      - kafka
    command: [ "sh", "-c", "sleep 15 && echo 'hello kafka' | $$KAFKA_HOME/bin/kafka-console-producer.sh --topic=test --broker-list=192.168.99.100:9092" ]

Hi, I might solved this:

version: '3.4'
services:
  zookeeper:
    image: wurstmeister/zookeeper:3.4.6
    ports:
      - 2181:2181
    healthcheck:
      test: ["CMD", "/opt/zookeeper/zkServer.sh", "status"]
      interval: 5s
      timeout: 5s
      retries: 3
    volumes:
      - ./data/zookeeper:/opt/zookeeper/data
  kafka:
    image: wurstmeister/kafka:0.10.0.0
    ports:
      - 9092:9092
    environment:
      KAFKA_PORT: 9092
      KAFKA_DELETE_TOPIC_ENABLE: "true"
      KAFKA_BROKER_ID: 0
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ADVERTISED_HOST_NAME: "${DOCKERHOST}"
      KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://${DOCKERHOST}:9092"
      KAFKA_ZOOKEEPER_CONNECT: "${DOCKERHOST}:2181"
      KAFKA_LOG_DIRS: /kafka/logs
    depends_on:
      - zookeeper
    volumes:
      - ./data/kafka/:/kafka/
    extra_hosts:
      - "dockerhost:$DOCKERHOST"

While DOCKERHOST is: DOCKERHOST=$(ifconfig | grep -E "([0-9]{1,3}\.){3}[0-9]{1,3}" | grep -v 127.0.0.1 | awk '{ print $2 }' | cut -f2 -d: | head -n1)

Ok, documenting here so I hope it helps others - using the IP address from ifconfig docker0 works to provide the right discoverability and connectivity between brokers, zookeeper, and external apps that are producers and consumers.

To isolate the ip I found this command from here on StackOverflow that works inside the container and modified it for this purpose:

export KAFKA_ADVERTISED_HOST_NAME=$(route -n | awk '/UG[ \t]/{print $2}')

Going to create a pull request eventually to get that into the startup script

In the meantime, I set this in the fig.yml to pick up the environment variable from the current shell:

kafka:
  environment:
    KAFKA_ADVERTISED_HOST_NAME:

and run this before running fig up:

export KAFKA_ADVERTISED_HOST_NAME=$(ifconfig docker0 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}')

which I adapted from here

@wurstmeister In MacOS, it seems I still can’t make it work. All my services are running in docker natively on MacOS (without docker-machine). All services share the same docker network, and i can ping each service name from the running docker container. I don’t need expose kafka broker for external use. So I expect the following config should work, but it doesn’t. For consumer in myapp, it reported “error”: “kafka server: Offset’s topic has not yet been created.”; for producer in myapp, it reported “error”: “circuit breaker is open”.

Any idea is highly appreciated !

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: "kafka"
      KAFKA_ADVERTISED_PORT: "9092"
      KAFKA_CREATE_TOPICS: "test:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  myapp: # which tries to kafka through "kafka:9092"
    build: .
    depends_on:
      - kafka
    command: ...

Hi,

did you set the KAFKA_ADVERTISED_HOST_NAME environment variable? (https://github.com/wurstmeister/kafka-docker/blob/master/fig.yml#L12)

Hi,

Thanks for making this image! I’m also encountering this issue. The problem with setting ADVERTISED_HOST_NAME to the host ip (eth0 or wlan0) is that you need to open the ports on the host firewall. It’s also a bit “ugly” in the sense that I would prefer my containers (that are all running on the same host) to be able to communicate through internal ips.

Maybe the default value of the ADVERTISED_HOST_NAME could be set to docker0 ip. The current behaviour (setting manually ADVERTISED_HOST_NAME) would still work. What do you think?

Hi,

please check https://issues.apache.org/jira/browse/KAFKA-1092 regarding the ADVERTISED_HOST_NAME parameter.

You should set this to the name/ip of the host that runs docker. If you run docker natively on a linux box, you can set this parameter to your host ip address (e.g. the IP of eth0), which will allow you to access kafka from other machines in the network.

If you set it to your docker0 IP address you can only access kafka from the host that runs docker.