strimzi-kafka-operator: [Bug] Cluster DNS not correctly set in Kafka advertised addresses

Describe the bug The Kubernetes cluster DNS (defaults to cluster.local) is not correctly set by the operator for Kafka brokers, as the brokers will advertise inter-namespace correct but intra-namespace incorrect DNS.

To Reproduce Steps to reproduce the behavior:

  1. Deploy operator: helm install --namespace kube-system --generate-name --set "watchAnyNamespace=true,kubernetesServiceDnsDomain=sigma" strimzi/strimzi-kafka-operator
  2. Deploy Kafka cluster:
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: development-default
  namespace: development
spec:
  kafka:
    replicas: 2
    version: 2.4.0
    listeners:
      plain: {}
    storage:
      type: persistent-claim
      size: 20Gi
      class: ceph-volatile
    jvmOptions:
      -Xms: 2048m
      -Xmx: 2048m
    config:
      auto.create.topics.enable: "true"
  zookeeper:
    replicas: 1
    jvmOptions:
      -Xms: 1024G
      -Xmx: 1024G
    storage:
      type: persistent-claim
      size: 1Gi
      class: ceph-volatile
  1. Note that operator namespace differs from Kafka namespace.
  2. Confirm that operator environment variables contain your cluster DNS - “sigma” this case. KUBERNETES_SERVICE_DNS_DOMAIN: sigma
  3. Confirm that Kafka brokers would not have correct DNS advertised (“sigma” is missing): advertised.listeners=REPLICATION-9091://development-default-kafka-0.development-default-kafka-brokers.development.svc:9091,PLAIN-9092://development-default-kafka-0.development-default-kafka-brokers.development.svc:9092
  4. From kube-system namespace network, instantiate a Kafka Producer (2.4.0 - same version) and dial Kafka. Notice the error.
  • Consumer bootstrap servers: development-default-kafka-brokers.development.svc.sigma:9092
java.net.UnknownHostException: development-default-kafka-1.development-default-kafka-brokers.development.svc
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
	at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104)

Expected behavior Kafka to advertise address development-default-kafka-1.development-default-kafka-brokers.development.svc.**sigma**

Environment:

  • Strimzi version: 0.16.2
  • Installation method: See above.
  • Kubernetes cluster: Kubernetes 1.17 installed on CoreOS with Kubespray
  • Infrastructure: Bare metal, storage provisioned by Ceph

Small note:

  • The Kafka Consumer is instantiated by a development PC that has VPN access to the cluster through an OVPN pod sitting on kube-system namespace and exposed through NodePort.
  • I have confirmed that from any kube-system pod the Kafka borkers can not be telnetted on development namespace by simply calling telnet development-default-kafka-1.development-default-kafka-brokers.development.svc 9092.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 18 (9 by maintainers)

Commits related to this issue

Most upvoted comments

@zzvara Just FYI: Since 0.20.0, this is now configurable on the listener level with the useServiceDnsDomain option:

    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
        configuration:
          useServiceDnsDomain: true

Docs: https://strimzi.io/docs/operators/latest/full/using.html#property-listener-config-dns-reference

For me, this issue happens through an OVPN connection, where the OVPN client asks the OVPN server container to resolve the Kafka brokers. The OVPN server fails to do so since it is in a separate namespace from the Kafka brokers AND the Kafka brokers do not provide an FQDN!

This should be fixed. Brokers should provide FQDN so that everyone would be happy across the cluster.

I don’t think it is deprecated. It is just not used where it is not needed. I also assume that impelmenting this https://github.com/strimzi/proposals/blob/master/005-improving-configurability-of-kafka-listeners.md migth give users more flexibility to deal with some more scenarios.