strimzi-kafka-operator: [Bug] Cluster DNS not correctly set in Kafka advertised addresses
Describe the bug The Kubernetes cluster DNS (defaults to cluster.local) is not correctly set by the operator for Kafka brokers, as the brokers will advertise inter-namespace correct but intra-namespace incorrect DNS.
To Reproduce Steps to reproduce the behavior:
- Deploy operator:
helm install --namespace kube-system --generate-name --set "watchAnyNamespace=true,kubernetesServiceDnsDomain=sigma" strimzi/strimzi-kafka-operator - Deploy Kafka cluster:
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: development-default
namespace: development
spec:
kafka:
replicas: 2
version: 2.4.0
listeners:
plain: {}
storage:
type: persistent-claim
size: 20Gi
class: ceph-volatile
jvmOptions:
-Xms: 2048m
-Xmx: 2048m
config:
auto.create.topics.enable: "true"
zookeeper:
replicas: 1
jvmOptions:
-Xms: 1024G
-Xmx: 1024G
storage:
type: persistent-claim
size: 1Gi
class: ceph-volatile
- Note that operator namespace differs from Kafka namespace.
- Confirm that operator environment variables contain your cluster DNS - “sigma” this case.
KUBERNETES_SERVICE_DNS_DOMAIN: sigma - Confirm that Kafka brokers would not have correct DNS advertised (“sigma” is missing):
advertised.listeners=REPLICATION-9091://development-default-kafka-0.development-default-kafka-brokers.development.svc:9091,PLAIN-9092://development-default-kafka-0.development-default-kafka-brokers.development.svc:9092 - From kube-system namespace network, instantiate a Kafka Producer (2.4.0 - same version) and dial Kafka. Notice the error.
- Consumer bootstrap servers:
development-default-kafka-brokers.development.svc.sigma:9092
java.net.UnknownHostException: development-default-kafka-1.development-default-kafka-brokers.development.svc
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104)
Expected behavior
Kafka to advertise address development-default-kafka-1.development-default-kafka-brokers.development.svc.**sigma**
Environment:
- Strimzi version: 0.16.2
- Installation method: See above.
- Kubernetes cluster: Kubernetes 1.17 installed on CoreOS with Kubespray
- Infrastructure: Bare metal, storage provisioned by Ceph
Small note:
- The Kafka Consumer is instantiated by a development PC that has VPN access to the cluster through an OVPN pod sitting on kube-system namespace and exposed through NodePort.
- I have confirmed that from any kube-system pod the Kafka borkers can not be telnetted on development namespace by simply calling
telnet development-default-kafka-1.development-default-kafka-brokers.development.svc 9092.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 18 (9 by maintainers)
Commits related to this issue
- Handle more kube DNS resolution formats Add support of xxx.yyy.zzz.svc kube addresses DNS resolution, on top of already supported "service" and "service.namespace". This is required to handle Strimz... — committed to agrenott/telepresence by agrenott 4 years ago
- Handle more kube DNS resolution formats Add support of xxx.yyy.zzz.svc kube addresses DNS resolution, on top of already supported "service" and "service.namespace". This is required to handle Strimz... — committed to agrenott/telepresence by agrenott 4 years ago
- Handle more kube DNS resolution formats Add support of xxx.yyy.zzz.svc kube addresses DNS resolution, on top of already supported "service" and "service.namespace". This is required to handle Strimz... — committed to agrenott/telepresence by agrenott 4 years ago
@zzvara Just FYI: Since 0.20.0, this is now configurable on the listener level with the
useServiceDnsDomainoption:Docs: https://strimzi.io/docs/operators/latest/full/using.html#property-listener-config-dns-reference
For me, this issue happens through an OVPN connection, where the OVPN client asks the OVPN server container to resolve the Kafka brokers. The OVPN server fails to do so since it is in a separate namespace from the Kafka brokers AND the Kafka brokers do not provide an FQDN!
This should be fixed. Brokers should provide FQDN so that everyone would be happy across the cluster.
I don’t think it is deprecated. It is just not used where it is not needed. I also assume that impelmenting this https://github.com/strimzi/proposals/blob/master/005-improving-configurability-of-kafka-listeners.md migth give users more flexibility to deal with some more scenarios.