coredns: Core DNS is not resolving external ip
We have a kubernetes service that points to an external service ip. External service IP is the ec2 instance internal IP address where the mysql service is running at 3306
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
mysql ExternalName <none> 172.31.21.116 3306/TCP 20m
- When using CORE DNS, the external IP service do not resolve
/ # nslookup mysql.letmetry
Server: 100.64.0.10
Address 1: 100.64.0.10 kube-dns.kube-system.svc.cluster.local
nslookup: can't resolve 'mysql.letmetry'
- When using KUBE DNS, the external IP service resolves
/ # nslookup mysql.letmetry.svc.cluster.local.
Server: 100.64.0.10
Address 1: 100.64.0.10 kube-dns.kube-system.svc.cluster.local
Name: mysql.letmetry.svc.cluster.local.
Address 1: 172.31.21.116 ip-172-31-21-116.ap-south-1.compute.internal
Please suggest why this is happening. The logs of CORE DNS when this request is made -
100.96.109.30:41248 - [19/Nov/2018:14:56:25 +0000] 4 "AAAA IN mysql.letmetry.default.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,rd,ra 151 110.391µs
100.96.109.30:42034 - [19/Nov/2018:14:56:25 +0000] 5 "AAAA IN mysql.letmetry.svc.cluster.local. udp 50 false 512" NXDOMAIN qr,rd,ra 143 91.647µs
100.96.109.30:41174 - [19/Nov/2018:14:56:25 +0000] 6 "AAAA IN mysql.letmetry.cluster.local. udp 46 false 512" NXDOMAIN qr,rd,ra 139 93.808µs
100.96.109.30:34985 - [19/Nov/2018:14:56:25 +0000] 7 "AAAA IN mysql.letmetry.ap-south-1.compute.internal. udp 60 false 512" NXDOMAIN qr,rd,ra 60 1.458123ms
100.96.109.30:45287 - [19/Nov/2018:14:56:25 +0000] 8 "A IN mysql.letmetry. udp 32 false 512" NXDOMAIN qr,rd,ra 108 102.671µs
100.96.109.30:36956 - [19/Nov/2018:14:56:25 +0000] 9 "A IN mysql.letmetry.default.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,rd,ra 151 91.6µs
100.96.109.30:51749 - [19/Nov/2018:14:56:25 +0000] 10 "A IN mysql.letmetry.svc.cluster.local. udp 50 false 512" NXDOMAIN qr,rd,ra 143 92.572µs
100.96.109.30:54959 - [19/Nov/2018:14:56:25 +0000] 11 "A IN mysql.letmetry.cluster.local. udp 46 false 512" NXDOMAIN qr,rd,ra 139 87.971µs
100.96.109.30:42924 - [19/Nov/2018:14:56:25 +0000] 12 "A IN mysql.letmetry.ap-south-1.compute.internal. udp 60 false 512" NXDOMAIN qr,rd,ra 60 1.420079ms
System info
- It is a default installation of Core DNS with kops. No custom configuration was made. We just tried to replace kube dns with core dns using kops and are stuck here.
image: gcr.io/google_containers/coredns:1.1.3- kubernetes version:
1.9.8 - kops version:
1.10.0 - CoreFile
Corefile: |-
.:53 {
errors
log
health
kubernetes cluster.local. in-addr.arpa ip6.arpa {
pods insecure
upstream
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
proxy . /etc/resolv.conf
cache 30
reload
}
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 16 (10 by maintainers)
Commits related to this issue
- feat: resolve externalname services which contains an IP address Fix #2324 Signed-off-by: Sylvain Rabot <s.rabot@lectra.com> — committed to sylr/coredns by deleted user 5 years ago
- feat: resolve externalname services which contain an IP address Fix #2324 Signed-off-by: Sylvain Rabot <s.rabot@lectra.com> — committed to sylr/coredns by deleted user 5 years ago
@nvenky this happens when you have some
ServiceswithexternalNameas an IP instead of a hostname, as mentioned above.Turn the IPs into hostnames to fix this.
Got into this “issue” from the ingress-nginx controller perspective - version 0.18.0 or later outputs the following line:
2018/11/19 15:53:42 [error] 110379#110379: *24398057 [lua] dns.lua:61: resolve(): server returned error code: 3: name error, context: ngx.timerThis happens when any
externalNameis an IP, and it tries to query the nameserver with the IP as an A record.Both CoreDNS and ingress-nginx are working according to spec, and this detail should be in the official Kubernetes docs for clarity.
Not respecting the spec and going beyond it are not the same thing. You’ll only make the transition smoother for kube-dns users like myself who unwillingly got into this situation.
We should release note this in K8s (@rajansandeep ?) as a change in behavior, but CoreDNS is working according to the spec. It’s actually a bug in kube-dns - an accident of the skydns code and the missing validations.
@sylr This is the Kubernetes spec and CoreDNS follows it as intended. Docs have been equally updated to make this detail visible.
If you want to push for this change, a good place for that would be SIG-Network in Kubernetes. If accepted and implemented there, only then I assume CoreDNS would propagate the change.
Perhaps. Although strictly speaking, kube-dns is violating the Cluster DNS Spec in behaving that way.
IMO, it’s not intended that the ExternalName field should be able to contain an IP address. All documentation I can find assumes it will be a name, not an IP. The API should probably not allow IPs in that field.
Diving into the k8s API validation code, it looks like the Service validation is letting IPv4s slip through because it thinks they are domain names (after all, an IPV4 address is a series of valid dns labels delimited by dots). For example, looking at the code, an IPv6 address would fail validation because they contain colons. In other words, the API appears to be accepting an IPv4 address as a domain name, not as an IP. This is the regex the API uses …
[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*(used inIsDNS1123Subdomain()which is used when validating the externalName field for ExternalName services).OK, the issue is, I believe, that the external name field is supposed to hold a DNS name, not an IP. For example, per the Cluster DNS spec, this field is expected to be a DNS name because it must be used to construct a CNAME record (which can only contain a DNS name target).
If you want to point a cluster service to an external IP, you can do so with a headless service and static endpoint.