coredns: Custom DNS Entries For Kubernetes using rewrite not working always

Issue: We are trying to implement secure communication (https) between our pods (containers) with our CA certificate (we have purchased one) inside our kubernetes cluster.

What we need: So In order to solve our issue, we are trying to set our core dns to resolve custom A records with service discovery. In another words, I need that 1 container will be resolvable by domain that I will choose to another domain (different services)

What we expect: We expect that other domains (not just my-namespace.svc.cluster.local) will be resolvable: from inside the apigw container: ping api.my.domain This should return the ip of the api container.

How I tried to solve: So we tried to use the rewrite with the following ConfigMap file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        rewrite name regex (.*)\.my\.domain {1}.default.svc.cluster.local
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           upstream
           fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        proxy . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }

Results: This bring some wired results: for curl or wget is working from the apigw to the api. but for ping or especially dns.js in our node js is not working.

We are getting the node js error:

Nov 29 07:38:18 apigw gateway[36]: warn: [EG:policy]  Error: getaddrinfo ENOTFOUND api.my.domain api.my.domain:5443
Nov 29 07:38:18 apigwgateway[36]:     at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:67:26)
Nov 29 07:38:21 apigwnode: gethostby*.getanswer: asked for "api.my.domain.default.svc.cluster.local", got "api.default.svc.cluster.local"
Nov 29 07:38:26 apigwnode: gethostby*.getanswer: asked for "api.my.domain.default.svc.cluster.local", got "api.default.svc.cluster.local"

looks like dns.js library can’t work with my rewrite rule while other Apps like curl can.

I also tried to add $ in the rewrite: rewrite name regex (.*)\.my\.domain$ {1}.default.svc.cluster.local but it cause to the curl and wget not to work either.

any idea of how to solve my issue? is the rewrite rule is not right? maybe a different way? like create another zone for my.domain and maybe sync it some how from the default zone (in my case default.svc.cluster.local)

Versions:

kubectl:

Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:57:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

kubeadm

kubeadm version: &version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:54:02Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

cni version:

v0.10.0-s390x
#using: https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

os:

Ubuntu 18.04.1 LTS

FYI - Coredns is already builtin in the version above

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 15 (9 by maintainers)

Most upvoted comments

working !!! Thank you so much…

Final coredns configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        rewrite stop {
           name regex (.*)\.my\.domain\.$ {1}.default.svc.cluster.local
           answer name (.*)\.default\.svc\.cluster\.local\.$ {1}.my.domain
        }          
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           upstream
           fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        proxy . /etc/resolv.conf
        cache 30
        reload
        loadbalance
    }

+34

dorsany on Nov 29, 2018

Try anchoring the regex matches to the end with \.$… the rewrite is matching the search domains, and doing the rewrite incorrectly …

e.g.

        rewrite stop {
           name regex (.*)\.my\.domain\.$ {1}.default.svc.cluster.local
           answer name (.*)\.default\.svc\.cluster\.local\.$ {1}.my.domain
        }

chrisohaver on Nov 29, 2018

So don’t they overlap?does the first take precedence?

Yes, according to: https://github.com/coredns/coredns/blob/master/plugin/rewrite/README.md#syntax

Also, you should add beginning/end anchors to your regex string, otherwise it could match the middle of a longer domain.

chrisohaver on Aug 20, 2021

No, Go regex doesn’t support look aheads … but this simpler config would work…

rewrite name exact testinternabla.prod.net. ca-test-internal1.prod.net.
rewrite name regex testinternal(.*)\.prod\.net\. {1}-test-internal1.prod.net.

chrisohaver on Aug 20, 2021

Looks like you need to also do a response rewrite. Some clients don’t like it when the answer they get contains a question that does not match the question they asked. The response rewrite can alter the question in the response, so it matches the original request. IMO rewrite should just do this for you automatically for all rewrites, but alas it does not.

See the section on Response Rewrites in the docs https://coredns.io/plugins/rewrite/ In your config, this would looks something like…

    rewrite stop {
        name regex (.*)\.my\.domain {1}.default.svc.cluster.local
        answer name (.*)\.default\.svc\.cluster\.local {1}.my.domain 
    }

chrisohaver on Nov 29, 2018