coredns: Breaking changes in ReWrite Plugin CoreDNS 1.8.4

AKS is currently running CoreDNS 1.8.3 and upgrading to CoreDNS 1.8.4 in September. A breaking change with the rewrite plugin with CoreDNS 1.8.4 has been identified.

This is an example for 1.8.3

<domain to be rewritten>.com:53 {
        errors
        cache 30
        rewrite name substring <domain to be rewritten>.com default.svc.cluster.local
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        forward .  /etc/resolv.conf # you can redirect this to a specific DNS server such as 10.0.0.10, but that server must be able to resolve the rewritten domain name
    }

The below is an example for 1.8.4 that won’t break, which requires customers to update there configuration.

  <domain to be rewritten>.com:53 {
  log
  errors
  rewrite stop {
    name regex (.*)\.<domain to be rewritten>.com {1}.default.svc.cluster.local
    answer name (.*)\.default\.svc\.cluster\.local {1}.<domain to be rewritten>.com
  }
  forward . /etc/resolv.conf # you can redirect this to a specific DNS server such as 10.0.0.10, but that server must be able to resolve the rewritten domain name

What happened: AKS is advising customers to update there rewrite plugin.

What you expected to happen: No breaking changes during patch version upgrades.

How to reproduce it (as minimally and precisely as possible): Apply a configuration in 1.8.3 and upgrade to 1.8.4.

Anything else we need to know?:

Environment: Azure Kubernetes Service 1.20+

the version of CoreDNS: 1.8.4
Corefile:
logs, if applicable:
OS (e.g: cat /etc/os-release):
Others:

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 17 (11 by maintainers)

Most upvoted comments

Cc @mandelsoft

chrisohaver on Aug 28, 2021

We hope new coredns is backward compatible to the old config, and the default behavior should be the same as the old one.

Yes, it should. Note however, that with the prior configuration some client resolvers will complain about the name mismatch, even in 1.8.3.

More detail … There are 3 types of the “question name” at play.

The original question that the client made - taken from the Question section of the request message.
Response Question Name: From the Question section of the response message.
Answer Record Name: From the names of records in the Answer section of the response message

Most client resolvers will fail when the original question != Response Question Name. Hence this was always restored in the response so they would be equal (until 1.8.4 by accident).

After early versions of the rewrite plugin had been released for some time, we got reports that some client resolvers were failing when the original question != Answer Record Name, so the answer rewrite option was added (long ago, in 1.0.5).

chrisohaver on Sep 9, 2021

The below is an example for 1.8.4 that won’t break, which requires customers to update there configuration.

 <domain to be rewritten>.com:53 {
 log
 errors
 rewrite stop {
   name regex (.*)\.<domain to be rewritten>.com {1}.default.svc.cluster.local
   answer name (.*)\.default\.svc\.cluster\.local {1}.<domain to be rewritten>.com
 }
 forward . /etc/resolv.conf # you can redirect this to a specific DNS server such as 10.0.0.10, but that server must be able to resolve the rewritten domain name

FWIW, For 1.8.4, a much simpler workaround is to the use the answer auto option (new in 1.8.4) … e.g.

rewrite name substring <domain to be rewritten>.com default.svc.cluster.local answer auto

chrisohaver on Aug 30, 2021

The answer rewrite is required for both 1.8.3 and 1.8.4, otherwise a client may reject the answer due to the response name not matching the request name.

According to docs, the only time happens automatically is for rewrite name exact rules. https://github.com/coredns/coredns/blob/master/plugin/rewrite/README.md#response-rewrites

So for a rewrite name substring rule, request/response names will be mismatched unless a response rewrite option is appended.

chrisohaver on Aug 27, 2021