coredns: plugin/rewrite: queries fail in Rails
When I’m rewriting certain entries inside kubernetes cluster, DNS resolution in Rails app starts failing like follows:
irb(main):012:0> Resolv::DNS.new().getaddress("staging.mydomain.com")
Resolv::ResolvError: DNS result has no information for staging.mydomain.com
from /usr/local/lib/ruby/2.3.0/resolv.rb:386:in `getaddress'
from (irb):12
from /usr/local/bundle/gems/railties-4.2.10/lib/rails/commands/console.rb:110:in `start'
from /usr/local/bundle/gems/railties-4.2.10/lib/rails/commands/console.rb:9:in `start'
from /usr/local/bundle/gems/railties-4.2.10/lib/rails/commands/commands_tasks.rb:68:in `console'
from /usr/local/bundle/gems/railties-4.2.10/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
from /usr/local/bundle/gems/railties-4.2.10/lib/rails/commands.rb:17:in `<top (required)>'
from bin/rails:9:in `require'
from bin/rails:9:in `<main>'
Corefile:
.:53 {
errors
health
log
autopath @kubernetes
rewrite {
name regex staging.mydomain.com aws-loadbalancer-id.us-east-1.elb.amazonaws.com
answer name aws-loadbalancer-id.us-east-1.elb.amazonaws.com staging.mydomain.com
}
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
upstream
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
proxy . /etc/resolv.conf
cache 30
reload
loadbalance
}
Versions:
/app # rails -v
Rails 4.2.10
/app # ruby -v
ruby 2.3.4p301 (2017-03-30 revision 58214) [x86_64-linux-musl]
Container is based on Alpine Linux v3.4
At thre same time dig produces completely expected output:
; <<>> DiG 9.11.3 <<>> staging.mydomain.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2946
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;staging.mydomain.com. IN A
;; ANSWER SECTION:
staging.mydomain.com. 6 IN A 52.202.38.144
staging.mydomain.com. 6 IN A 54.236.98.217
staging.mydomain.com. 6 IN A 34.199.165.122
;; Query time: 1 msec
;; SERVER: 100.64.0.10#53(100.64.0.10)
;; WHEN: Tue Aug 21 17:08:12 UTC 2018
;; MSG SIZE rcvd: 161
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 52 (9 by maintainers)
Commits related to this issue
- plugin/rewrite: bug fixes * add closing dot for suffix rewrite rule * improve rule syntax checks Resolves: #1881 Partial Resolution: #2041 — committed to greenpau/coredns by greenpau 6 years ago
- plugin/rewrite: bug fixes * add closing dot for suffix rewrite rule * improve rule syntax checks Resolves: #1881 Partial Resolution: #2041 — committed to greenpau/coredns by greenpau 6 years ago
- plugin/rewrite: bug fixes * add closing dot for suffix rewrite rule * improve rule syntax checks Resolves: #1881 Partial Resolution: #2041 — committed to greenpau/coredns by greenpau 6 years ago
- plugin/rewrite: bug fixes * add closing dot for suffix rewrite rule * improve rule syntax checks * add size recalculation for rewritten answers Resolves: #1881 Partial Resolution: #2041 — committed to greenpau/coredns by greenpau 6 years ago
- plugin/rewrite: bug fixes * add closing dot for suffix rewrite rule * improve rule syntax checks * add size recalculation for rewritten answers Resolves: #1881 Partial Resolution: #2041 — committed to greenpau/coredns by greenpau 6 years ago
- plugin/rewrite: bug fixes * add closing dot for suffix rewrite rule * improve rule syntax checks * add size recalculation for rewritten answers Resolves: #1881 Partial Resolution: #2041 — committed to greenpau/coredns by greenpau 6 years ago
- plugin/rewrite: bug fixes * add closing dot for suffix rewrite rule * improve rule syntax checks * add size recalculation for rewritten answers Resolves: #1881 Partial Resolution: #2041 — committed to greenpau/coredns by greenpau 6 years ago
- plugin/rewrite: bug fixes * add closing dot for suffix rewrite rule * improve rule syntax checks Resolves: #1881 Partial Resolution: #2041 — committed to greenpau/coredns by greenpau 6 years ago
That’s an interesting question and you have pointed me out in the right direction. I didn’t use any overlay networks and simply relied on
kopsto bootstrap the cluster and set up networking and routes appropriately. That was not the case. What made me think that this is CoreDNS problem is thatkube-dnsstill worked with my 1-node setup, probably because it was running in the pods on the same node, while CoreDNS pods are running on the master. Now when I added an overlay network everything magically started to work. @greenpau @johnbelamaric thanks for your time and sorry for disturbing you.iptables-saveis even better, it gets all the tables.On Fri, Sep 21, 2018 at 11:45 AM Paul G. notifications@github.com wrote:
hi guys, and I guess I’m facing the same or very similar issue to @osnagovskyi
I noticed that DNS query packets get lost (though sometimes they are randomly proxied to AWS DNS) and here’s what I’ve found out
inside the container
To look at the link-level communication I ran
tcpdumpon the node (this output corresponds to thedigcommand above):So, as you see I didn’t get the response from CoreDNS at all. Needless to say there were no entries in CoreDNS log in its container. And on the other hand when I send requests directly to AWS DNS (my cluster is on AWS) here’s the response I get:
Also, here’s what my CoreDNS configmap like:
[ Quoting notifications@github.com in “Re: [coredns/coredns] plugin/rewrit…” ]
This shouldn’t matter, and if it does it’s a bug in miekg/dns (I think - should create some time to have a proper look)
@greenpau, he changed the config.
i.e. he removed the answer name re-write when doing the tcpdump. The initial tests were done with the answer name re-write, as described in the original issue description.
If you revisit that again, you can see that the answer is being re-written, per dig output. If it was not working, the names in the response would have been
aws-loadbalancer-id.us-east-1.elb.amazonaws.com. Yet ruby borks on the response.