kubernetes: Pods on AWS with native networking can't access Amazon provided DNS
When using native networking in in AWS (i.e. with options cloud-provider=aws, cluster-name=<cluster_name>, allocate-node-cidrs=true, cluster-cidr=<pod_network>, configure-cbr0=true, bridge=cbr0, iptables=false, and ip-msq=false as appropriate to kube-apiserver, kube-controller-manager, kubelet, and docker), everything works except that Pods, including kube-dns, can’t make DNS queries to the Amazon provided DNS server in the subnet. That means Pods can’t resolve external hostnames.
It appears that the Amazon provided DNS server won’t respond to queries with an address other than one within the subnet. The solution to the problem is to masquerade traffic from the Pod network to the Amazon provided DNS server in the subnet.
sudo iptables --table nat -A POSTROUTING -s <pod_cidr> -d <dns_ip>/32 -j MASQUERADE
Arguably this should be done by kubeletwhen it is passed the configure-cbr0 option alongside the cloud-provider=aws option.
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Reactions: 1
- Comments: 17 (11 by maintainers)
Commits related to this issue
- multi-node/aws: Add native AWS networking. Requires the addition of a masquerade rule in workers from the pod network to the Amazon provided DNS. It appears that the Amazon provided DNS does not res... — committed to eliaslevy/coreos-kubernetes by eliaslevy 9 years ago
- multi-node/aws: Add native AWS networking. Requires the addition of a masquerade rule in workers from the pod network to the Amazon provided DNS. It appears that the Amazon provided DNS does not res... — committed to eliaslevy/coreos-kubernetes by eliaslevy 9 years ago
@therc while #19389 is of concern, it is not related to not being able to connect to the Amazon DNS server in the subnet, nor do I think its related to the RDS issue you mentioned.
Note that the masquerade rule with the hardcoded 10.0.0.0/8 network in kubelet says to masquerade traffic is not destined to 10.0.0.0/8. As the Amazon DNS and RDS are assigned IPs within the VPC network address, if you use the Kubernetes default of 10.0.0.0/8, then pod traffic to the Amazon DNS or RDS won’t be masqueraded.
Where #19389 fixed, then kubelet would be creating the same masquerade rule with whatever network you assigned the Kubernetes VPC, but the Amazon DNS and RDS would be assigned addresses within that network also. So in this case connections to them from the pods would not trigger the masquerade rule either. The behavior is the same in both cases.
The problem is not that #19389 has not been fixed, but at least for the Amazon DNS it is that with native network traffic the DNS server will see Pod traffic coming from 10.2.0.0/16 and that the DNS server filters out any traffic not coming from its local network of 10.0.0.0/16. That is why I suggest kubelet need to add a masquerade rule specific to masquerading Pod traffic to the Amazon DNS server.
I suspect the RDS issue you observed is similar, except that in that case you should be able to modify the RDS security group so that it accepts connections from the Pod’s CIDR, thus not requiring a masquerade rule.