kops: api: loadbalancer: Internal does not create correct Route53 entries

Operating Environment kops 1.8.0 aws k8s 1.8.4

What we expect to happen Our kops spec looks like this:

apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: 2017-12-21T14:37:50Z
  name: ecom-2.dev.domain.com
spec:
  api:
    loadBalancer:
      type: Internal
...
  masterInternalName: api.internal.ecom-2.dev.domain.com

Per the documentation here, we expect kops to create and maintain a route53 record for api.internal.ecom-2.dev.domain.com, which points to an AWS loadbalancer created for the masters.

What actually happens We get instead a Route53 record that is a list of A records, corresponding to the masters .

Why this matters Our workloads experience failures because the nodes cannot talk to a master any more–probably because they are using the ip address directly. When the node dies, workloads get failures doing lookups using kube-dns-- research has shown that this is due to the fact that kube-dns on the node is timing out trying to get to the masters via the old ip.

We can temporarily fix the issue by manually updating the Route53 record to point to the load balancer. We’ve verified that if we do this, and THEN kill a master, the workloads are not affected.

The problem is that when the new master comes back on line, the cluster re-writes the Route53 name back to A records-- so the workaround will not survive rolling updates.

I believe this is the root cause for #4247, and #3702, which I am going to close in favor of this.

I think this may be the actual root cause of #2634. The solution mentioned there is exactly as I describe, but I suspect would not survive a rolling update.

What we need

We need to know what configuration we have wrong that causes us to get an internal name pointing to the masters directly, and how to fix it.

Alternatively, a workaround that simply disables kops from updating the internal DNS record would be super helpful.

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 3
Comments: 19 (10 by maintainers)

Commits related to this issue

GH-4252 Only manage internal DNS zone if configuration has been specified — committed to mellowplace/kops by deleted user 6 years ago
GH-4252 Only manage internal DNS zone if configuration has been specified — committed to mellowplace/kops by mellowplace 6 years ago
GH-4252 Only manage internal DNS zone if configuration has been specified — committed to mellowplace/kops by mellowplace 6 years ago
GH-4252 Better name for the config value and also add to v1alpha1 API — committed to mellowplace/kops by mellowplace 6 years ago

Most upvoted comments

Ok thanks for the info - I see I’ve gotten the wrong end of the stick on the proposed fix because api.internal is created with the placeholder IP, so you’re right this wouldn’t work, the record would never be pointed to the ELB.

I think we’d be totally fine with using the same ELB. I can’t think of a reason why you’d want a separate one, unless for auditing purposes maybe. Or you’ve split your VPC’s in such a way that the k8s nodes couldn’t see the ELB, is that possible?

Our config relating to this BTW is

spec:
  api:
    loadBalancer:
      type: Internal

We don’t specify the DNS setting, so we’d kind of just assumed the ELB was used for everything, do you think some people are expecting this is not the case? If so then I guess we would have to have an explicit setting. Perhaps something like the following…

spec:
  api:
    loadBalancer:
      type: Internal
      useApiInternal: true

Then on the basis of useApiInternal being present and true the record could be created to point to the ELB and the dns-controller annotation excluded.

mellowplace on Jun 29, 2018