external-dns: external-dns cannot automatically change record type in Google DNS

I’m migrating from managing DNS records using CNAMEs

ie.

external-dns.alpha.kubernetes.io/target:

to a more standard setup of using an A record to point at a LoadBalancer.

When I removed the target annotation, I get this error:

time="2018-01-04T16:28:41Z" level=info msg="Del records: test-consul.dev.foo.io. CNAME [helm.dev.foo.io.]"
time="2018-01-04T16:28:41Z" level=info msg="Add records: test-consul.dev.foo.io. CNAME [10.1.1.8]"
time="2018-01-04T16:28:42Z" level=error msg="googleapi: Error 400: Invalid value for 'entity.change.additions[0].rrdata[0]': '10.1.1.8', invalid"

It looks like it’s trying to replace the CNAME value with an IP, rather than delete the CNAME record and create a new A record.

Manually removing the old CNAME DNS record allows external-dns to do the right thing.

time="2018-01-04T16:31:44Z" level=info msg="Add records: test-consul.dev.foo.io. A [10.1.1.8]"
time="2018-01-04T16:31:44Z" level=info msg="Add records: helm-dev-test-consul.dev.foo.io. TXT ["heritage=external-dns,external-dns/owner=helm-dev"]"

Any hints on how to avoid a manual migration here?

About this issue

Original URL
State: open
Created 6 years ago
Comments: 16 (8 by maintainers)

Commits related to this issue

Disallow fmt.Errorf usage through code validation (#433) Came across during a code review, new contributors can end up using fmt.Errorf which does not create stack trace at the error site --and also ... — committed to lou-lan/external-dns by ahmetb 5 years ago

Most upvoted comments

@stealthybox unfortunately we have limited to no access to Google Cloud (read: that would cost us money that we don’t have) that makes really hard to reproduce and fix those issues. I’m unsure if @linki or @njuettner can get a cluster on GKE to test that.

I totally second the “help wanted” label, please feel free to jump in.

One of the goals of #540 is to get ExternalDNS to a state where we can have resources to run such tests. I’m slowly working on it, hoping to get it to a final state by the end of the month.

Raffo on Jun 10, 2019

Our team just ran into this with the Google Provider. The behavior is definitely buggy.

The use case was changing from Google Internal LoadBalancers to a single ILB fronting a shared Ingress Controller.

This meant Services previously having annotations for A records got removed and changed to Ingress Objects with annotations for CNAME records of the same hostname.

What we found is that when you have an A record in CloudDNS that collides with a requested CNAME target on the Ingress object, external-dns sometimes does not do the Delete and Re-add of the record. (sometimes it works with no human intervention.)

What’s concerning is that when it does not re-add, external-dns is unable to create new CNAME records. The controller seems to stop doing work, but the loop is still running and outputs logs like so:

time="2018-04-10T18:28:34Z" level=error msg="googleapi: Error 400: Invalid value for 'entity.change.additions[14].rrdata[0]': 'internal-ingress.companyci.com'
More details:
Reason: invalid, Message: Invalid value for 'entity.change.additions[14].rrdata[0]': 'internal-ingress.companyci.com'
Reason: invalid, Message: Invalid value for 'entity.change.additions[15].rrdata[0]': 'internal-ingress.companyci.com'

Manually deleting the colliding A records from CloudDNS seems to resolve the issue, and the controller picks up the necessary work and makes the new CNAME records.

This might be a race condition in the controller code or a bug in the provider specific implementation. I haven’t dug into it.

App Version: 0.4.8 Chart Version: 0.5.1

stealthybox on Apr 10, 2018