external-dns: External DNS does not delete AWS aliases

I have installed external DNS v0.5.14 on our kubernetes cluster in AWS (values bellow). Right now we are using AWS aliases (not CNAME) for correct resolution to internet-facing load balancer which is pointed to nginx-ingress. So we annotate each ingress which should be accessible over internet with following statements:

kubernetes.io/ingress.class: nginx-external
external-dns.alpha.kubernetes.io/target: external-nlb.domain
external-dns.alpha.kubernetes.io/alias: "true"

The creation of route53 record works great, however if I delete the ingress, external-dns is unable to delete the records and yields following errors:

time="2019-07-18T08:06:21Z" level=error msg="Failed to submit all changes for the following zones: [/hostedzone/Z3UTxxx]"
time="2019-07-18T08:07:20Z" level=info msg="Desired change: DELETE hello.domain CNAME"
time="2019-07-18T08:07:20Z" level=info msg="Desired change: DELETE hello.domain TXT"
time="2019-07-18T08:07:21Z" level=error msg="InvalidChangeBatch: [Tried to delete resource record set [name='hello.domain.', type='CNAME'] but it was not found]\n\tstatus code: 400, request id: d26532cc-10c3-45b1-9901-1a003717bb46"

I believe this is a bug because there is no way for external-dns to tell whether the record was alias or CNAME after ingress deletion. Would it be possible to either add some marker to TXT record which which would assure that correct type is deleted or try to delete both CNAME and alias after ingress deletion?

Values:

annotationFilter: kubernetes.io/ingress.class=nginx-external
aws:
  accessKey: AKIAxxx
  credentialsPath: /.aws
  region: eu-central-1
  roleArn: ""
  secretKey: xxx
domainFilters: []
image:
  name: registry.opensource.zalan.do/teapot/external-dns
  tag: v0.5.14
interval: 1m
logLevel: info
podAnnotations:
  external-dns.alpha.kubernetes.io/target: nlb.domain
policy: sync
provider: aws
rbac:
  create: true
registry: txt
resources:
  limits:
    memory: 50Mi
  requests:
    cpu: 10m
    memory: 50Mi
sources:
- service
- ingress
txtOwnerId: Z3Uxxx
zoneIdFilters:
- Z3Uxxx

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 10
Comments: 16 (5 by maintainers)

Most upvoted comments

It looks like deletion of ALIAS records works if referencing a amazonaws.com address directly. Then external-dns looks up the hostedZoneId from a set of hardcoded values and deletes the ALIAS. If the address is not found among the hardcoded values the record is not considered an ALIAS and it fails to delete the record because it does not match with a record in Route53.

I think we can store the hostedZoneId with the endpoints and make some changes to how external-dns determines to use an alias or not and apply the hostedZoneId without hardcoded values. This apply to the aws provider only. I may be able to make a PR on this. Any comments?

spohner on Jan 9, 2020

/remove-lifecycle stale

ghost on Jul 30, 2020

We are affected by the same problem. We use external DNS for preview environments, and this forces us to manually go and delete records every now and then when AWS starts complaining about the number of records.

zaro on Apr 21, 2020