external-dns: Records for istio-gateway not being created

This issue seems weird to me as it was working literally since I started using this last Thursday. But today it won’t create Route53 records for istio-gateways.

Every night, I destroy the EKS cluster I’m working on to not incur overnight charges. Then in the morning, I bring it back up and redeploy istio along with the sample apps I’m trying out and other custom workloads.

So far, every single time, the A records have been created and deleted as usual. But today they don’t seem to work for istio gateways, though they continue to work for the nginx ingresses.

helm deployment

helm install --debug stable/external-dns \
  --name=external-dns \
  --namespace=kube-system \
  --set image.tag=latest \
  --set sources[0]=service \
  --set sources[1]=ingress \
  --set sources[2]=istio-gateway \
  --set provider=aws \
  --set aws.secretKey="${AWS_SECRET_ACCESS_KEY}" \
  --set aws.accessKey="${AWS_ACCESS_KEY_ID}" \
  --set aws.region=us-east-1 \
  --set registry=txt \
  --set txtOwnerId=123456789 \
  --set policy=sync \
  --set logLevel=debug \
  --set resources.requests.cpu=100m \
  --set resources.requests.memory=128Mi \
  --set resources.limits.cpu=1500m \
  --set resources.limits.memory=256Mi \
  --set rbac.create=true \
  --set rbac.apiVersion=v1

I’ve tried filtering domains with no avail.

Log Output (NOT WORKING)

time="2019-02-05T19:49:21Z" level=info msg="Created Kubernetes client https://172.20.0.1:443"
time="2019-02-05T19:49:21Z" level=info msg="Created Istio client"
time="2019-02-05T19:49:26Z" level=debug msg="Considering zone: /hostedzone/ID1 (domain: domain1.cloud.)"
time="2019-02-05T19:49:26Z" level=debug msg="Considering zone: /hostedzone/ID2 (domain: domain2.cloud.)"
time="2019-02-05T19:49:26Z" level=debug msg="Considering zone: /hostedzone/ID3 (domain: domain3.cloud.)"
time="2019-02-05T19:49:26Z" level=debug msg="Considering zone: /hostedzone/ID4 (domain: domain4.cloud.)"
time="2019-02-05T19:49:26Z" level=debug msg="Considering zone: /hostedzone/ID5 (domain: domain5.cloud.)"
time="2019-02-05T19:49:26Z" level=debug msg="Considering zone: /hostedzone/ID6 (domain: domain6.cloud.)"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service default/details"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service default/kubernetes"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service default/productpage"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service default/ratings"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service default/reviews"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service graphql-api/api-server"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service graphql-api/postgres"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service graphql-api/prisma"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service istio-system/grafana"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service istio-system/istio-citadel"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service istio-system/istio-egressgateway"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service istio-system/istio-galley"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service istio-system/istio-ingressgateway"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service istio-system/istio-pilot"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service istio-system/istio-policy"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service istio-system/istio-sidecar-injector"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service istio-system/istio-telemetry"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service istio-system/jaeger-agent"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service istio-system/jaeger-collector"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service istio-system/jaeger-query"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service istio-system/kiali"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service istio-system/prometheus"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service istio-system/servicegraph"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service istio-system/tracing"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service istio-system/zipkin"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service kube-system/calico-typha"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service kube-system/cluster-autoscaler-aws-cluster-autoscaler"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service kube-system/external-dns"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service kube-system/kube-dns"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from service kube-system/tiller-deploy"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from gateway default/bookinfo-gateway"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from gateway graphql-api/api-server-gateway"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from gateway istio-system/istio-autogenerated-k8s-ingress"
time="2019-02-05T19:49:26Z" level=debug msg="No endpoints could be generated from gateway istio-system/istio-telemetry-gateway"


----SOME EXTRA LOGGING REMOVED (CONSIDERING DOMAIN XYZ LOGS)---


time="2019-02-05T19:49:27Z" level=info msg="All records are already up to date"

Log Output for working with NGINX

time="2019-02-05T19:52:21Z" level=debug msg="Considering zone: /hostedzone/ID1 (domain: domain1.cloud.)"
time="2019-02-05T19:52:21Z" level=debug msg="Considering zone: /hostedzone/ID2 (domain: domain2.cloud.)"
time="2019-02-05T19:52:21Z" level=debug msg="Considering zone: /hostedzone/ID3 (domain: domain3.cloud.)"
time="2019-02-05T19:52:21Z" level=debug msg="Considering zone: /hostedzone/ID4 (domain: domain4.cloud.)"
time="2019-02-05T19:52:21Z" level=debug msg="Considering zone: /hostedzone/ID5 (domain: domain5.cloud.)"
time="2019-02-05T19:52:21Z" level=debug msg="Considering zone: /hostedzone/ID6 (domain: domain6.cloud.)"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service default/details"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service default/kubernetes"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service default/nginx"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service default/productpage"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service default/ratings"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service default/reviews"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service graphql-api/api-server"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service graphql-api/postgres"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service graphql-api/prisma"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service ingress/nginx-ingress-controller"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service istio-system/grafana"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service istio-system/istio-citadel"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service istio-system/istio-egressgateway"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service istio-system/istio-galley"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service istio-system/istio-ingressgateway"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service istio-system/istio-pilot"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service istio-system/istio-policy"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service istio-system/istio-sidecar-injector"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service istio-system/istio-telemetry"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service istio-system/jaeger-agent"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service istio-system/jaeger-collector"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service istio-system/jaeger-query"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service istio-system/kiali"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service istio-system/prometheus"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service istio-system/servicegraph"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service istio-system/tracing"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service istio-system/zipkin"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service kube-system/calico-typha"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service kube-system/cluster-autoscaler-aws-cluster-autoscaler"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service kube-system/external-dns"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service kube-system/kube-dns"
time="2019-02-05T19:52:21Z" level=debug msg="No endpoints could be generated from service kube-system/tiller-deploy"
time="2019-02-05T19:52:21Z" level=debug msg="Endpoints generated from ingress: default/nginx: [domain1.cloud 0 IN CNAME NLB-CNAME-123456.elb.us-east-1.amazonaws.com []]"


----SOME EXTRA LOGGING REMOVED (CONSIDERING DOMAIN XYZ LOGS)---


time="2019-02-05T19:52:22Z" level=debug msg="Adding domain1.cloud. to zone domain1.cloud. [Id: /hostedzone/ID1]"
time="2019-02-05T19:52:22Z" level=debug msg="Adding domain1.cloud. to zone domain1.cloud. [Id: /hostedzone/ID2]"
time="2019-02-05T19:52:22Z" level=info msg="Desired change: CREATE domain1.cloud A"
time="2019-02-05T19:52:22Z" level=info msg="Desired change: CREATE domain1.cloud TXT"
time="2019-02-05T19:52:22Z" level=info msg="2 record(s) in zone domain1.cloud. were successfully updated"

The istio version did not change 1.0.5…

I’ve tried helm installs and manual kubectl installs just to try with the same results.

Any thoughts appreciated.