external-dns: Unable to connect to EKS control plane endpoints

When deploying external DNS within a EKS cluster I encountered issues with external-dns connecting to the Kubernetes control plane endpoints.

time="2019-04-10T15:02:34Z" level=info msg="Created Kubernetes client https://172.20.0.1:443"
time="2019-04-10T15:03:34Z" level=fatal msg="failed to sync cache: timed out waiting for the condition"

EKS Server version: Server Version: v1.12.6-eks-d69f1b

The Kubernetes service was configured correctly, and other pods were able to communicate with the control plane endpoint.

NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   172.20.0.1   <none>        443/TCP   1d

The resources deployed:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: ["core"]
  resources: ["services"]
  verbs: ["get","watch","list"]
- apiGroups: ["core"]
  resources: ["pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions"]
  resources: ["ingresses"]
  verbs: ["get","watch","list"]
- apiGroups: ["core"]
  resources: ["nodes"]
  verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: default
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: registry.opensource.zalan.do/teapot/external-dns:v0.5.12
        args:
        - --source=service
        - --domain-filter=example.com
        - --provider=aws
        - --policy=upsert-only
        - --aws-zone-type=private
        - --registry=txt
        - --txt-owner-id=example-com

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 17 (5 by maintainers)

Most upvoted comments

I think in case of @szymonpk described, you also have to allow external-dns to get nodes

- apiGroups: ["core"]
  resources: ["nodes"]
  verbs: ["get", "list"]

njuettner on Apr 18, 2019

Latest also have issues in EKS (1.11.9), I am not sure if it is related:

ERROR: logging before flag.Parse: W0418 05:26:24.282304       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work
....
ERRRO: logging before flag.Parse: E0418 05:32:55.296176       1 round_trippers.go:291] CancelRequest not implemented
ERRRO: logging before flag.Parse: E0418 05:32:55.296383       1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
ERRRO: logging before flag.Parse: E0418 05:32:56.123149       1 reflector.go:322] pkg/mod/k8s.io/client-go@v8.0.0+incompatible/tools/cache/reflector.go:99: Failed to watch *v1.Node: unknown (get nodes)
ERRRO: logging before flag.Parse: E0418 05:32:57.130803       1 reflector.go:322] pkg/mod/k8s.io/client-go@v8.0.0+incompatible/tools/cache/reflector.go:99: Failed to watch *v1.Node: unknown (get nodes)

Reverting to 0.5.11 helps.

szymonpk on Apr 18, 2019

Also worth noting, external-dns version v0.5.11 works correctly in this environment.

kristaxox on Apr 15, 2019

@kristaxox We released v0.5.14 yesterday. This fixes the noisy logs you saw. Let me know if it works for you and we can close the ticket.

njuettner on May 15, 2019

Sorry this PR isn’t relevant, it was the switch to go modules in v0.5.13.

When we used dep we supressed those logs by overwriting the glog package, which was used in Kubernetes (now it’s klog). We can suppress it again by simply overwriting the package in go mod.

[[projects]]	
  branch = "master"	
  digest = "1:b12aff239810a9fa71e901a712a52f9da4c6e536852e943be693dec1d4519dfd"	
  name = "github.com/golang/glog"	
  packages = ["."]	
  pruneopts = ""	
  revision = "3fa5b9870d1d29f6d7907b29f1ae8c6eeb403829"	
  source = "github.com/kubermatic/glog-logrus"

I will create a PR for that. Again those logs are non-critical by I do understand they pretty annoying which needs to be fixed.

njuettner on Apr 20, 2019

I can conform that 0.5.11 works in EKS

johan-smits on Apr 17, 2019