external-dns: AWS cross account access with OIDC provider in EKS & externalDNS not working

Hello to everyone,

I hope this is the right place to ask my question. If not please point me to the correct place or documentation.

We are in the process of setting up an AWS EKS cluster that uses the OIDC provider for IAM access control to AWS resources. So far this works fine for the ALB ingresss controler. Now we want to also add externalDNS to create DNS records in Route53 for zones hosted in another AWS account.

We have a MGMT AWS account (ID: 222233334444) that is hosting the Route53 DNS zones and a test stage AWS account (ID: 999988887777) where we run the EKS cluster. Now we want to setup externalDNS for a cross account access to add/change the DNS entries hosted in the MGMT account. See below for the config files and error message we get.

Please let me know if I missed something or you need more information.

Michael

====== config details ====== When I try to setup externalDNS I get the following error:

deployment.apps/external-dns created
serviceaccount/external-dns created
clusterrole.rbac.authorization.k8s.io/external-dns created
clusterrolebinding.rbac.authorization.k8s.io/external-dns-viewer created

kubectl logs -n kube-system external-dns-XYZ
time="2020-05-28T16:14:23Z" level=info msg="config: {Master: KubeConfig: RequestTimeout:30s IstioIngressGatewayServices:[] ContourLoadBalancerService:heptio-contour/contour SkipperRouteGroupVersion:zalando.org/v1 Sources:[service ingress] Namespace: AnnotationFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:aws GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s DomainFilter:[test.local] ExcludeDomains:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType:private AWSZoneTagFilter:[] AWSAssumeRole:\"arn:aws:iam::222233334444:role/externalDNSAccessRoute53CrossAccountTest\" AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] OVHEndpoint:ovh-eu PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:upsert-only Registry:txt TXTOwnerID:\"test account\" TXTPrefix: Interval:1m0s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:debug TXTCacheInterval:0s ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host: RFC2136Port:0 RFC2136Zone: RFC2136Insecure:false RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s NS1Endpoint: NS1IgnoreSSL:false TransIPAccountName: TransIPPrivateKeyFile:}"
time="2020-05-28T16:14:23Z" level=info msg="Instantiating new Kubernetes client"
time="2020-05-28T16:14:23Z" level=debug msg="kubeMaster: "
time="2020-05-28T16:14:23Z" level=debug msg="kubeConfig: "
time="2020-05-28T16:14:23Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2020-05-28T16:14:23Z" level=info msg="Created Kubernetes client https://172.20.0.1:443"
time="2020-05-28T16:14:25Z" level=info msg="Assuming role: \"arn:aws:iam::222233334444:role/externalDNSAccessRoute53CrossAccountTest\""
time="2020-05-28T16:14:26Z" level=error msg="AccessDenied: User: arn:aws:sts::999988887777:assumed-role/externalDNSEKSrole/${ID} is not authorized to perform: sts:AssumeRole on resource: \"arn:aws:iam::222233334444:role/externalDNSAccessRoute53CrossAccountTest\"\n\tstatus code: 403, request id: ${REQUESt_ID}"

IAM Role for the test account:

Trust relationships
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::999988887777:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/${OIDC_ID}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.eu-central-1.amazonaws.com/id/${OIDC_ID}:sub": "system:serviceaccount:kube-system:external-dns",
          "oidc.eks.eu-central-1.amazonaws.com/id/${OIDC_ID}:aud": "sts.amazonaws.com"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::999988887777:root"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Attached IAM policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": [
                "arn:aws:iam::222233334444:role/externalDNSAccessRoute53CrossAccountTest
            ]
        }
    ]
}

IAM Role for the MGMT account:

Trust relationships
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::999988887777:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {}
    }
  ]
}

Attached IAM policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "route53:ChangeResourceRecordSets"
            ],
            "Resource": [
                "arn:aws:route53:::hostedzone/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "route53:ListHostedZones",
                "route53:ListResourceRecordSets"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Kubernetes YAML configuration for externalDNS:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
  namespace: kube-system
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      automountServiceAccountToken: true
      containers:
      - name: external-dns
        image: registry.opensource.zalan.do/teapot/external-dns:v0.7.1
        args:
        - --source=service
        - --source=ingress
        - --domain-filter=test.local
        - --provider=aws
        - --aws-assume-role="arn:aws:iam::222233334444:role/externalDNSAccessRoute53CrossAccountTest"
        - --policy=upsert-only
        - --aws-zone-type=private
        - --registry=txt
        - --txt-owner-id="test account"
        - --log-level=debug
      securityContext:
        fsGroup: 65534
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
  namespace: kube-system
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::999988887777:role/externalDNSEKSrole
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: [""]
  resources: ["services","endpoints","pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions"]
  resources: ["ingresses"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: kube-system