external-dns: Unable to use IAM Service Account on GKE
I’m seeing the following in the logs for my external-dns pod on GKE:
time="2018-03-29T00:57:30Z" level=info msg="config: &{Master: KubeConfig: Sources:[service ingress] Namespace: AnnotationFilter: FQDNTemplate: Compatibility: PublishInternal:false Provider:google GoogleProject:MY-PROJECT DomainFilter:[MY.MANAGED.ZONE] AWSZoneType: AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: CloudflareProxied:false InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InMemoryZones:[] Policy:upsert-only Registry:txt TXTOwnerID:default TXTPrefix:external-dns Interval:1m0s Once:false DryRun:false LogFormat:text MetricsAddress::7979 LogLevel:debug}"
time="2018-03-29T00:57:30Z" level=info msg="Connected to cluster at https://10.55.240.1:443"
time="2018-03-29T00:57:30Z" level=error msg="googleapi: Error 403: Forbidden, forbidden"
time="2018-03-29T00:58:30Z" level=error msg="googleapi: Error 403: Forbidden, forbidden"
time="2018-03-29T00:59:30Z" level=error msg="googleapi: Error 403: Forbidden, forbidden"
time="2018-03-29T01:00:30Z" level=error msg="googleapi: Error 403: Forbidden, forbidden"
I created an issue for this in the charts repo, but after doing so, I thought it might be better to raise the issue here. Feel free to close this one if it’s the wrong place, or let me know if it’s the right place and I’ll close the other ticket.
Helm version:
"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844"
Kubernetes version:
1.9.4-gke.1
Installed via:
Helm chart stable/external-dns
Args:
(as passed through the helm chart to the container, obtained via kubectl describe pod foo)
--log-level=debug
--domain-filter=MY.MANAGED.ZONE
--policy=upsert-only
--provider=google
--txt-prefix=external-dns
--source=service
--source=ingress
--registry=txt
--google-project=MY-PROJECT
Credentials:
Set via env var: GOOGLE_APPLICATION_CREDENTIALS: /etc/secrets/service-account/credentials.json
Credentials obtained from downloaded JSON file while creating a service account in Cloud Console Web UI.
Docker image:
registry.opensource.zalan.do/teapot/external-dns:v0.4.8
Helm command:
helm upgrade\
--install\
--recreate-pods\
--namespace=kube-system\
--set domainFilters[0]="MY.MANAGED.ZONE"\
--set extraArgs.registry=txt\
--set logLevel=debug\
--set provider=google\
--set google.serviceAccountSecret=external-dns\
--set google.project=MY-PROJECT\
--set txtPrefix="external-dns"\
external-dns stable/external-dns
I am able to kubectl exec my way onto the pod and verify that the file /etc/secrets/service-account/credentials.json file is in place. In troubleshooting, I’ve granted the service account full owner permissions across the entire project, and it doesn’t seem to have had any effect.
Steps to repro, as best as I can figure:
- Create a GCP project
- Create a managed zone in Cloud DNS (not sure if this part is strictly necessary to trigger the behavior)
- Create a GKE cluster (K8S version:
1.9.4-gke.1) - Create a GKE node pool (K8S version:
1.9.4-gke.1) - Login to the cluster
- Create an ingress which matches your managed zone (not sure if this part is strictly necessary to trigger the behavior)
- Create an IAM service account.
- Grant the IAM service account full project owner permissions.
- Create a K8S secret with a data key of
credentials.jsonand its value as the JSON object downloaded from the IAM service account creation dialog. - Run
helm initto install tiller in the cluster. - Run
helm repo updateto get the latest version of the chart (I installed0.5.2) - Run
helm upgradecommand I put further up in the issue. - Run
kubectl logs -f THE_POD_NAMEto see the error above.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 4
- Comments: 27 (7 by maintainers)
Hi all,
I’m using the linked helm chart with GCE and a service account without any issue.
I’m supplying the helm chart with the following values:
Where domain.tld is a secret with key: credentials.json and value my downloaded service account credentials.
The container boots up, launches the external-dns binary, it has the following environment variables:
If I cat out
/etc/secrets/service-account/credentials.jsonI get back the credentials I submitted via the secret.My logs state the following:
After wasting a number of hours on this issue I managed to get it working without any overly generous permissions.
terraform-google-modules/kubernetes-engine/google//modules/beta-private-clusterwithworkload-identityInstall external-dns based on the RBAC manifest here
Add policy binding and service annotation
Note:
If your dns belongs to a different project, then manually create the service account in the other project and assign
DNS administratoraccess.I may have more information pertaining to this issue.
I am able to use cloud-dns secret key linked to an serviceAccount with
--role roles/dns.adminpermission configured on external-dns and cert-manager inus-east1. 👍However, I am unable to do this on
europe-west1for example and the node scope role"https://www.googleapis.com/auth/ndev.clouddns.readwrite"is necessary to make it work on the region otherwise with the credentials.json secret set just like in theus-east1region I get thegoogleapi: Error 403: Forbidden, forbidden"Just wanted to chime into the thread because it helped me and if someone sees this, don’t worry you are not going crazy. This is something that the Google Cloud engineers have either added for extra security or the cloud-dns API authorization is currently broken in certain regions.
This seems to be a problem with IAM roles and permission in GKE/Google Cloud for certain regions. 👈 💔
Thanks for the clear steps @prabhu, I had trouble running this in a different namespace other than default and workload identity. I did the steps you mentioned from scratch and it worked as expected. Not sure if there may be some hard coded stuff regarding the token in this image:
registry.opensource.zalan.do/teapot/external-dns:latestWithout using helm, I encountered this
Error 403: Forbidden, forbiddenerror on GKE 1.10.5-gke.0 if my IAM service account wasn’t set up correctly with the project role.To reproduce:
If I deployed external-dns now, I see these errors in the log:
Assign the project
ownerrole:Now it works:
PS I also made the mistake of using
gcloud iam service-account add-iam-policy-bindingwhich treats the service account as a resource, not an identity.Any updates on this issue? We’re also seeing the same 403 error when deploying ExternalDNS to a GKE cluster (following the guide above, we’re not using Helm).