cloud-sql-go-connector: Periodic Cloud SQL IAM service account authentication failed errors
Question
Periodically, we see the following error:
server error (FATAL: Cloud SQL IAM service account authentication failed for user “<some_user>>@<some_account>.iam” (SQLSTATE 28000))
This happens in background jobs that are in the process of being shut down. Do you have a suggestion on how to avoid this situation in GKE?
We’ve never experienced this specific issue when using passwords, so I presume there’s some interplay between the app, its environment (GKE), and how IAM is handled by the pods when it receives a shutdown signal.
We’re setting up the token sources as described in WithIAMAuthNTokenSources and then cloudsqlconn.WithPrivateIP() along with the instance’s connection name.
Code
No response
Additional Details
cloud.google.com/go/cloudsqlconn v1.9.0
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (8 by maintainers)
For brevity, we ended up catching
SQLSTATE 28000(invalid_authorization_specification) errors and retried them with a backoff when attempting to begin a transaction.Pseudo:
Minor update, it does appear to be working as intended. We’ll continue to monitor this. But I think we should keep this issue open because I suspect this will happen again and hopefully we’ll get more details on the underlying error.
Thank you for your support.