cloud-sql-go-connector: Periodic Cloud SQL IAM service account authentication failed errors

Question

Periodically, we see the following error:

server error (FATAL: Cloud SQL IAM service account authentication failed for user “<some_user>>@<some_account>.iam” (SQLSTATE 28000))

This happens in background jobs that are in the process of being shut down. Do you have a suggestion on how to avoid this situation in GKE?

We’ve never experienced this specific issue when using passwords, so I presume there’s some interplay between the app, its environment (GKE), and how IAM is handled by the pods when it receives a shutdown signal.

We’re setting up the token sources as described in WithIAMAuthNTokenSources and then cloudsqlconn.WithPrivateIP() along with the instance’s connection name.

Code

No response

Additional Details

cloud.google.com/go/cloudsqlconn v1.9.0

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (8 by maintainers)

Most upvoted comments

For brevity, we ended up catching SQLSTATE 28000 (invalid_authorization_specification) errors and retried them with a backoff when attempting to begin a transaction.

Pseudo:

if pgErr := (*pgconn.PgError)(nil); errors.As(err, &pgErr) {
	if pgErr.Code == pgerrcode.InvalidAuthorizationSpecification{
		// This is an invalid authorization error. Retry.
	}
	// Handle all other errors.
}

Minor update, it does appear to be working as intended. We’ll continue to monitor this. But I think we should keep this issue open because I suspect this will happen again and hopefully we’ll get more details on the underlying error.

Thank you for your support.

Connection info refresh operation scheduled at 2024-05-16T13:20:46Z (now + 56m0s)
Connection info refresh operation complete
Connection info refresh operation started
Connection info added to cache
Connection info refresh operation scheduled at 2024-05-16T13:20:25Z (now + 56m0s)
Connection info refresh operation complete
Connection info refresh operation started
Connection info added to cache