cloud-sql-proxy: "Cloud SQL IAM service account authentication failed for user ..." intermittent errors when connecting to Postgres

Bug Description

Running cloud-sql-proxy sometimes it gets stuck in a state where we cant connect, returning Cloud SQL IAM service account authentication failed for user errors errors.

When this happens the proxy container must restart before it works again, i.e. retrying doesn’t help. It seems to happen if we connect multiple times in a row. E.g. start proxy, connect successfully, disconnect, connect again and it hangs but I’m not sure.

I’ve managed to reproduce it locally with the same error.

Example code (or command)

I’ve managed to reproduce with the following (note it’s rather tricky to reproduce, and might require multiple tries):

cloud-sql-proxy --port 5432 --auto-iam-authn --impersonate-service-account=<SA> <postgres instance>

# In a different terminal (password prompt doesn't matter)
psql --host 127.0.0.1 --port 5432 --username=<USER>

<Ctrl+D>
# Connect again
psql --host 127.0.0.1 --port 5432 --username=<USER>

> psql: error: connection to server at "127.0.0.1", port 5432 failed: FATAL:  Cloud SQL IAM service account authentication failed for user ...

Stacktrace

psql: error: connection to server at "127.0.0.1", port 5432 failed: FATAL:  Cloud SQL IAM service account authentication failed for user ...

Steps to reproduce?

See example code.

Environment

  1. MacOS/Linux
  2. Cloud SQL Proxy version: v2.9.0 and v2.11.0
  3. Proxy invocation command: cloud-sql-proxy --port 5432 --auto-iam-authn --impersonate-service-account=<SA> <postgres instance>

Additional Details

I swear I saw another issue on this that was resolved in this repo, but I can’t find it anylonger.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 4
  • Comments: 19 (7 by maintainers)

Most upvoted comments

We have tried running versions 2.11.4 and 2.12.0 and both versions seem to suffer from something that looks a lot like this issue when a client kills and recreates connections on a regular basis .

We’ve not been able to figure out what exactly happens but we do see that if our deployments (12 pods in total) start creating a total of 250 new connections per second (done by a library we do not maintain or own) that our cloud-sql-proxy pods start throwing this error more or less constantly.

Might this be related or does this sound like a new different issue?

Release of version v2.11.3 has the new Go Connector version used and should now resolve this issue 🚀

Actually I will wait till we release this officially to close out

@otto-nordander-yubico Glad your testing went successfully 😄 Thanks so much for raising this issue, we greatly appreciate it!

I will close this out, if the issue re-surfaces feel free to re-open this.

👏 👏 @enocom

The latest release of the Go Connector has a fix: https://github.com/GoogleCloudPlatform/cloud-sql-go-connector/releases/tag/v1.10.1.

Nice, I can pull that in and run my tests to verify tomorrow!

One thing you could try is enable debug logging which would help us understand if the background refresh is behaving as expected. It’s possible the token we’re sending to the server is expired somehow. The other possibility is that the backend IAM check is failing for some other reason.

Yes that’s a good point, here I ran with --debug, --debug-logs. This is when running locally:

cloud-sql-proxy version 2.11.0+darwin.arm64
2024/05/14 20:51:13 Impersonating service account with Application Default Credentials
2024/05/14 20:51:13 [xxxxx] Connection info added to cache
2024/05/14 20:51:13 [xxxxx] Listening on 127.0.0.1:5432
2024/05/14 20:51:13 The proxy has started successfully and is ready for new connections!
2024/05/14 20:51:13 [xxxxx] Connection info refresh operation started
2024/05/14 20:51:13 Enabling pprof endpoints at localhost:9091
2024/05/14 20:51:14 [xxxxx] Accepted connection from 127.0.0.1:55371 <--- me connecting is successfully
2024/05/14 20:51:14 [xxxxx] Connection info refresh operation started
2024/05/14 20:51:15 [xxxxx] Connection info refresh operation complete
2024/05/14 20:51:15 [xxxxx] Current certificate expiration = 2024-05-14T19:51:14Z
2024/05/14 20:51:15 [xxxxx] Connection info refresh operation scheduled at 2024-05-14T19:47:14Z (now + 56m0s)
2024/05/14 20:51:15 [xxxxx] Now = 2024-05-14T18:51:15Z, Current cert expiration = 2024-05-14T19:51:14Z
2024/05/14 20:51:15 [xxxxx] Cert is valid = true
2024/05/14 20:51:15 [xxxxx] Dialing x.x.x.x:3307
2024/05/14 20:51:16 [xxxxx] Connection info refresh operation complete
2024/05/14 20:51:16 [xxxxx] Current certificate expiration = 2024-05-14T19:51:15Z
2024/05/14 20:51:16 [xxxxx] Connection info refresh operation scheduled at 2024-05-14T19:47:15Z (now + 56m0s)
2024/05/14 20:51:16 [xxxxx] client closed the connection <---- me exiting the first connection
2024/05/14 20:51:17 [xxxxx] Accepted connection from 127.0.0.1:55387  <---- me opening new connection
2024/05/14 20:51:17 [xxxxx] Now = 2024-05-14T18:51:17Z, Current cert expiration = 2024-05-14T19:51:15Z
2024/05/14 20:51:17 [xxxxx] Cert is valid = true
2024/05/14 20:51:17 [xxxxx] Dialing x.x.x.x:3307
2024/05/14 20:51:18 [xxxxx] client closed the connection <---- this is because psql failed to connect

After the proxy has reached this broken state, every connection try logs the following (must restart proxy for it to work again):

2024/05/14 20:58:07 [xxxxx] Accepted connection from 127.0.0.1:57119
2024/05/14 20:58:07 [xxxxx] Now = 2024-05-14T18:58:07Z, Current cert expiration = 2024-05-14T19:54:49Z
2024/05/14 20:58:07 [xxxxx] Cert is valid = true
2024/05/14 20:58:07 [xxxxx] Dialing x.x.x.x:3307
2024/05/14 20:58:08 [xxxxx] client closed the connection

Edit: perhaps some helping information, when this happens I can see that the server asks the client for the password and its sent. When it’s working as expected the password is never emitted.