cloud-sql-proxy: "Cloud SQL IAM service account authentication failed for user ..." intermittent errors when connecting to Postgres
Bug Description
Running cloud-sql-proxy sometimes it gets stuck in a state where we cant connect, returning Cloud SQL IAM service account authentication failed for user errors errors.
When this happens the proxy container must restart before it works again, i.e. retrying doesn’t help. It seems to happen if we connect multiple times in a row. E.g. start proxy, connect successfully, disconnect, connect again and it hangs but I’m not sure.
I’ve managed to reproduce it locally with the same error.
Example code (or command)
I’ve managed to reproduce with the following (note it’s rather tricky to reproduce, and might require multiple tries):
cloud-sql-proxy --port 5432 --auto-iam-authn --impersonate-service-account=<SA> <postgres instance>
# In a different terminal (password prompt doesn't matter)
psql --host 127.0.0.1 --port 5432 --username=<USER>
<Ctrl+D>
# Connect again
psql --host 127.0.0.1 --port 5432 --username=<USER>
> psql: error: connection to server at "127.0.0.1", port 5432 failed: FATAL: Cloud SQL IAM service account authentication failed for user ...
Stacktrace
psql: error: connection to server at "127.0.0.1", port 5432 failed: FATAL: Cloud SQL IAM service account authentication failed for user ...
Steps to reproduce?
See example code.
Environment
- MacOS/Linux
- Cloud SQL Proxy version: v2.9.0 and v2.11.0
- Proxy invocation command:
cloud-sql-proxy --port 5432 --auto-iam-authn --impersonate-service-account=<SA> <postgres instance>
Additional Details
I swear I saw another issue on this that was resolved in this repo, but I can’t find it anylonger.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 4
- Comments: 19 (7 by maintainers)
We have tried running versions 2.11.4 and 2.12.0 and both versions seem to suffer from something that looks a lot like this issue when a client kills and recreates connections on a regular basis .
We’ve not been able to figure out what exactly happens but we do see that if our deployments (12 pods in total) start creating a total of 250 new connections per second (done by a library we do not maintain or own) that our cloud-sql-proxy pods start throwing this error more or less constantly.
Might this be related or does this sound like a new different issue?
Release of version v2.11.3 has the new Go Connector version used and should now resolve this issue 🚀
Actually I will wait till we release this officially to close out
@otto-nordander-yubico Glad your testing went successfully 😄 Thanks so much for raising this issue, we greatly appreciate it!
I will close this out, if the issue re-surfaces feel free to re-open this.
👏 👏 @enocom
Nice, I can pull that in and run my tests to verify tomorrow!
Yes that’s a good point, here I ran with
--debug, --debug-logs. This is when running locally:After the proxy has reached this broken state, every connection try logs the following (must restart proxy for it to work again):
Edit: perhaps some helping information, when this happens I can see that the server asks the client for the password and its sent. When it’s working as expected the password is never emitted.