cloud-sql-proxy: TLS Handshake failures with Go 1.23
Bug Description
When attempting to connect through the proxy, I’m getting connection reset by peer
Starting the proxy is fine
2024/09/12 08:58:14 Authorizing with Application Default Credentials
...
2024/09/12 08:58:15 [redacted-prod:europe-west1:redacted-live-773ed67c] Listening on /tmp/redacted-prod:europe-west1:redacted-live-773ed67c
2024/09/12 08:58:15 [redacted-prod:europe-west1:redacted-live-1c3b6da9] Listening on /tmp/redacted-prod:europe-west1:redacted-live-1c3b6da9
2024/09/12 08:58:15 [redacted-prod:europe-west1:redacted-live-6aab2946] Listening on /tmp/redacted-prod:europe-west1:redacted-live-6aab2946
2024/09/12 08:58:15 [redacted-prod:europe-west1:redacted-live-23a55252] Listening on /tmp/redacted-prod:europe-west1:redacted-live-23a55252
2024/09/12 08:58:15 The proxy has started successfully and is ready for new connections!
2024/09/12 08:58:47 [redacted-prod:europe-west1:redacted-live-156c5af9] Accepted connection from
but when I attempt to make the actual connection, I get
2024/09/12 08:58:47 [redacted-up-prod:europe-west1:redacted-live-6aab2946] failed to connect to instance: Dial error: handshake failed (connection name = "redacted-up-prod:europe-west1:redacted-live-6aab2946"): read tcp 172.22.164.170:53244->redacted:3307: read: connection reset by peer
In the client (Datagrip 2024.2.1) I see the error Socket is not connected.
On the CloudSQL side, I can also see a warning log in Logs explorer like 1893634 [Warning] [MY-000000] [Server] Failed to connect to mysql: 1893634. Server-side error number: 1159
Stacktrace
Output from tcpdump
tcpdump: data link type PKTAP
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type PKTAP (Apple DLT_PKTAP), snapshot length 524288 bytes
08:58:47.153440 IP 172.22.164.170.53244 > redacted.3307: Flags [S], seq 2094967633, win 65535, options [mss 1360,nop,wscale 6,nop,nop,TS val 1730041868 ecr 0,sackOK,eol], length 0
08:58:47.216068 IP redacted.3307 > 172.22.164.170.53244: Flags [S.], seq 542659392, ack 2094967634, win 32384, options [mss 1332,sackOK,TS val 3082097024 ecr 1730041868,nop,wscale 7], length 0
08:58:47.216124 IP 172.22.164.170.53244 > redacted.3307: Flags [.], ack 1, win 2062, options [nop,nop,TS val 1730041931 ecr 3082097024], length 0
08:58:47.216806 IP 172.22.164.170.53244 > redacted.3307: Flags [.], seq 1:1321, ack 1, win 2062, options [nop,nop,TS val 1730041931 ecr 3082097024], length 1320
08:58:47.216810 IP 172.22.164.170.53244 > redacted.3307: Flags [P.], seq 1321:1486, ack 1, win 2062, options [nop,nop,TS val 1730041931 ecr 3082097024], length 165
08:58:47.264582 IP redacted.3307 > 172.22.164.170.53244: Flags [R.], seq 1, ack 1321, win 2062, length 0
Steps to reproduce?
This happens every time I try connecting though the terminal
Environment
- OS type and version: macos 14.6.1
- Cloud SQL Proxy version 2.13
- Proxy invocation command:
cloud-sql-proxy --config-file ~/cloudsql/cloudsql-config.toml
Content of cloudsql-config.toml
#CloudSQL proxy config
unix-socket = "/tmp"
private-ip = true
...
instance-connection-name-30 = "redacted-prod:europe-west1:redacted-live-156c5af9"
instance-connection-name-31 = "redacted-prod:europe-west1:redacted-live-773ed67c"
instance-connection-name-32 = "redacted-prod:europe-west1:redacted-live-1c3b6da9"
instance-connection-name-33 = "redacted-prod:europe-west1:redacted-live-6aab2946"
instance-connection-name-34 = "redacted-prod:europe-west1:redacted-live-23a55252"
Additional Details
This happens only when I setup the proxy using the terminal. If I use Intellij/Datagrip, and use the built in MySQL CloudSql proxy driver, I can connect without issues.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 23 (5 by maintainers)
Yes!
GODEBUG=tlskyber=0indeed works with 2.13, i.e. the error goes away and I can connect just as when downgrading to v2.11.3.I’m experiencing the same issue when trying to connect through the proxy. I receive a “connection reset by peer” error during the handshake process. Here’s a snippet from my logs:
This issue occurs consistently (direct access & via python’s connectors), I have tried reinitializing from my main device & my alt but nothing seems to work. Other users (on same project, with the same permissions) are not facing this issue even though they are following the same connection procedure & running the same OS with the latest updates.
UPDATE: Downgrading to v2.11.3 seems to have solved the issue for now as a work around, will try to identify the version where the issue started
UPDATE 2: After some git bisecting the commit that started the issue was
fd6bd49242c884508f641c754eb5cec5d28ac665with the bump to go 1.23, reverting the version solves the issue@dan-lind Great to hear it!
I am going to look at open Go issue’s and see if I need to create one as it seems like the new default preferences implemented in https://github.com/golang/go/issues/67061 are not playing nicely with VPNs
Confirming that downgrading to v2.11.3 solve the issue for me as well!
@TokieSan Thanks for the detailed analysis and extra data points! I would recommend pinning to an older version for now, in the meantime I will investigate this further to see if there is maybe something on the server side that is causing issues with the latest Go 1.23 TLS changes.
We do yes, I’ve reached out to them referring to this issue.