cloud-sql-proxy: failed to connect to instance: Dial error: failed to dial
Bug Description
Since we’ve upgraded from 1.*.* to 2.*.* we have been noticing ‘a lot’ of “failed to connect to instance: Dial error: failed to dial (connection name = “******”): dial tcp **.***.***.**:3307: i/o timeout” errors in the SQL Proxy container. We have been trying to debug this issue, but there doesn’t really seem to be a clear explanation for it. It is happening across multiple (PHP) applications using different frameworks. Downgrading back to 1.*.* seems to resolve the issue, which (I think) rules out that it is network related. The database is not under (very) heavy load and we’re only using ~5% of our connections limit.
Example code (or command)
...
- command:
- /cloud-sql-proxy
- --health-check
- --http-address=0.0.0.0
- --credentials-file=/secrets/cloudsql/cloudsqlproxy-credentials.json
- --max-sigterm-delay=60s
- --structured-logs
- --quiet
- toppy-***:europe-west4:toppy-***-database
image: eu.gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.2.0
imagePullPolicy: IfNotPresent
name: cloudsqlproxy-container
ports:
- containerPort: 3306
protocol: TCP
resources:
requests:
cpu: 100m
memory: 100Mi
securityContext:
allowPrivilegeEscalation: true
privileged: false
readOnlyRootFilesystem: false
runAsGroup: 65532
runAsNonRoot: true
runAsUser: 65532
startupProbe:
failureThreshold: 20
httpGet:
path: /startup
port: 9090
scheme: HTTP
periodSeconds: 1
successThreshold: 1
timeoutSeconds: 5
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /secrets/cloudsql
mountPropagation: None
name: cloudsqlproxy-service-account
readOnly: true
...
Stacktrace
n/a
Steps to reproduce?
- Not sure, included the sidecar yaml
Environment
- OS type and version: GKE 1.25.8-gke.500, Container-Optimized OS with containerd (cos_containerd)
- Cloud SQL Proxy version (
./cloud-sql-proxy --version): 2.2.0 - Proxy invocation command (for example,
./cloud-sql-proxy --port 5432 INSTANCE_CONNECTION_NAME):/cloud-sql-prox --health-check --http-address=0.0.0.0 --credentials-file=/secrets/cloudsql/cloudsqlproxy-credentials.json --max-sigterm-delay=60s --structured-logs --quiet toppy-***:europe-west4:toppy-***-database
Additional Details
- Connection is made over a public IP
Edit: updated yaml
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 32 (12 by maintainers)
I don’t want to dig up old issues, but we had this exact issue happen to us during a migration from an old dedicated server to GKE.
We had been using Google Cloud SQL for our production database for a few months now and had no issues connecting to it externally through a Public IP. So, fairly confident everything would go well, we moved our application to GKE.
This was our setup:
sqlauthproxyas the database hostEverything was fine during testing. Then we swapped our DNS to go from the old dedicated server to GKE, which caused all of our users to now go to the GKE cluster and suddenly the
sqlauthproxylogs were being spammed with:In the application this issue manifested itself as intermittent long running requests that blocked everything. For example, 20 requests would go through just fine, then 1 random request would hang for ~30 seconds, complete (successfully, so DB queries did go through), and then back to normal for the next x requests.
Monitoring showed acceptable load on the database and the containers. We debugged the entire pipeline from our ingress to nginx, php-fpm,… and quickly reverted our DNS back to the old server while we investigated the issue.
In Apache benchmark we could reproduce the issue with some simple load testing. It looked like this:
To fix:
--privateflag to the proxy container argumentsNow the benchmarks have a more acceptable distribution (50% 100ms, 100% 3000ms).
We’re going to try swapping the DNS back after running some more tests to see if this was the culprit of our botched migration or if we have to dig deeper.
tl;dr: Some networking issue between a private GKE cluster and a public SQL instance using sql auth proxy was causing ~2% of requests to time out when the application was under load, but it could not be reproduced if there was no load. Switching to private ip on the SQL instance fixed the issue in synthetic benchmarks.
@jackwotherspoon many thanks for your help, in fact it turns out that it was my mistake, if you don’t mind I’ll document here what I did.
TLDR is:
The Cloud SQL proxy is attempting to connect to the Public IP address of the Cloud SQL instance. I was unable to get to that IP from my container and edited my egress NAT policy following this. Once I added a CIDR for the Cloud SQL Public IP address (x.x.x.0/24), things started working (afaik).
Answers to @jackwotherspoon question:
I was actually referring to the DB_NAME and I was using the name of a database running on the database instance, though not initially!
Longer version:
This is what I did after the input from @jackwotherspoon: Corrected the typo in the image I was pulling for the Cloud SQL proxy. When
kubectl applying the deployment I got CrashLoopBackOff errors, this is because the 1.x.x image appears to look for acommand:, instead of anargs:entry. I spotted this in this video and this is what I used:when this finally ran I got an error message:
this convinced me (more than the 2.x.x output), that this was a connectivity issue. It was also at this point that I realised that the IP address was the public IP of the Cloud SQL instance, which I hadn’t grokked previously.
I then switched back to version 2.3.0 and when I remembered to also switch to the
argsformat, that all worked as well - so far.I wanted to know if I could get to that IP from my container, so I did:
I can’t use ping in my container so I used:
and observed that it timed out.
I then decided I needed to configure my egress.
I looked at this which then lead me to this.
I created an
egress-nat-policy.yamlas described in that final document and I defined a CIDR for x.x.x.0/24 and applied it to my cluster:At this point the
wgetfailed instantly witherror getting responseAnd my container started correctly and connected to the database.
Yep, our instance had a public IP only when we saw the issues (private was disabled). When we switched, we enabled the private IP on the instance, then added the
--private-ipflag to our cloud-sql-proxy command.I fixed it by enabling dynamic port allocation on my Cloud NAT.
The NAT had been running out of source ports and emitting these messages into Cloud Logging:
Got it. It’s unclear why public IP would cause this problem, but in any case private IP is a better path for both performance and security.
Thanks for the info @jault3. It’s helpful.
FWIW v1 proxy would try to dial public IP first and then fallback to private IP. V2 defaults to public and expects users to explicitly choose private IP (
--private-ip). Did your instance have a public IP when you saw these failures?I know this is closed, but just wanted to add that I ran into this issue as well. Hopefully the extra details help!
I am running Kong inside our GKE clusters, and it was repeatedly crashing and restarting because of failures to connect to its database. We tried increasing memory and CPU on both the Kong pods and the CloudSQL instance, increasing connection timeouts from Kong to postgres, etc. Pretty much anything we could think of to increase we tried. All the metrics we were seeing were showing minimal usage across the board, and no real blocking happening on the CloudSQL instance (both cpu and io) either. We finally tried switching to a private IP on the CloudSQL instance yesterday and the errors stopped. We were getting about 40 error logs every 30 minutes, but since 3pm yesterday we haven’t had a single one, which is great! But also confusing as to why that fixed it.
We are running the
us.gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.6.0image. We never really saw this be an issue on the v1 proxy image, but it’s been awhile since we ran that.@enocom since the problems seemed somehow ‘connection’ related I was looking into the GKE DNS settings, which now allows you to use ‘Cloud DNS’ instead of ‘kube-dns’. I’m not sure if it was the private ip option (not going through egress/ingress NAT) or DNS which in the end resolved the issue since both were applied at the same time.
@Swahjak Thanks for raising an issue on the Cloud SQL Proxy 😄
handing this over to @hessjcg who is our GKE specialist, he should be able to shed some light on this and hopefully help you get to the bottom of additional errors being seen.