istio: Can't connect to Google Cloud Sql

I confirm that solution suggested by @veryhumble works.

It’s actually documented at https://istio.io/blog/2018/egress-tcp/

Here is the complete configuration that works for me:

# Cloud SQL Proxy makes requests to `www.googleapis.com`
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: www.googleapis.com
  namespace: istio-example-cloud-sql
spec:
  hosts:
  - www.googleapis.com
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
  location: MESH_EXTERNAL
---
# see https://istio.io/blog/2018/egress-tcp/
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: cloud-sql-instance
  namespace: istio-example-cloud-sql
spec:
  hosts:
  # although blog article says that `hosts` field is ignored for TCP service entries,
  # clients fail to establish connection when `hosts` is omitted, 
  # namely "curl: (56) Recv failure: Connection reset by peer"
  #
  # use `gcloud sql instances list` to find out the IP address of your Google Cloud Instance
  - X.X.X.X
  addresses:
  # use `gcloud sql instances list` to find out the IP address of your Google Cloud Instance
  - X.X.X.X/32 # a block of IPs in CIDR notation
  ports:
  - name: tcp
    number: 3307 # at the moment, Google Cloud SQL always available on port 3307
    protocol: tcp # enable TCP traffic
  location: MESH_EXTERNAL

Using the cloudsql proxy does provide the benefit of automating the SSL cert setup to your SQL instance vs directly connecting to private IP.

I’ve been using cloudsql proxy w/ istio 1.4+ for a while now as both a deployment + sidecar approach, one of the problems that i’ve seen is the race condition during startup that leads to the connection refused errors for the proxy trying to talk to google APIs during startup.

The workaround i’ve been using until sidecars KEP is available is to simply use shell/curl to wait until the proxy is ready before starting up cloud_sql_proxy (rather then just adding a sleep 10 delay)

Unless you are running with a restricted mesh REGISTRY_ONLY you then don’t need to create any ServiceEntry unless you want additional observability context.

  command: ["/bin/sh", "-c"]
  args:
  - |
    while ! curl -s -f http://127.0.0.1:15020/healthz/ready; do sleep 5; done
    exec /cloud_sql_proxy -dir=/cloudsql -term_timeout=180s
  env:
  - name: INSTANCES
    value: PROJECT:REGION:INSTANCE=tcp:0.0.0.0:3306

Note: Recent 1.16+ proxy releases have moved to distroless/no shell so you either need to build your custom image until https://github.com/GoogleCloudPlatform/cloudsql-proxy/issues/371 or use a older image.

FROM alpine:3.11.6

ENV FILE="https://storage.googleapis.com/cloudsql-proxy/v1.17/cloud_sql_proxy.linux.amd64"
RUN apk add --no-cache curl && \
    curl -SL --output /cloud_sql_proxy "$FILE" && \
    chmod +x /cloud_sql_proxy

CMD ["/cloud_sql_proxy"]

This should be kept open; there’s no solution to it.

sigh

This issue shouldn’t be closed as there is no solution yet. Can someone (@vadimeisenbergibm ?) reopen?

I’m currently stuck with the https://www.googleapis.com/sql/v1beta4/projects/pc-internal-dev/instances/pc-internal-dev-1?alt=json&prettyPrint=false: oauth2: cannot fetch token: Post https://oauth2.googleapis.com/token: dial tcp 172.217.212.95:443: connect: connection refused issue.

I’m using Istio 1.2-release on 1.13.11-gke.9

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: www.googleapis.com
spec:
  hosts:
  - www.googleapis.com
  - oauth2.googleapis.com
  - accounts.google.com
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: cloud-sql-instance
spec:
  hosts:
  - www.googleapis.com
  ports:
  - name: tcp
    number: 3306
    protocol: tcp
  addresses:
  - 104.x.x.x
  endpoints:
  - address: 104.x.x.x
  location: MESH_EXTERNAL
  resolution: STATIC

I’m also using mTLS. I’m not currently using the private IP approach - but if I we’re, since I’m using mTLS in Istio, and my CloudSQL is using SSL - I’m not sure the best way to set this up?

Any update on this? I am experiencing the same problem using the cloudsql-proxy sidecar.

Edit: I got the connection somehow working. I added a sidecar with the postgres client and tried connecting from there manually. In the logs of the cloudsql-proxy i saw that it’s trying to connect to the postgresql db using port 3307. So I added the following ServiceEntry:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: cloudsql-external
spec:
  hosts:
  - IP_ADDRESS
  addresses:
  - IP_ADDRESS/32
  ports:
  - name: tcp
    number: 3307
    protocol: tcp
  location: MESH_EXTERNAL

It now prompts correctly the password after psql -h 127.0.0.0 -U proxyuser -d test . But its stuck on that password prompt. So not quite resolved yet.

Leaving a checklist that probably will save few hours for someone like me.

1. Check that you have containerPort defined:

ports:
- containerPort: 5432

1. Check that service port is defined, targets to correct container port, and starts with tcp:

type: ClusterIP # check
spec:
  ports:
  - name: tcp
    protocol: TCP
    port: 5432       # check
    targetPort: 5432 # check

1. Check binding address and port for cloudsql binary:

containers:
- command:
  - /cloud_sql_proxy
  - -instances=PROJECT:COMPUTE_ZONE:INSTANCE=tcp:0.0.0.0:5432 # check this: tcp:0.0.0.0: won't produce any error.
  - -credential_file=/secrets/cloudsql/credentials.json
  - -term_timeout=30s
  - -use_http_health_check  # Since 1.25.0
  - -health_check_port=8090 # See: https://github.com/GoogleCloudPlatform/cloudsql-proxy/blob/main/examples/k8s-health-check/proxy_with_http_health_check.yaml

UPDATE: I got it working with a slightly different configuration using jdbc Socket Factory connecting to CloudSQL Mysql 2.Gen.

Difference to @yskopets @veryhumble configuration:

• No “Adresses field” /CIDR block in cloud-sql-instance service entry
• Resolution: DNS for the service entry that allows the connection to the CloudSQL Instance

# Pod makes requests to `www.googleapis.com` for Authentication
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: www.googleapis.com
spec:
  hosts:
  - "*.googleapis.com"
  - 169.254.169.254 # metadata.google.internal   -> IP was being logged in istio-proxy with 404
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  location: MESH_EXTERNAL

---
# jdbc connection to CloudSQL instance
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: cloud-sql-instance
#  namespace: istio-example-cloud-sql
spec:
  hosts:
  # use `gcloud sql instances list` to find out the IP address of your Google Cloud Instance
  - 104.155.22.115
  ports:
  - name: tcp
    number: 3307 # at the moment, Google Cloud SQL always available on port 3307
    protocol: tcp # enable TCP traffic
  location: MESH_EXTERNAL
  resolution: DNS

@vadimeisenbergibm I can confirm this works without the istio-proxy as a sidecar. All the proposed workarounds are using ServiceEntry to allow this URLs, but if the outboundTrafficPolicy.mode is set to ALLOW_ANY, why are all this steps needed?

Is there another way to tell the sidecar to allow all egress traffic?

If it helps the ones that are stuck; using TCP connection with cloud_sql_proxy worked for me:

        - name: cloudsql-proxy
          command: [
              "-instances=<instance-name>=tcp:5432",
              "-ip_address_types=PRIVATE",
              "-credential_file=/secrets/cloudsql/credentials.json",
            ]

But removing the TCP option and trying to use Unix Socket, always failed:

        - name: cloudsql-proxy
          command: [
              "/cloud_sql_proxy",
              "-dir=/cloudsql",
              "-instances=<instance-name>",
              "-ip_address_types=PRIVATE",
              "-credential_file=/secrets/cloudsql/credentials.json",
            ]

With this error on the Cloud SQL logs:

Get https://www.googleapis.com/sql/v1beta4/projects/<instance-name>?alt=json&prettyPrint=false: oauth2: cannot fetch token: Post https://oauth2.googleapis.com/token: dial tcp 74.125.141.95:443: connect: connection refused

Using both; private or public IP and with any ServiceEntry combination given on this post so far didn’t work while using Unix Socket

@Kampe I don’t think listing IP address in hosts field is possible (istio version 1.1.8), how did you get around that. also do you have the case where you enable SSL on cloudSQL side? did you manage to get that done somehow?

@ermik The first question is if you need to direct the egress traffic thru a gateway, or you can just direct the traffic directly from the sidecar.

If you need the gateway, and you want to use wildcard hosts like *.googleapis.com, see this example how to do it: https://preliminary.istio.io/docs/examples/advanced-gateways/wildcard-egress-hosts/