cloud-sql-proxy: Memory leak when connecting from proxy

I’m not sure if this is a proxy problem but it’s likely. It doesn’t seem to affect my servers (for the most part), but my Cloud SQL log is flooded by these, nonstop. This has been the case for quite a while (months).

There are this discussion, this one too, this issue in Go SQL driver discussing similar things. There’s a buganizer issue (can’t find it now) that claims to fix it but doesn’t.

Wanted to open an issue so that it can hopefully be investigated.

Full log:

{
 insertId:  "*"   
 logName:  "projects/*/logs/cloudsql.googleapis.com%2Fmysql.err"   
 receiveTimestamp:  "2017-10-10T00:21:54.291672943Z"   
 resource: {
  labels: {
   database_id:  "*"     
   project_id:  "*"     
   region:  "us-west1"     
  }
  type:  "cloudsql_database"    
 }
 severity:  "ERROR"   
 textPayload:  "2017-10-10T00:21:48.435386Z 705912 [Note] Aborted connection 705912 to db: '*' user: '*' host: 'cloudsqlproxy~*' (Got an error reading communication packets)"   
 timestamp:  "2017-10-10T00:21:48.435860Z"   
}

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 4
Comments: 53 (20 by maintainers)

Most upvoted comments

We’re seeing similar issues which are causing production issues, I wonder what the best approach is since this is quite critical to always work.

RobinUS2 on Jun 6, 2018

@AthenaShi hit this issue today AND yesterday – but it seems to resolve itself? We currently have higher tier google support – can you help us / look at our instances? - update: sent email

savraj on Jan 23, 2018

We’ve confirmed that the root cause of the “error” is a MySQL memory leak on our side with connections from proxy (which includes proxy client and GAE), the fix is checked in.

However we’re in prod freeze now, and the rollout won’t be scheduled until Nov 28 with ETA by end of this year.

Same time there’re some methods to mitigate the issue:

Try connect through other way (ip, etc), the leak is particular via proxy
Connect via proxy with some connection pooling which would improve it.
Or notify Cloud SQL to reboot mysqld (or the instance) to make it temporarily work.

Sorry again and hope this would help.

AthenaShi on Nov 17, 2017

FYI, a Public Issue Tracker have been opened by my current Google Support contact to follow up the release of the Cloud SQL memory leak fix https://issuetracker.google.com/issues/73649904

pdecat on Feb 26, 2018

So there’s something going on with cloud sql - it may be the proxy, or it may be the database itself. I don’t know what part exactly is having issue, and hence I’m hoping someone in the cloud sql team can help identify.

All I know is that:

This is not an isolated issue as many others seem to have the same problem on GCP.
This doesn’t happen on AWS, at least not for me.

I understand that it’s an error thrown from mysql. Why mysql and cloud sql doesn’t play nice with each other is the question here (and in particular second gen).

danqing on Oct 20, 2017

The rollout is done for all instances now.

If you still see problems please send your instance name to:cloud-sql@google.com so I can take a look from internal.

AthenaShi on Jan 12, 2018

I see, I do see your update on one of our public buganizer problem. That one is for connection from GAE, I don’t believe its your case since you connect directly from the proxy client.

If you don’t mind, would you please send an email with following information to cloud-sql@google.com so I may be able to help out looking at some internal data.

with a link to this page
the full name of your cloud sql instance.
the command your run or code snippet to connect through Cloud SQL proxy

AthenaShi on Oct 20, 2017