cloud-sql-go-connector: refresh failed: context deadline exceeded

Bug Description

As previously stated on https://github.com/GoogleCloudPlatform/cloud-sql-go-connector/issues/381 and https://github.com/GoogleCloudPlatform/cloud-sql-go-connector/issues/370, we see this error from time to time:

rpc error: code = Internal desc = database read error: failed to connect to `host=localhost user=xxx@xxx.iam database=db`: dial error (timeout: refresh failed: context deadline exceeded)

And when it happens, the service seems keep being in this “error state” once in a while.

This graph shows how often it could happen for us: image

Example code (or command)

How we setup the connection:

        driver := "cloudsql-postgres"

	options := make([]cloudsqlconn.Option, 0)
	options = append(options, cloudsqlconn.WithIAMAuthN())
	options = append(options, cloudsqlconn.WithDefaultDialOptions(cloudsqlconn.WithPrivateIP()))

	cleanup, err := pgxv4.RegisterDriver(driver, options...)
	if err != nil {
		return nil, nil, fmt.Errorf("could not register CloudSQL driver: %w", err)
	}

	dbHost := fmt.Sprintf("%s:%s:%s", cfg.CloudSQL.GCPProject, cfg.CloudSQL.DBRegion, cfg.CloudSQL.InstanceName)
	dbUser := strings.TrimSuffix(cfg.CloudSQL.User, ".gserviceaccount.com")
	connStr := fmt.Sprintf(
		"host=%s user=%s dbname=%s sslmode=disable",
		dbHost,
		dbUser,
		cfg.CloudSQL.DBName,
	)

	db, err := sql.Open(driver, connStr)
	db.SetMaxOpenConns(2)
	db.SetMaxIdleConns(2)
	db.SetConnMaxLifetime(time.Hour)
...

Stacktrace

We don’t have strackrace available now but we logged the context deadline before the code calls onto db to do a simple read. This is what we could see:

ctx deadline left before calling db: 9962 ms

So this means there is around 10s still available but then the call to db failed with

failed to connect to host=localhost user=xxx@xxx.iam database=db: dial error (timeout: refresh failed: context deadline exceeded)

How to reproduce

That is difficult to say. Run a long-running service, randomly do simple db read, this shows up from time to time

Environment

  1. OS type and version: google Cloudrun
  2. Go version: go version v1.19.3
  3. Connector version: v1.0.1

Thank you, and if you need any other info we could provide, let us know.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 22 (10 by maintainers)

Most upvoted comments

Thanks for all the extra info folks. Let me spend some time with this and see what’s going on here. I’ll report back here.

That’s good to hear. As usual, feel free to open an issue if you see something not working well.

Thanks a lot, I was just about to ask whether it could be related to the rate limiter as the “only context” was thrown into the wait methods of the limiter immediately after the context was created 😃

Sorry for the slow response here.

Inside our refresh code, we have a single context with a timeout: https://github.com/GoogleCloudPlatform/cloud-sql-go-connector/blob/main/internal/cloudsql/refresh.go#L309.

If we’re seeing the error refresh failed: context deadline exceeded, it means that context is timing out while waiting for Admin API calls to complete. By default, the timeout is set to 30 seconds. It seems surprising that the Admin API could not complete an HTTP response in 30 seconds. What might be happening is that the CPU is either throttled (as in Cloud Run) or maxed out (as in GKE).

The GKE angle is easier to debug. So let’s start there and then return to Cloud Run. When you see the GKE error, what kind of CPU usage do you see for the Proxy? Typically we recommend 1 CPU and 2 GiB of memory, but throughput and number of connections will affect those numbers.

Finally, the Go Connector supports tracing. It would be very helpful to capture one of these context deadline exceeded errors in a trace. That would provide some strong evidence towards the root cause. See https://github.com/GoogleCloudPlatform/cloud-sql-go-connector#enabling-metrics-and-tracing.

Meanwhile, I’ll try to reproduce this myself.

So one option would be adding support for a refresh option that changes all refreshes to be lazy. This would increase connection latency but would at least allow clients to have a way to avoid these CPU throttling issues.

Thanks for the update @liufuyang. I’m mistaking minScale for CPU throttling. And we can leave this open.

It should work to just get the connection a second time if the first one fails - the 2nd will block on the API request completing.

Thank you for the feedback. We used to have minscale set as 0 but for dealing with this issue we have previously set minscale as 1 (still one for now) but it seems not fully resolving it (maybe it reduces the frequency I am not sure).

But we haven’t tested with turning of the CPU throttling yet. It has always been on. We were thinking if we turn it off, it bring some extra cost, but I guess we can give it a try as well.

Thanks for the bug report @liufuyang. This might be related to how Cloud Run throttles the CPU when the service goes to sleep. We’ve made some improvements in the Java connector that we can probably make here as well.

Does your Cloud Run service scale to 0?