terraform-provider-google: google_sql_database randomly errors with failure waiting for insertion

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave “+1” or “me too” comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment
If an issue is assigned to the “modular-magician” user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to “hashibot”, a community member has claimed the issue already.

Terraform version

0.11.7

Terraform resources affected

google_sql_database

Terraform Configuration Files

resource "google_sql_database" “some_db” {
   name      = “some_db”
  instance  = "${google_sql_database_instance.master.name}"
  charset   = "UTF8"
  collation = "en_US.UTF8"
  project   = "${var.gcp_project}"
}

Debug Output

This error is completely random and very difficult to get logs for.

Panic Output

Expected Behavior

The db should have been created

Actual Behavior

apply fails

Error: Error applying plan:

1 error(s) occurred:

google_sql_database.some_db: 1 error(s) occurred:
google_sql_database.some_db: Error, failure waiting for insertion of some_db into some_db_instance:

Steps to Reproduce

terraform apply

Important Factoids

References

#0000

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 23
Comments: 25 (7 by maintainers)

Most upvoted comments

Thanks- I reached out to the team internally and they’re going to look into it. In the meantime, I’m preparing a PR that’ll add retries in more places.

danawillow on Oct 4, 2018

we are experiencing the same issue starting about 2018-10-02 14:00 UTC-7. Google Cloud SQL has consistently responded with googleapi: Error 503: Service temporarily unavailable., serverException across random different instance. Sometimes it would be 1 instance, other times 5 instances.

We have not been able to get a successful terraform plan in the last 20hrs with consistent retrying at different times of the day/night.

francisd on Oct 3, 2018

we’re suffering the same error googleapi: Error 503: Service temporarily unavailable., serverException on almost every plan run. There are no changes on our infra setup, in fact the database hasn’t been touched in a while; the problem used to be very infrequent before, now is almost blocking deployments (if we insist on running the plan multiple times we may get lucky once in a while).

Just for the record, we tried using v1.17.1 and 1.18 of the provider with very similar results.

prodriguezdefino on Oct 3, 2018

We’ve also been running into this issue non-stop for the past 2 days on existing/old google_sql_database resources.

The only work around is to add --parallelism=1 or -target plan/apply on non-sql resources.

This seems to be a major issue for many of us.

bluemalkin on Oct 3, 2018

@danawillow thanks for tackling this out! I would expect this change be included in a minor release, is there any ETA for it?

On a separate note, the root cause of this seems to be related to some instability/flakiness in the API resource which the TF resource tries to GET from in order to refresh the state, although this has happened also for different TF resources related with CloudSQL service. Is there any updates on regard of this? Maybe is hitting some sort of quota limit per IP or something else, but in any case the message could be a little bit more descriptive than Service temporarily unavailable.

prodriguezdefino on Oct 4, 2018