terraform-provider-digitalocean: 429 Too many requests or 429 API Rate limit exceeded

Terraform Version

Terraform v0.13.5
+ provider registry.terraform.io/-/aws v3.22.0
+ provider registry.terraform.io/-/digitalocean v2.3.0
+ provider registry.terraform.io/digitalocean/digitalocean v2.3.0
+ provider registry.terraform.io/hashicorp/aws v3.22.0

Expected Behavior

Terraform to complete successfully

Actual Behavior

Terraform is getting errors back from DO API stating either Too Many Requests or API Rate limit exceeded.

Steps to Reproduce

Manage 100 domain records and it will fail while trying to refresh the state.

Important Factoids

I am managing 10 domains with about 10-20 records each. I have dropped parallelism down to 1 and I still can’t get a successful run without hitting one of these two errors. This isn’t an absurdly large terraform run, and I am unable to get past just a handful of state refreshes before it errors out.

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 15
Comments: 24 (6 by maintainers)

Commits related to this issue

Switch to retryable http client Addresses #546 — committed to tback/terraform-provider-digitalocean by tback 2 years ago
Add godo's rate limiter configuration & retryable http client #546 (#967) * add godo's rate limit configuration to provider * set default rate limit to 240 req/m * docs: add requests_per_second... — committed to digitalocean/terraform-provider-digitalocean by DanielHLelis a year ago

Most upvoted comments

We’ve just released version 2.28.0 of this provider. It adds experimental support for automatically retrying requests that fail with 429 or 500-level response codes. It can be enabled by setting the DIGITALOCEAN_HTTP_RETRY_MAX environment variable or the http_retry_max argument in the provider configuration.

Please let us know if you have any feedback on this functionality. We will be looking to enable it by default in a future release.

Additionally, it adds support for configuring client-side rate-limiting to enforce quality of service. It can be enabled by setting the DIGITALOCEAN_REQUESTS_PER_SECOND environment variable or the requests_per_second argument in the provider configuration.

Thanks to @DanielHLelis for working with us on this!

andrewsomething on Apr 21, 2023

Can you link the PR here too?

Here: #967

DanielHLelis on Mar 28, 2023

I’ve hit the same problems of rate limits (to the point at which I couldn’t even run a single plan anymore) and opened a PR with proposed changes that fixes/mitigate this issue. If anyone would like to take a look at the fork: DanielHLelis/terraform-provider-digitalocean-ratelimit. It basically limits the number of requests per second and/or uses the retryable HTTP client from Hashicorp to retry the request after an error (like the 422).

To try it you just need to install it and do the override as explained in the CONTRIBUTING.md.

Hoping to get it into mainstream soon.

DanielHLelis on Mar 28, 2023

I just installed the provider manually with a provider override and it worked for me as long as it took me to migrate away from DO.

tback on Feb 28, 2023

Based on the Docs

Requests through the API are rate limited per OAuth token. Current rate limits:

5,000 requests per hour
250 requests per minute (5% of the hourly total)

So one thing you can possibly do is reach out to support and see if they can increase the limit for requests per minute. The only other thing you might want to investigate is looking at breaking down the terraform code into a module per domain so that you won’t hit the limit. I realize this might not be possible or suitable for your use case by something to think about.

lfarnell on Jan 18, 2021