terraform-provider-auth0: Missing handling for HTTP 429 Too Many Requests Response

Checklist

I have looked into the README and have not found a suitable solution or answer.
I have looked into the documentation and have not found a suitable solution or answer.
I have searched the issues and have not found a suitable solution or answer.
I have searched the Auth0 Community forums and have not found a suitable solution or answer.
I agree to the terms within the Auth0 Code of Conduct.

Description

A change in 92a913858eefb1953e5d642310f9e16df71509be acknowledges that there are rate limits for Auth0’s APIs (there also are authentication rate limits). However, it is not really a proper fix, it just means that anyone using terraform may suffer from these failures instead of anyone trying to contribute to this project.

Expectation

When terraform-provider-auth0 makes a request, it should look at the rate limit response headers and store their values.
If a response indicates that it’s approaching a rate limit, it should slow down.
If a response is a 429, trigger a full backoff. Wait until an appropriate amount of time has elapsed and retry.
It should be possible to revert 92a913858eefb1953e5d642310f9e16df71509be and tests should pass, albeit more slowly.

Reproduction

Given a developer (or free) auth0 tenant
Set up a management api

generate terraform json which has the following keys:

  "auth0-yyy-client_id": {
  "auth0-monitoring-client_id": {
  "auth0-monitoring-client_secret": {
  "auth0-xxx_auth0_client_id": {
  "auth0-xxx_auth0_client_id": {
  "auth0-xxx_auth0_client_id": {
  "auth0-xxx_auth0_client_id": {
"auth0_resource_server": {
"auth0_user": {
"auth0_tenant": {
"auth0_client": {
"auth0_client_grant": {
"auth0": {

perform a terraform apply

Or somehow create a terraform plan that involves 6-12 tasks which should result in at least 6 requests in under 1 second which is apparently in excess of 2 requests per second with a burst of ??

To review, you can go to the auth0 /logs endpoint and search for client_id:... for the client id from 2. Then for each event, open the url in a new tab and copy out the url and timestamp.

I’ve provided a sample including a bit of data about the 429:

Timestamp	Event
2022-08-08T18:43:14.458Z
2022-08-08T18:43:58.628Z
2022-08-08T18:43:58.896Z
2022-08-08T18:43:58.659Z
2022-08-08T18:43:58.436Z
2022-08-08T18:43:58.598Z
2022-08-08T18:43:58.786Z
2022-08-08T18:43:58.886Z	api_limit - Global limit has been reached

Auth0 Terraform Provider version

v0.14.0_x4

Terraform version

0.12.26

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 3
Comments: 27 (10 by maintainers)

Most upvoted comments

I can confirm that we also recently updated to the 1.0.0-beta.1 release

luislew on Aug 11, 2023

@willvedd the limits were not just in the tenant but in our pipeline. They all starting showing up when we updated the terraform provider from 0.36 to the latest 1.0.0-beta.1 release. We never had an issue prior to updating.

yourinium on Aug 11, 2023

Hey folks 👋🏻 ,

We sincerely apologize for the delay in our response. We fully empathize with the frustration you must be experiencing due to the rate limit issue, especially when dealing with larger config updates.

As you correctly pointed out, increasing the number of retries will not provide a viable solution; it will only exacerbate the problem.

Regrettably, our current capacity limitations have prevented us from implementing a comprehensive fix at this moment. Nevertheless, we do have some workarounds that should provide immediate relief:

When executing terraform apply, we recommend using the -parallelism=1 option. By default, terraform apply runs with a parallelism setting of 10, which can trigger the rate limit much sooner. While this may extend the operation duration, it will prevent further failures. (example: terraform apply -parallelism=1)
If possible, consider breaking down your tf config into smaller segments for application.

Please rest assured that our team is fully committed to addressing this issue promptly. We are currently regrouping and will prioritize resolving this matter asap ⚡ .

Thank you for your patience and understanding 🙏🏻

Further updates will follow the next days.

sergiught on Aug 21, 2023

@sergiught no problem for us so far with this version. Thanks for the fix!

glehmann on Aug 29, 2023

is there a time frame for the resolution of this bug? We’ve just switched to auth0 and this bug is breaking quite a lot of our CD runs 😦

Hey. That’s also our case. We contacted the support and they said they are working on it. No ETA though.

doino-gretchenliev on Aug 21, 2023

is there a time frame for the resolution of this bug? We’ve just switched to auth0 and this bug is breaking quite a lot of our CD runs 😦

glehmann on Aug 21, 2023

Same for us!! Thank you for getting this out so quick!! @sergiught

yourinium on Aug 29, 2023

Awesome to hear @glehmann 🙌🏻 thanks for checking! I’ll proceed to close this issue down then.

sergiught on Aug 29, 2023

Hey folks 👋🏻

We just released https://registry.terraform.io/providers/auth0/auth0/1.0.0-beta.2 with a fix for the rate limit retry issues. Please give it a go and let us know if you encounter any other issues 🙏🏻 .

Appreciate everyones patiences with the resolution!

sergiught on Aug 25, 2023

Hey folks 👋🏻 ,

An update from our side, I have a fix for the rate limit issues within https://github.com/auth0/terraform-provider-auth0/pull/788, while the code change has been approved, I’m still in the process of running some more load tests against it to ensure it’s the right approach. As soon as we can confirm everything we’ll make sure to make it available for you folks in a v1.0.0-beta.2 version.

sergiught on Aug 25, 2023

Thank you for the update and the commitment to getting this fixed!

yourinium on Aug 21, 2023

@kshvesov I made a similar comment on the linked PR: https://github.com/auth0/terraform-provider-auth0/pull/779#issuecomment-1679655372

luislew on Aug 16, 2023

We have also been seeing 429s in Terraform Cloud for recent changes, e.g.:

╷
│ Error: 429 Too Many Requests: Global limit has been reached
│ 
│   with module.emails.auth0_email_template.enrollment_email,
│   on ../../modules/emails/main.tf line 60, in resource "auth0_email_template" "enrollment_email":
│   60: resource "auth0_email_template" "enrollment_email" {
│ 
╵
Operation failed: failed running terraform apply (exit 1)

and

╷
│ Error: 429 Too Many Requests: Global limit has been reached
│ 
│   with module.rbac.auth0_role_permissions.idt_ff_offline_reader,
│   on ../../modules/rbac/main.tf line 123, in resource "auth0_role_permissions" "idt_ff_offline_reader":
│  123: resource "auth0_role_permissions" "idt_ff_offline_reader" {
│ 
╵

in two separate runs today.

luislew on Aug 11, 2023

Tried to migrate from an earlier version of the terraform package and I am getting an insane amount of 429 errors…it’s super frustrating. No real way to get around it either.

yourinium on Aug 7, 2023

For transparency I’ve created an internal backlog ticket to track this work: DXCDT-200.

sergiught on Aug 10, 2022

Appreciate the feedback @jsoref! You’re right and indeed we could avoid those 429s completely by reading the X-Ratelimit-*, we’ll add this into our backlog to improve the developer experience, however considering functionality isn’t actually broken and 429s are handled right now by the go-auth0 SDK by letting them happen and then waiting for the reset, I’ll be closing this issue down.

As for #266 we could continue the conversation over there. Please check https://github.com/auth0/terraform-provider-auth0/issues/266#issuecomment-1210808567.

sergiught on Aug 10, 2022