Hi all! 👋 Wanted to give an update with our current status. The Terraform AWS Provider, Terraform DevEx and Terraform Core teams are all engaged in working towards a solution. The main problem is peak memory usage and this peak occurs when Terraform makes calls to configured providers to load their resource/data source schemas. The Terraform protocol contains a single RPC which asks for all schemas regardless of the specific resources configured. This means that as the AWS provider grows, the memory requirements for using it also grow. Some specific resources such as QuickSight and WAFv2 have extremely large nested schemas which can have an outsized effect on memory as we have seen.

Changing Terraform to load only the schemas for configured resources is likely feasible, but it is a very large undertaking requiring changes to the protocol (and thus changes to Terraform, terraform-plugin-framework, terraform-plugin-sdk and the provider). This change should decrease the memory footprint significantly for most users, and memory requirements would vary based on the particular resource configured. At this point we are investigating the feasibility of this change, but can make no indication of timings.

Short term however, there are a number of smaller optimizations we can do which we hope will alleviate some of the pressure while we research more longer term improvements. Some of these modifications will ship in the next version of the AWS provider. We will also be more intentional about accepting PR submissions for resources/data sources with particularly massive schema at least until their memory requirements are more closely understood. In some cases we may need to leave these out until we can resolve the root issue. More updates to come when we have details we can share.

Hi @at-wat 👋, thank you for the legwork in profiling this issue! Just to let you know we are actively investigating this with high priority in order to find a resolution. Will update this thread once we have a way forward.

Updated the reproduction repo: https://github.com/at-wat/terraform-provider-aws-v4.67-memory-usage/

and confirmed that the memory usage is significantly reduced on terraform-provider 5.20.0 and terraform 1.6.0

Happy Friday everyone!

I do have an update for you. Terraform 1.6 (currently in alpha) will include new functionality that will allow a cached provider schema to be used which should significantly reduce memory consumption for configurations including multiple instances of the same provider. In the near future terraform-plugin-go / terraform-plugin-sdk and terraform-plugin-framework will all be updated to take advantage of this, and provider authors (including the AWS Provider maintainers) will need to update these dependencies for this to take effect. I can’t give timings for when these changes will all be available, but this should be a significant step forward for those experiencing memory issues with multiple AWS providers configured.

Additionally, we have introduced a regex cache to the provider (released in v5.14.0) which in testing seems to have a significant impact on memory consumption.

We are continuing to work with the Terraform Core and DevEx teams on further improvements, and will update this issue with new information as I have it.

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave “+1” or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

Thanks for the research @at-wat , we’ve had to double the size of our codebuild runners to avoid these kind of errors that are due to memory exhaustion: [“The plugin.(*GRPCProvider).ValidateDataResourceConfig request was cancelled.”]

Like @n00borama, we’ve had to double the size of our runners to avoid these errors which of course has a negative cost impact 🙁

Also tried to play around with parallelism by lowering it from the default of 10 to 4 but no joy.

Hi @breathingdust, hope all is well. Are there any updates on this you could share out of curiosity? Have there been any developments?

https://github.com/hashicorp/terraform-provider-aws/issues/31722#issuecomment-1573255055 New large object appeared in pprof graph of v5.1.0 is from internal/service/quicksight/schema.visualsSchema()

It may be related to the new quicksight resources:

• added in 4.67.0
  • https://github.com/hashicorp/terraform-provider-aws/issues/30453
  • https://github.com/hashicorp/terraform-provider-aws/issues/31310
  • https://github.com/hashicorp/terraform-provider-aws/issues/31309
• added in 5.1.0
  • https://github.com/hashicorp/terraform-provider-aws/issues/31542
  • https://github.com/hashicorp/terraform-provider-aws/issues/31448