atlantis: Error using shared Terraform providers with a large number of parallel project plans
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
- Please do not leave “+1” or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
- If you are interested in working on this issue or have submitted a pull request, please leave a comment.
Overview of the Issue
This is an edge case error.
When planning a large number of projects in parallel (in our case 108), the following error appears in several plan outputs:
Error: fork/exec .terraform/providers/registry.terraform.io/hashicorp/aws/3.75.1/linux_amd64/terraform-provider-aws_v3.75.1_x5: text file busy
Looking into it further, it seems that a single provider is shared across all plans running in parallel, linked in each directory. I believe (but don’t know) that this is because all of the plans are attempting to use the same AWS provider and it can possibly only be accessed by so many at once?
This is obviously sensible to avoid re-downloading the same provider n times per pull request.
This was only noticed once the lock formula was updated to include a hash of the path under workspace names to allow for parallel plans for different states using the same workspace name https://github.com/runatlantis/atlantis/pull/2180
The fix is to break the PR into a smaller number of projects per PR (worked with 90)
Reproduction Steps
- Have an Atlantis repo with a large number of projects
- Run a parallel plan for all projects at the same time
Additional Context
Occurred during the migration from ECS to EKS for Atlantis and all of the states needed updated
About this issue
- Original URL
- State: open
- Created 5 years ago
- Reactions: 116
- Comments: 23 (11 by maintainers)
A workaround for those who do not wish to use lock files is to set the
plugin_cache_may_break_dependency_lock_fileoption, for example by setting theTF_PLUGIN_CACHE_MAY_BREAK_DEPENDENCY_LOCK_FILE=trueenvironment variable for all of Atlantis, or as a workflow step likeThis allows
terraform initto re-use the provider from the cache despite the lack of a lock file, meaning it does not try to re-download and overwrite it.For me, this at least solved the case where the issue was
open: …: text file busy. I have not encountered thefork/exec: …: text file busyerror but I suspect it happens when the race goes the other way: oneinitis currently writing to the file, then another tries to run it. This may still happen with this setting, but it should only happen once per provider version. After it has been cached once, any number of runs can use it in parallel.Even when committing lock files this is still an issue for us when setting
parallel_plan: truein our atlantis.yaml files for repositories with a large number of configurations to plan. The workaround I’ve used is to modify your atlantis workflow with some custom commands to “trick” terraform into pre-populating the plugin cache directory.This pretty much breaks plugin caching as atlantis/terraform will download the plugins for every configuration/state that is being planned. For us it is a worthwhile trade off to deal with added bandwidth costs over constantly re-planning individual configurations and dealing with lost developer productivity that the re-planning causes.
Hello
The way to fix this is to commit the lock file in your source control tool if you can. Since you will probably build the
.terraform.lock.hclfrom a different machine and not from the one that runs Atlantis, make sure to populate the lock file with all the architectures you use in your infrastructure.For example, we do this for our one:
If is all good, on the first run you may see a log line like this:
And after that, all the following ones should look like this:
note the
Using previously-installedMore info about the lock file can be found here
This isn’t necessarily a downloading problem. It looks like a concurrent filesystem access problem. Not pulling from hashicorp’s registry may speed up the downloads but it does not stop N terraform procs from trying to read the same file from disk all at once. From reading terraform docs it is clearly stated that the plugin cache dir is not concurrency safe. That is the real root of the issue here. It isn’t an atlantis bug…however since atlantis can run many
initoperations simultaneously it is prone to triggering this behavior in terraform.We are using Atlantis
v0.23.2 (commit: 5774453) (build date: 2023-03-03T23:50:16Z)along with terraformv1.4.2and still see the issue.For us, it seems to happen as soon as two different PRs plan concurrently, no matter how big they are. Tomorrow we’ll try version v0.23.3 of Atlantis and let you know, even tho it seems that the release doesn’t tackle any of these issues.
We started to see this behavior a couple of weeks ago, it was all good before, so not sure what could be the trigger, to be honest.
Is there any way of forcing Atlantis to use different binaries for terraform and the providers and not cache anything? We have around 30/40 projects, and being able to plan one at a time is a very big problem
This worked for me. Thanks @matthiasr
What if you deploy your one registry internally and point TF to that URL? in that case the providers will be downloaded from there instead.
https://github.com/outsideris/citizen for example.
On Wed, Aug 30, 2023 at 2:02 PM Jason Reslock @.***> wrote:
Thanks! I’ll give that a try.