atlantis: Transient errors during parallel plan from `.terraform.lock`: `plugins are not consistent`

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
  • Please do not leave “+1” or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

I’m occasionally getting some transient errors when running atlantis plan; currently, I have ATLANTIS_PARALLEL_POOL_SIZE set to 3. It’s most typically on states that have a lot of providers, giving me two possible theories:

  • all the states planning at once are causing the Terraform registry to rate-limit us, but the client is giving a confusing error message
  • OR, it’s a race condition due to parallel plans possibly using the same cache (Using integrations/foo vN.N.N from the shared cache directory in the output; see link below for why this is not guaranteed to be safe)

I’m not able to reproduce it right at the moment, and don’t have the exact error handy, so I’ll update here next time the issue comes up

Reproduction Steps

atlantis plan

Note: this is not consistently reproducible

Logs

n/a

Environment details

  • Atlantis version: v0.19.7
  • Atlantis flags: see below

Atlantis server-side config file: All config is from env vars / flags (with some kube secret references / other irrelevant stuff omitted)

            - name: ATLANTIS_ATLANTIS_URL
              value: https://xxx/
            - name: ATLANTIS_DEFAULT_TF_VERSION
              value: "1.2.5"
            - name: ATLANTIS_HIDE_PREV_PLAN_COMMENTS
              value: "true"
            - name: ATLANTIS_PARALLEL_POOL_SIZE
              value: "3"
            - name: ATLANTIS_PORT
              value: "4141"
            - name: ATLANTIS_REPO_ALLOWLIST
              value: github.com/orgname/*
            - name: TF_CLI_ARGS_apply
              value: "-compact-warnings"
            - name: TF_CLI_ARGS_init
              value: "-compact-warnings"
            - name: TF_CLI_ARGS_plan
              value: "-compact-warnings"

Repo atlantis.yaml file:

---
version: 3
parallel_plan: true

projects:
# [...]

Any other information you can provide about the environment/deployment. —>

Additional Context

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 36
  • Comments: 26 (20 by maintainers)

Most upvoted comments

for those who are still having this issue. There is now a setting that disables the plugin cache (thanks to #3720 !), so option 4 in https://github.com/runatlantis/atlantis/issues/2412#issuecomment-1455082428 is not needed anymore (but thank you for the solution!)

https://www.runatlantis.io/docs/server-configuration.html#use-tf-plugin-cache

If it were a cache per state, you wouldn’t need the new flag - it just helps avoid redownloading when there’s no lockfile and the cache is already populated.

I would guess users that both don’t check in a lockfile and set TF_PLUGIN_CACHE_DIR would want to set that flag once upgrading to 1.4.x

I agree with you that it’s odd that this issue comes up at all, if Atlantis doesn’t already do something to encourage terraform to share a cache directory, and especially since I think the .terraform directory would also be new / unique per PR, per state.

Regarding terraform 1.4 and the new kocking behavior, it seems that hashicorp has added a new flag that needs to be set to retain 1.3.x and below behavior in 1.4.x+.

TF_PLUGIN_CACHE_MAY_BREAK_DEPENDENCY_LOCK_FILE=true

@leonardocaylent

  1. My workaround assumes you want a shared, persistent, concurrent-safe plugin cache. If you’re doing Option 4 above, you’re only caching providers locally (which may improve performance when replanning the same PR, but it’s not perfect). You don’t need my workaround if that’s what you’re doing since no two inits are going to run at the same time on the same project.

  2. My workaround doesn’t have many prerequisites: have the tools used installed, run Atlantis on a system and filesystem that supports OverlayFS and flock, and little else. EFS might work, I haven’t tested it. Our setup is Atlantis on ECS with a persistent EBS volume mounted in Atlantis’ working directory. That’s it.

  3. You don’t need any other environment variables.

I’m pretty sure this won’t be incorporated onto Terraform’s code. Terraform has a commitment to staying compatible with other systems that do not support OverlayFS and flock. There are a number of proposals ongoing for fixing the plugin cache’s concurrency safety, but discussion is slow for valid reasons. My workaround is supposed to be a temporary thing using Linux primitives readily available in most CI/CD environments (such as Atlantis).

Here’s how we made Terraform’s plugin cache concurrent-safe (for parallel planning) while this gets fixed upstream.

Hope it helps!

#82 - looks like Atlantis may set it

It seems that parallel planning/applying is enabled, each of their terraform inits impact each other.

Some options for contributors

  1. Stagger the parallel runs so they do not conflict with each other
    • seems easy to implement by adding an arbitrary wait
  2. separate the runner from the server and allow the runner its own isolating
    • arguably the correct way to resolve this
    • would have additional benefits
    • large change which would require a lot of work

Please let me know if i missed any options. As always, we welcome prs