atlantis: Atlantis fails to complete plan for a large environment.

With the latest Atlantis container, we’ve run into an issue where our largest TF environment never completes its Atlantis plan. This environment has a couple hundred resources tracked in it. The logs indicate that it reaches the “terraform workspace show” section of the auto-plan, but the actual plan never completes:

2019/02/01 20:45:49+0000 [INFO] project_command_builder.go:156 team/eng-terraform#110: 2 projects are to be planned based on their when_modified config

2019/02/01 20:45:49+0000 [INFO] project_locker.go:74 team/eng-terraform#110: Acquired lock with id “team/eng-terraform/envs/prod-global/iam-permissions/default”

2019/02/01 20:45:49+0000 [DBUG] project_command_runner.go:131 team/eng-terraform#110: Acquired lock for project

2019/02/01 20:45:49+0000 [DBUG] working_dir.go:78 team/eng-terraform#110: Clone directory “/home/atlantis/.atlantis/repos/team/eng-terraform/110/default” already exists, checking if it’s at the right commit

2019/02/01 20:45:49+0000 [DBUG] working_dir.go:100 team/eng-terraform#110: Repo is at correct commit “<REDACTED>” so will not re-clone

2019/02/01 20:45:52+0000 [INFO] terraform_client.go:171 team/eng-terraform#110: Successfully ran “terraform init -input=false -no-color” in “/home/atlantis/.atlantis/repos/team/eng-terraform/110/default/envs/prod-global/iam-permissions”

2019/02/01 20:45:52+0000 [INFO] terraform_client.go:171 team/eng-terraform#110: Successfully ran “terraform workspace show” in “/home/atlantis/.atlantis/repos/team/eng-terraform/110/default/envs/prod-global/iam-permissions”

Beyond that we get no further logs entries for this environment.

If I manually delete the lock on the environment and comment “atlantis plan” again on the PR, I get this error:

the default workspace is currently locked by another command that is running for this pull request–wait until the previous command is complete and try again

Since there are no more locks available to delete from the UI, the only solution is to manually apply the Terraform changes outside of Atlantis and then merge the PR, despite the CI check failure.

We do not have this behavior in our other, smaller environments (approx a dozen resources) – just this very large one.

This does not occur in version 0.4.13 – the plan successfully completes for the same large environment.

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 17 (10 by maintainers)

Most upvoted comments

Okay so I’ve created a Docker image lkysow/atlantis-debug:v1 off of this branch: https://github.com/runatlantis/atlantis/tree/custom-debug-image.

I’d like you to

  • Run that image with --log-level=debug
  • If Atlantis hangs, go to the new /debug/pprof/ endpoint and click on “full goroutine stack dump” and copy the data.
  • Also collect all the log entries and send them to me. You can do this via our Slack channel for security.
  • Finally, can you set the environment variable ATLANTIS_OLD_EXEC=true and then restart Atlantis and re-run the plan. This will cause Atlantis to use the old method of execution. Let me know if that one works.

Thanks so much for helping me debug this.