atlantis: Atlantis fails to complete plan for a large environment.
With the latest Atlantis container, we’ve run into an issue where our largest TF environment never completes its Atlantis plan. This environment has a couple hundred resources tracked in it. The logs indicate that it reaches the “terraform workspace show” section of the auto-plan, but the actual plan never completes:
2019/02/01 20:45:49+0000 [INFO] project_command_builder.go:156 team/eng-terraform#110: 2 projects are to be planned based on their when_modified config
2019/02/01 20:45:49+0000 [INFO] project_locker.go:74 team/eng-terraform#110: Acquired lock with id “team/eng-terraform/envs/prod-global/iam-permissions/default”
2019/02/01 20:45:49+0000 [DBUG] project_command_runner.go:131 team/eng-terraform#110: Acquired lock for project
2019/02/01 20:45:49+0000 [DBUG] working_dir.go:78 team/eng-terraform#110: Clone directory “/home/atlantis/.atlantis/repos/team/eng-terraform/110/default” already exists, checking if it’s at the right commit
2019/02/01 20:45:49+0000 [DBUG] working_dir.go:100 team/eng-terraform#110: Repo is at correct commit “<REDACTED>” so will not re-clone
2019/02/01 20:45:52+0000 [INFO] terraform_client.go:171 team/eng-terraform#110: Successfully ran “terraform init -input=false -no-color” in “/home/atlantis/.atlantis/repos/team/eng-terraform/110/default/envs/prod-global/iam-permissions”
2019/02/01 20:45:52+0000 [INFO] terraform_client.go:171 team/eng-terraform#110: Successfully ran “terraform workspace show” in “/home/atlantis/.atlantis/repos/team/eng-terraform/110/default/envs/prod-global/iam-permissions”
Beyond that we get no further logs entries for this environment.
If I manually delete the lock on the environment and comment “atlantis plan” again on the PR, I get this error:
the default workspace is currently locked by another command that is running for this pull request–wait until the previous command is complete and try again
Since there are no more locks available to delete from the UI, the only solution is to manually apply the Terraform changes outside of Atlantis and then merge the PR, despite the CI check failure.
We do not have this behavior in our other, smaller environments (approx a dozen resources) – just this very large one.
This does not occur in version 0.4.13 – the plan successfully completes for the same large environment.
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 17 (10 by maintainers)
Okay so I’ve created a Docker image
lkysow/atlantis-debug:v1off of this branch: https://github.com/runatlantis/atlantis/tree/custom-debug-image.I’d like you to
--log-level=debug/debug/pprof/endpoint and click on “full goroutine stack dump” and copy the data.ATLANTIS_OLD_EXEC=trueand then restart Atlantis and re-run the plan. This will cause Atlantis to use the old method of execution. Let me know if that one works.Thanks so much for helping me debug this.