atlantis: Nil pointer error when manually running "atlantis plan" on Gitlab after upgrading Atlantis to 0.23.2
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
- Please do not leave “+1” or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
- If you are interested in working on this issue or have submitted a pull request, please leave a comment.
Overview of the Issue
After upgrading Atlantis to version 0.23.2, I ran into an issue where Atlantis would error out with a “invalid memory address or nil pointer dereference” exception.
runtime error: invalid memory address or nil pointer dereference
runtime/panic.go:260 (0x44e05c)
runtime/signal_unix.go:837 (0x44e02c)
github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:207 (0xce2f66)
github.com/runatlantis/atlantis/server/events/vcs/proxy.go:76 (0xce8744)
github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:32 (0xce9015)
github.com/runatlantis/atlantis/server/events/plan_command_runner.go:174 (0xf12a37)
github.com/runatlantis/atlantis/server/events/plan_command_runner.go:290 (0xf140eb)
github.com/runatlantis/atlantis/server/events/command_runner.go:296 (0xef86c3)
runtime/asm_amd64.s:1598 (0x46b200)
The specific scenario is manually triggering a plan operation because auto plan did not run. Looking at the code, I think the issue was introduced in #2979. Specifically, it was introduced at the following line. https://github.com/runatlantis/atlantis/blob/739d6e435b2b3e6f76b4c49f0872d5d7856dba07/server/events/plan_command_runner.go#L174
When fetching the status of a Gitlab merge request, a nil pointer dereference exception is thrown if Atlantis has not added a commit status to the merge request’s commit. This behavior is described in more detail in #1852. If auto plan runs, then the commit status will exist and the manual plan operation succeeds. If a project has a repo-level configuration file, then auto plan will fail to run because of #3122. If I manually trigger an apply operation, then the commit status will appear even though the apply operation will fail. After that, I can manually trigger a plan operation. Another scenario where auto plan doesn’t run is when Atlantis is configured to ignore draft merge requests. Removing the draft status and manually triggering an auto plan will result in the nil pointer exception.
The combination of this issue and #3122 makes version 0.23 almost unusable on Gitlab. I’ve had to revert my deployment to version 0.22.3.
Reproduction Steps
The simplest way to reproduce the issue is to disable auto plan by configuring Atlantis to ignore draft merge requests.
- Deploy Atlantis with
allow-draft-prsset tofalse. - Create a new draft merge request in Gitlab.
- Remove the draft status.
- Comment
atlantis plan
Logs
Here are logs for both the creation of a merge request and the manual run of atlantis plan. The merge request contained one change to a .tf file in the root of the project.
| @timestamp | @message |
|---|---|
| 2023-03-07 18:00:14.371 | {“level”:“debug”,“ts”:“2023-03-07T18:00:14.371Z”,“caller”:“server/middleware.go:45”,“msg”:“POST /events – from 172.31.5.135:11816”,“json”:{}} |
| 2023-03-07 18:00:14.371 | {“level”:“debug”,“ts”:“2023-03-07T18:00:14.371Z”,“caller”:“events/events_controller.go:109”,“msg”:“handling GitLab post”,“json”:{}} |
| 2023-03-07 18:00:14.371 | {“level”:“debug”,“ts”:“2023-03-07T18:00:14.371Z”,“caller”:“events/events_controller.go:492”,“msg”:“request valid”,“json”:{}} |
| 2023-03-07 18:00:14.371 | {“level”:“debug”,“ts”:“2023-03-07T18:00:14.371Z”,“caller”:“events/events_controller.go:496”,“msg”:“handling as comment event”,“json”:{}} |
| 2023-03-07 18:00:14.371 | {“level”:“debug”,“ts”:“2023-03-07T18:00:14.371Z”,“caller”:“logging/simple_logger.go:157”,“msg”:“Ignoring non-command comment: "**Error: goroutine panic. This is a bug…"”,“json”:{}} |
| 2023-03-07 18:00:14.371 | {“level”:“debug”,“ts”:“2023-03-07T18:00:14.371Z”,“caller”:“server/middleware.go:72”,“msg”:“POST /events – respond HTTP 200”,“json”:{}} |
| 2023-03-07 18:00:13.983 | {“level”:“error”,“ts”:“2023-03-07T18:00:13.982Z”,“caller”:“events/command_runner.go:427”,“msg”:“PANIC: runtime error: invalid memory address or nil pointer dereference\nruntime/panic.go:260 (0x44e05c)\nruntime/signal_unix.go:837 (0x44e02c)\ngithub.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:207 (0xce2f66)\ngithub.com/runatlantis/atlantis/server/events/vcs/proxy.go:76 (0xce8744)\ngithub.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:32 (0xce9015)\ngithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:174 (0xf12a37)\ngithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:290 (0xf140eb)\ngithub.com/runatlantis/atlantis/server/events/command_runner.go:296 (0xef86c3)\nruntime/asm_amd64.s:1598 (0x46b200)\n”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”},“stacktrace”:“github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).logPanics\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:427\nruntime.gopanic\n\truntime/panic.go:890\nruntime.panicmem\n\truntime/panic.go:260\nruntime.sigpanic\n\truntime/signal_unix.go:837\ngithub.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable\n\tgithub.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:207\ngithub.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable\n\tgithub.com/runatlantis/atlantis/server/events/vcs/proxy.go:76\ngithub.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus\n\tgithub.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:32\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).run\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:174\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:290\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:296”} |
| 2023-03-07 18:00:13.576 | {“level”:“debug”,“ts”:“2023-03-07T18:00:13.576Z”,“caller”:“server/middleware.go:45”,“msg”:“POST /events – from 172.31.5.135:11816”,“json”:{}} |
| 2023-03-07 18:00:13.576 | {“level”:“debug”,“ts”:“2023-03-07T18:00:13.576Z”,“caller”:“events/events_controller.go:109”,“msg”:“handling GitLab post”,“json”:{}} |
| 2023-03-07 18:00:13.576 | {“level”:“debug”,“ts”:“2023-03-07T18:00:13.576Z”,“caller”:“events/events_controller.go:492”,“msg”:“request valid”,“json”:{}} |
| 2023-03-07 18:00:13.576 | {“level”:“debug”,“ts”:“2023-03-07T18:00:13.576Z”,“caller”:“events/events_controller.go:496”,“msg”:“handling as comment event”,“json”:{}} |
| 2023-03-07 18:00:13.576 | {“level”:“info”,“ts”:“2023-03-07T18:00:13.576Z”,“caller”:“events/events_controller.go:545”,“msg”:“parsed comment as command="plan" verbose=false dir="" workspace="" project="" flags=""”,“json”:{}} |
| 2023-03-07 18:00:13.576 | {“level”:“debug”,“ts”:“2023-03-07T18:00:13.576Z”,“caller”:“events/events_controller.go:577”,“msg”:“executing command”,“json”:{}} |
| 2023-03-07 18:00:13.576 | {“level”:“debug”,“ts”:“2023-03-07T18:00:13.576Z”,“caller”:“logging/simple_logger.go:157”,“msg”:“Processing…”,“json”:{}} |
| 2023-03-07 18:00:13.576 | {“level”:“debug”,“ts”:“2023-03-07T18:00:13.576Z”,“caller”:“server/middleware.go:72”,“msg”:“POST /events – respond HTTP 200”,“json”:{}} |
| 2023-03-07 17:58:41.730 | {“level”:“debug”,“ts”:“2023-03-07T17:58:41.730Z”,“caller”:“events/working_dir.go:256”,“msg”:"ran: git merge -q --no-ff -m atlantis-merge FETCH_HEAD. Output: ",“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}} |
| 2023-03-07 17:58:41.730 | {“level”:“info”,“ts”:“2023-03-07T17:58:41.730Z”,“caller”:“events/project_command_builder.go:351”,“msg”:“successfully parsed atlantis.yaml file”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}} |
| 2023-03-07 17:58:41.730 | {“level”:“debug”,“ts”:“2023-03-07T17:58:41.730Z”,“caller”:“events/project_finder.go:180”,“msg”:“checking if project at dir "." workspace "default" was modified”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}} |
| 2023-03-07 17:58:41.730 | {“level”:“info”,“ts”:“2023-03-07T17:58:41.730Z”,“caller”:“events/project_command_builder.go:359”,“msg”:“0 projects are to be planned based on their when_modified config”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}} |
| 2023-03-07 17:58:41.730 | {“level”:“info”,“ts”:“2023-03-07T17:58:41.730Z”,“caller”:“events/plan_command_runner.go:97”,“msg”:“determined there was no project to run plan in”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}} |
| 2023-03-07 17:58:41.723 | {“level”:“debug”,“ts”:“2023-03-07T17:58:41.723Z”,“caller”:“events/working_dir.go:256”,“msg”:“ran: git merge-base main FETCH_HEAD. Output: 3bace385c287f62f25034aaf3745c3ff8376de55”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}} |
| 2023-03-07 17:58:41.722 | {“level”:“debug”,“ts”:“2023-03-07T17:58:41.722Z”,“caller”:“events/working_dir.go:256”,“msg”:“ran: git fetch head +refs/heads/test_gitlab_runner_15_9_ecr_cache_images:. Output: From https://gitlab.example.com/devops/development-application-resources\n * branch test_gitlab_runner_15_9_ecr_cache_images -> FETCH_HEAD\n * [new branch] test_gitlab_runner_15_9_ecr_cache_images -> head/test_gitlab_runner_15_9_ecr_cache_images”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}} |
| 2023-03-07 17:58:41.410 | {“level”:“debug”,“ts”:“2023-03-07T17:58:41.410Z”,“caller”:“events/working_dir.go:256”,“msg”:"ran: git remote add head https://group_43_bot:<redacted>@gitlab.example.com/devops/development-application-resources.git. Output: ",“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}} |
| 2023-03-07 17:58:41.408 | {“level”:“debug”,“ts”:“2023-03-07T17:58:41.408Z”,“caller”:“events/working_dir.go:256”,“msg”:“ran: git clone --branch main --single-branch https://group_43_bot:<redacted>@gitlab.example.com/devops/development-application-resources.git /home/atlantis/.atlantis/repos/devops/development-application-resources/164/default. Output: Cloning into ‘/home/atlantis/.atlantis/repos/devops/development-application-resources/164/default’…”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}} |
| 2023-03-07 17:58:41.059 | {“level”:“debug”,“ts”:“2023-03-07T17:58:41.059Z”,“caller”:“events/project_command_builder.go:284”,“msg”:“0 files were modified in this pull request”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}} |
| 2023-03-07 17:58:41.059 | {“level”:“debug”,“ts”:“2023-03-07T17:58:41.059Z”,“caller”:“events/project_command_builder.go:326”,“msg”:“got workspace lock”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}} |
| 2023-03-07 17:58:41.059 | {“level”:“info”,“ts”:“2023-03-07T17:58:41.059Z”,“caller”:“events/working_dir.go:224”,“msg”:“creating dir "/home/atlantis/.atlantis/repos/devops/development-application-resources/164/default"”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}} |
| 2023-03-07 17:58:40.918 | {“level”:“debug”,“ts”:“2023-03-07T17:58:40.918Z”,“caller”:“server/middleware.go:45”,“msg”:“POST /events – from 172.31.5.73:2274”,“json”:{}} |
| 2023-03-07 17:58:40.918 | {“level”:“debug”,“ts”:“2023-03-07T17:58:40.918Z”,“caller”:“events/events_controller.go:109”,“msg”:“handling GitLab post”,“json”:{}} |
| 2023-03-07 17:58:40.918 | {“level”:“debug”,“ts”:“2023-03-07T17:58:40.918Z”,“caller”:“events/events_controller.go:492”,“msg”:“request valid”,“json”:{}} |
| 2023-03-07 17:58:40.918 | {“level”:“debug”,“ts”:“2023-03-07T17:58:40.918Z”,“caller”:“events/events_controller.go:499”,“msg”:“handling as pull request event”,“json”:{}} |
| 2023-03-07 17:58:40.918 | {“level”:“info”,“ts”:“2023-03-07T17:58:40.918Z”,“caller”:“events/events_controller.go:602”,“msg”:“identified event as type "opened"”,“json”:{}} |
| 2023-03-07 17:58:40.918 | {“level”:“debug”,“ts”:“2023-03-07T17:58:40.918Z”,“caller”:“logging/simple_logger.go:157”,“msg”:“Processing…”,“json”:{}} |
| 2023-03-07 17:58:40.918 | {“level”:“debug”,“ts”:“2023-03-07T17:58:40.918Z”,“caller”:“server/middleware.go:72”,“msg”:“POST /events – respond HTTP 200”,“json”:{}} |
| 2023-03-07 17:58:26.780 | {“level”:“debug”,“ts”:“2023-03-07T17:58:26.780Z”,“caller”:“server/middleware.go:45”,“msg”:“POST /events – from 172.31.5.73:30530”,“json”:{}} |
| 2023-03-07 17:58:26.780 | {“level”:“debug”,“ts”:“2023-03-07T17:58:26.780Z”,“caller”:“events/events_controller.go:109”,“msg”:“handling GitLab post”,“json”:{}} |
| 2023-03-07 17:58:26.780 | {“level”:“debug”,“ts”:“2023-03-07T17:58:26.780Z”,“caller”:“events/events_controller.go:492”,“msg”:“request valid”,“json”:{}} |
| 2023-03-07 17:58:26.780 | {“level”:“debug”,“ts”:“2023-03-07T17:58:26.780Z”,“caller”:“logging/simple_logger.go:157”,“msg”:“Ignoring unsupported event”,“json”:{}} |
| 2023-03-07 17:58:26.780 | {“level”:“debug”,“ts”:“2023-03-07T17:58:26.780Z”,“caller”:“server/middleware.go:72”,“msg”:“POST /events – respond HTTP 200”,“json”:{}} |
Environment details
Atlantis Version: 0.23.2 Gitlab Version: 15.9.2 and 15.6.8
Atlantis server-side config file:
---
allow-draft-prs: true
automerge: true
checkout-strategy: merge
gitlab-hostname: gitlab.example.com
log-level: debug
silence-allowlist-errors: true
silence-no-projects: true
silence-vcs-status-no-plans: true
write-git-creds: true
Repo atlantis.yaml file:
---
version: 3
projects:
- dir: .
workflow: "production"
terraform_version: "v1.3.9"
Server side config
---
repos:
- id: "/.*/"
allowed_overrides: [ workflow ]
allow_custom_workflows: false
apply_requirements: [ "approved", "mergeable", "undiverged" ]
delete_source_branch_on_merge: true
workflows:
default:
plan:
steps:
- run: echo "The default workflow is not supported. Please select from 'development', 'production', 'staging', or 'uat'."
- run: return 1
apply:
steps:
- run: return 1
development:
plan:
steps:
- env:
name: ENVIRONMENT_NAME
value: development
- run: "/usr/local/bin/initialize.sh"
- run: "/usr/local/bin/validate.sh"
- plan
apply:
steps:
- apply
production:
plan:
steps:
- env:
name: ENVIRONMENT_NAME
value: production
- run: "/usr/local/bin/initialize.sh"
- run: "/usr/local/bin/validate.sh"
- plan
apply:
steps:
- apply
staging:
plan:
steps:
- env:
name: ENVIRONMENT_NAME
value: staging
- run: "/usr/local/bin/initialize.sh"
- run: "/usr/local/bin/validate.sh"
- plan
apply:
steps:
- apply
uat:
plan:
steps:
- env:
name: ENVIRONMENT_NAME
value: uat
- run: "/usr/local/bin/initialize.sh"
- run: "/usr/local/bin/validate.sh"
- plan
apply:
steps:
- apply
Additional Context
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 11
- Comments: 20 (9 by maintainers)
You are not the first to create a support ticket with gitlab.
https://gitlab.com/gitlab-org/gitlab/-/issues/398224
I think we could bring back #1981 until Gitlab fixes this and add a comment that is a temporary fix.
@GenPage @nitrocode What do you guys think?
I finally had some time to revisit this issue. I thing I haven’t mentioned yet is that I added
autoplansettings to my repo-level Atlantis files. I set enabled totrueand set the file pattern to auto plan for all file changes.Auto plan still did not run on 0.22 or 0.23.3 with those settings. Today, I upgraded Atlantis to 0.23.4. Auto plan is now working correctly when the
autoplansettings are present in the repo-level configuration file. I assume that https://github.com/runatlantis/atlantis/pull/3272 and possibly https://github.com/runatlantis/atlantis/pull/3277 are the reason it is working now. There are still certain scenarios where the nil pointer error occurs. For example, if I remove or limit thewhen_modifiedpattern(s) and then create an MR with changes that don’t match the pattern, the commit status is not created. If I add anatlantis plancomment to the merge request, the nil pointer error is triggered. The error still occurs in the draft status change scenario but I expected that based on the discussion above.My Gitlab installation is on version 15.9.4 now for what it is worth. It’ll be upgraded to 15.10 in the very near future.
https://gitlab.com/gitlab-org/gitlab/-/issues/398224
Thats the support request
@nitrocode
Any chance you could clarify this? When a Draft MR is marked
readyI would expect an auto-plan to take place.