atlantis: Nil pointer error when manually running "atlantis plan" on Gitlab after upgrading Atlantis to 0.23.2

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
  • Please do not leave “+1” or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

After upgrading Atlantis to version 0.23.2, I ran into an issue where Atlantis would error out with a “invalid memory address or nil pointer dereference” exception.

runtime error: invalid memory address or nil pointer dereference
runtime/panic.go:260 (0x44e05c)
runtime/signal_unix.go:837 (0x44e02c)
github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:207 (0xce2f66)
github.com/runatlantis/atlantis/server/events/vcs/proxy.go:76 (0xce8744)
github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:32 (0xce9015)
github.com/runatlantis/atlantis/server/events/plan_command_runner.go:174 (0xf12a37)
github.com/runatlantis/atlantis/server/events/plan_command_runner.go:290 (0xf140eb)
github.com/runatlantis/atlantis/server/events/command_runner.go:296 (0xef86c3)
runtime/asm_amd64.s:1598 (0x46b200)

The specific scenario is manually triggering a plan operation because auto plan did not run. Looking at the code, I think the issue was introduced in #2979. Specifically, it was introduced at the following line. https://github.com/runatlantis/atlantis/blob/739d6e435b2b3e6f76b4c49f0872d5d7856dba07/server/events/plan_command_runner.go#L174

When fetching the status of a Gitlab merge request, a nil pointer dereference exception is thrown if Atlantis has not added a commit status to the merge request’s commit. This behavior is described in more detail in #1852. If auto plan runs, then the commit status will exist and the manual plan operation succeeds. If a project has a repo-level configuration file, then auto plan will fail to run because of #3122. If I manually trigger an apply operation, then the commit status will appear even though the apply operation will fail. After that, I can manually trigger a plan operation. Another scenario where auto plan doesn’t run is when Atlantis is configured to ignore draft merge requests. Removing the draft status and manually triggering an auto plan will result in the nil pointer exception.

The combination of this issue and #3122 makes version 0.23 almost unusable on Gitlab. I’ve had to revert my deployment to version 0.22.3.

Reproduction Steps

The simplest way to reproduce the issue is to disable auto plan by configuring Atlantis to ignore draft merge requests.

  1. Deploy Atlantis with allow-draft-prs set to false.
  2. Create a new draft merge request in Gitlab.
  3. Remove the draft status.
  4. Comment atlantis plan

Logs

Here are logs for both the creation of a merge request and the manual run of atlantis plan. The merge request contained one change to a .tf file in the root of the project.


@timestamp @message
2023-03-07 18:00:14.371 {“level”:“debug”,“ts”:“2023-03-07T18:00:14.371Z”,“caller”:“server/middleware.go:45”,“msg”:“POST /events – from 172.31.5.135:11816”,“json”:{}}
2023-03-07 18:00:14.371 {“level”:“debug”,“ts”:“2023-03-07T18:00:14.371Z”,“caller”:“events/events_controller.go:109”,“msg”:“handling GitLab post”,“json”:{}}
2023-03-07 18:00:14.371 {“level”:“debug”,“ts”:“2023-03-07T18:00:14.371Z”,“caller”:“events/events_controller.go:492”,“msg”:“request valid”,“json”:{}}
2023-03-07 18:00:14.371 {“level”:“debug”,“ts”:“2023-03-07T18:00:14.371Z”,“caller”:“events/events_controller.go:496”,“msg”:“handling as comment event”,“json”:{}}
2023-03-07 18:00:14.371 {“level”:“debug”,“ts”:“2023-03-07T18:00:14.371Z”,“caller”:“logging/simple_logger.go:157”,“msg”:“Ignoring non-command comment: "**Error: goroutine panic. This is a bug…"”,“json”:{}}
2023-03-07 18:00:14.371 {“level”:“debug”,“ts”:“2023-03-07T18:00:14.371Z”,“caller”:“server/middleware.go:72”,“msg”:“POST /events – respond HTTP 200”,“json”:{}}
2023-03-07 18:00:13.983 {“level”:“error”,“ts”:“2023-03-07T18:00:13.982Z”,“caller”:“events/command_runner.go:427”,“msg”:“PANIC: runtime error: invalid memory address or nil pointer dereference\nruntime/panic.go:260 (0x44e05c)\nruntime/signal_unix.go:837 (0x44e02c)\ngithub.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:207 (0xce2f66)\ngithub.com/runatlantis/atlantis/server/events/vcs/proxy.go:76 (0xce8744)\ngithub.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:32 (0xce9015)\ngithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:174 (0xf12a37)\ngithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:290 (0xf140eb)\ngithub.com/runatlantis/atlantis/server/events/command_runner.go:296 (0xef86c3)\nruntime/asm_amd64.s:1598 (0x46b200)\n”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”},“stacktrace”:“github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).logPanics\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:427\nruntime.gopanic\n\truntime/panic.go:890\nruntime.panicmem\n\truntime/panic.go:260\nruntime.sigpanic\n\truntime/signal_unix.go:837\ngithub.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable\n\tgithub.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:207\ngithub.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable\n\tgithub.com/runatlantis/atlantis/server/events/vcs/proxy.go:76\ngithub.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus\n\tgithub.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:32\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).run\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:174\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:290\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:296”}
2023-03-07 18:00:13.576 {“level”:“debug”,“ts”:“2023-03-07T18:00:13.576Z”,“caller”:“server/middleware.go:45”,“msg”:“POST /events – from 172.31.5.135:11816”,“json”:{}}
2023-03-07 18:00:13.576 {“level”:“debug”,“ts”:“2023-03-07T18:00:13.576Z”,“caller”:“events/events_controller.go:109”,“msg”:“handling GitLab post”,“json”:{}}
2023-03-07 18:00:13.576 {“level”:“debug”,“ts”:“2023-03-07T18:00:13.576Z”,“caller”:“events/events_controller.go:492”,“msg”:“request valid”,“json”:{}}
2023-03-07 18:00:13.576 {“level”:“debug”,“ts”:“2023-03-07T18:00:13.576Z”,“caller”:“events/events_controller.go:496”,“msg”:“handling as comment event”,“json”:{}}
2023-03-07 18:00:13.576 {“level”:“info”,“ts”:“2023-03-07T18:00:13.576Z”,“caller”:“events/events_controller.go:545”,“msg”:“parsed comment as command="plan" verbose=false dir="" workspace="" project="" flags=""”,“json”:{}}
2023-03-07 18:00:13.576 {“level”:“debug”,“ts”:“2023-03-07T18:00:13.576Z”,“caller”:“events/events_controller.go:577”,“msg”:“executing command”,“json”:{}}
2023-03-07 18:00:13.576 {“level”:“debug”,“ts”:“2023-03-07T18:00:13.576Z”,“caller”:“logging/simple_logger.go:157”,“msg”:“Processing…”,“json”:{}}
2023-03-07 18:00:13.576 {“level”:“debug”,“ts”:“2023-03-07T18:00:13.576Z”,“caller”:“server/middleware.go:72”,“msg”:“POST /events – respond HTTP 200”,“json”:{}}
2023-03-07 17:58:41.730 {“level”:“debug”,“ts”:“2023-03-07T17:58:41.730Z”,“caller”:“events/working_dir.go:256”,“msg”:"ran: git merge -q --no-ff -m atlantis-merge FETCH_HEAD. Output: ",“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}}
2023-03-07 17:58:41.730 {“level”:“info”,“ts”:“2023-03-07T17:58:41.730Z”,“caller”:“events/project_command_builder.go:351”,“msg”:“successfully parsed atlantis.yaml file”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}}
2023-03-07 17:58:41.730 {“level”:“debug”,“ts”:“2023-03-07T17:58:41.730Z”,“caller”:“events/project_finder.go:180”,“msg”:“checking if project at dir "." workspace "default" was modified”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}}
2023-03-07 17:58:41.730 {“level”:“info”,“ts”:“2023-03-07T17:58:41.730Z”,“caller”:“events/project_command_builder.go:359”,“msg”:“0 projects are to be planned based on their when_modified config”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}}
2023-03-07 17:58:41.730 {“level”:“info”,“ts”:“2023-03-07T17:58:41.730Z”,“caller”:“events/plan_command_runner.go:97”,“msg”:“determined there was no project to run plan in”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}}
2023-03-07 17:58:41.723 {“level”:“debug”,“ts”:“2023-03-07T17:58:41.723Z”,“caller”:“events/working_dir.go:256”,“msg”:“ran: git merge-base main FETCH_HEAD. Output: 3bace385c287f62f25034aaf3745c3ff8376de55”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}}
2023-03-07 17:58:41.722 {“level”:“debug”,“ts”:“2023-03-07T17:58:41.722Z”,“caller”:“events/working_dir.go:256”,“msg”:“ran: git fetch head +refs/heads/test_gitlab_runner_15_9_ecr_cache_images:. Output: From https://gitlab.example.com/devops/development-application-resources\n * branch test_gitlab_runner_15_9_ecr_cache_images -> FETCH_HEAD\n * [new branch] test_gitlab_runner_15_9_ecr_cache_images -> head/test_gitlab_runner_15_9_ecr_cache_images”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}}
2023-03-07 17:58:41.410 {“level”:“debug”,“ts”:“2023-03-07T17:58:41.410Z”,“caller”:“events/working_dir.go:256”,“msg”:"ran: git remote add head https://group_43_bot:<redacted>@gitlab.example.com/devops/development-application-resources.git. Output: ",“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}}
2023-03-07 17:58:41.408 {“level”:“debug”,“ts”:“2023-03-07T17:58:41.408Z”,“caller”:“events/working_dir.go:256”,“msg”:“ran: git clone --branch main --single-branch https://group_43_bot:<redacted>@gitlab.example.com/devops/development-application-resources.git /home/atlantis/.atlantis/repos/devops/development-application-resources/164/default. Output: Cloning into ‘/home/atlantis/.atlantis/repos/devops/development-application-resources/164/default’…”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}}
2023-03-07 17:58:41.059 {“level”:“debug”,“ts”:“2023-03-07T17:58:41.059Z”,“caller”:“events/project_command_builder.go:284”,“msg”:“0 files were modified in this pull request”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}}
2023-03-07 17:58:41.059 {“level”:“debug”,“ts”:“2023-03-07T17:58:41.059Z”,“caller”:“events/project_command_builder.go:326”,“msg”:“got workspace lock”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}}
2023-03-07 17:58:41.059 {“level”:“info”,“ts”:“2023-03-07T17:58:41.059Z”,“caller”:“events/working_dir.go:224”,“msg”:“creating dir "/home/atlantis/.atlantis/repos/devops/development-application-resources/164/default"”,“json”:{“repo”:“devops/development-application-resources”,“pull”:“164”}}
2023-03-07 17:58:40.918 {“level”:“debug”,“ts”:“2023-03-07T17:58:40.918Z”,“caller”:“server/middleware.go:45”,“msg”:“POST /events – from 172.31.5.73:2274”,“json”:{}}
2023-03-07 17:58:40.918 {“level”:“debug”,“ts”:“2023-03-07T17:58:40.918Z”,“caller”:“events/events_controller.go:109”,“msg”:“handling GitLab post”,“json”:{}}
2023-03-07 17:58:40.918 {“level”:“debug”,“ts”:“2023-03-07T17:58:40.918Z”,“caller”:“events/events_controller.go:492”,“msg”:“request valid”,“json”:{}}
2023-03-07 17:58:40.918 {“level”:“debug”,“ts”:“2023-03-07T17:58:40.918Z”,“caller”:“events/events_controller.go:499”,“msg”:“handling as pull request event”,“json”:{}}
2023-03-07 17:58:40.918 {“level”:“info”,“ts”:“2023-03-07T17:58:40.918Z”,“caller”:“events/events_controller.go:602”,“msg”:“identified event as type "opened"”,“json”:{}}
2023-03-07 17:58:40.918 {“level”:“debug”,“ts”:“2023-03-07T17:58:40.918Z”,“caller”:“logging/simple_logger.go:157”,“msg”:“Processing…”,“json”:{}}
2023-03-07 17:58:40.918 {“level”:“debug”,“ts”:“2023-03-07T17:58:40.918Z”,“caller”:“server/middleware.go:72”,“msg”:“POST /events – respond HTTP 200”,“json”:{}}
2023-03-07 17:58:26.780 {“level”:“debug”,“ts”:“2023-03-07T17:58:26.780Z”,“caller”:“server/middleware.go:45”,“msg”:“POST /events – from 172.31.5.73:30530”,“json”:{}}
2023-03-07 17:58:26.780 {“level”:“debug”,“ts”:“2023-03-07T17:58:26.780Z”,“caller”:“events/events_controller.go:109”,“msg”:“handling GitLab post”,“json”:{}}
2023-03-07 17:58:26.780 {“level”:“debug”,“ts”:“2023-03-07T17:58:26.780Z”,“caller”:“events/events_controller.go:492”,“msg”:“request valid”,“json”:{}}
2023-03-07 17:58:26.780 {“level”:“debug”,“ts”:“2023-03-07T17:58:26.780Z”,“caller”:“logging/simple_logger.go:157”,“msg”:“Ignoring unsupported event”,“json”:{}}
2023-03-07 17:58:26.780 {“level”:“debug”,“ts”:“2023-03-07T17:58:26.780Z”,“caller”:“server/middleware.go:72”,“msg”:“POST /events – respond HTTP 200”,“json”:{}}

Environment details

Atlantis Version: 0.23.2 Gitlab Version: 15.9.2 and 15.6.8

Atlantis server-side config file:

---

allow-draft-prs: true
automerge: true
checkout-strategy: merge

gitlab-hostname: gitlab.example.com

log-level: debug

silence-allowlist-errors: true
silence-no-projects: true
silence-vcs-status-no-plans: true

write-git-creds: true

Repo atlantis.yaml file:

---
version: 3
projects:
  - dir: .
    workflow: "production"
    terraform_version: "v1.3.9"

Server side config

---
repos:
  - id: "/.*/"
    allowed_overrides: [ workflow ]
    allow_custom_workflows: false
    apply_requirements: [ "approved", "mergeable", "undiverged" ]
    delete_source_branch_on_merge: true
workflows:
  default:
    plan:
      steps:
        - run: echo "The default workflow is not supported.  Please select from 'development', 'production', 'staging', or 'uat'."
        - run: return 1
    apply:
      steps:
        - run: return 1

  development:
    plan:
      steps:
        - env:
            name: ENVIRONMENT_NAME
            value: development
        - run: "/usr/local/bin/initialize.sh"
        - run: "/usr/local/bin/validate.sh"
        - plan
    apply:
      steps:
        - apply
  production:
    plan:
      steps:
        - env:
            name: ENVIRONMENT_NAME
            value: production
        - run: "/usr/local/bin/initialize.sh"
        - run: "/usr/local/bin/validate.sh"
        - plan
    apply:
      steps:
        - apply
  staging:
    plan:
      steps:
        - env:
            name: ENVIRONMENT_NAME
            value: staging
        - run: "/usr/local/bin/initialize.sh"
        - run: "/usr/local/bin/validate.sh"
        - plan
    apply:
      steps:
        - apply
  uat:
    plan:
      steps:
        - env:
            name: ENVIRONMENT_NAME
            value: uat
        - run: "/usr/local/bin/initialize.sh"
        - run: "/usr/local/bin/validate.sh"
        - plan
    apply:
      steps:
        - apply

Additional Context

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 11
  • Comments: 20 (9 by maintainers)

Most upvoted comments

You are not the first to create a support ticket with gitlab.

https://gitlab.com/gitlab-org/gitlab/-/issues/398224

I think we could bring back #1981 until Gitlab fixes this and add a comment that is a temporary fix.

@GenPage @nitrocode What do you guys think?

I finally had some time to revisit this issue. I thing I haven’t mentioned yet is that I added autoplan settings to my repo-level Atlantis files. I set enabled to true and set the file pattern to auto plan for all file changes.

---
version: 3
projects:
  - dir: .
    workflow: "production"
    terraform_version: "v1.3.9"
    autoplan:
      when_modified: ["**/*"]
      enabled: true

Auto plan still did not run on 0.22 or 0.23.3 with those settings. Today, I upgraded Atlantis to 0.23.4. Auto plan is now working correctly when the autoplan settings are present in the repo-level configuration file. I assume that https://github.com/runatlantis/atlantis/pull/3272 and possibly https://github.com/runatlantis/atlantis/pull/3277 are the reason it is working now. There are still certain scenarios where the nil pointer error occurs. For example, if I remove or limit the when_modified pattern(s) and then create an MR with changes that don’t match the pattern, the commit status is not created. If I add an atlantis plan comment to the merge request, the nil pointer error is triggered. The error still occurs in the draft status change scenario but I expected that based on the discussion above.

My Gitlab installation is on version 15.9.4 now for what it is worth. It’ll be upgraded to 15.10 in the very near future.

I should clarify that I don’t know the expected behavior of auto plan when the draft status is removed from a merge request

@nitrocode

Any chance you could clarify this? When a Draft MR is marked ready I would expect an auto-plan to take place.