atlantis: Apply fails on Gitlab 16.4.0 with "Pull request must be mergeable before running apply."

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
  • Please do not leave “+1” or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

Running atlantis apply on an approved Gitlab PR fails intermittently with Apply Failed: Pull request must be mergeable before running apply..

Reproduction Steps

  1. Run atlantis apply on approved PR
  2. Atlantis comments with error: **Apply Failed**: Pull request must be mergeable before running apply.

Logs

Logs
August 23, 2023 at 17:03 (UTC+1:00)	{"level":"error","ts":"2023-08-23T16:03:28.369Z","caller":"events/instrumented_project_command_runner.go:84","msg":"Failure running apply operation: Pull request must be mergeable before running apply.","json":{"repo":"ume-platform-engineering/tf-env-data-engineering","pull":"189"},"stacktrace":"github.com/runatlantis/atlantis/server/events.RunAndEmitStats\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:84\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).Apply\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:46\ngithub.com/runatlantis/atlantis/server/events.runProjectCmds\n\tgithub.com/runatlantis/atlantis/server/events/project_command_pool_executor.go:48\ngithub.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:166\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:298"}
August 23, 2023 at 17:03 (UTC+1:00)	{"level":"info","ts":"2023-08-23T16:03:29.799Z","caller":"events/automerger.go:20","msg":"not automerging because project at dir \"XXX\", workspace \"default\" has status \"apply_errored\"","json":{"repo":"XXX","pull":"189"}}
August 23, 2023 at 17:03 (UTC+1:00)	{"level":"error","ts":"2023-08-23T16:03:32.912Z","caller":"logging/simple_logger.go:163","msg":"invalid key: e4d66b69-e7c7-495a-87fd-20f677f74b13","json":{},"stacktrace":"github.com/runatlantis/atlantis/server/logging.(*StructuredLogger).Log\n\tgithub.com/runatlantis/atlantis/server/logging/simple_logger.go:163\ngithub.com/runatlantis/atlantis/server/controllers.(*JobsController).respond\n\tgithub.com/runatlantis/atlantis/server/controllers/jobs_controller.go:92\ngithub.com/runatlantis/atlantis/server/controllers.(*JobsController).getProjectJobsWS\n\tgithub.com/runatlantis/atlantis/server/controllers/jobs_controller.go:70\ngithub.com/runatlantis/atlantis/server/controllers.(*JobsController).GetProjectJobsWS\n\tgithub.com/runatlantis/atlantis/server/controllers/jobs_controller.go:83\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2122\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.com/gorilla/mux@v1.8.0/mux.go:210\ngithub.com/urfave/negroni/v3.Wrap.func1\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:59\ngithub.com/urfave/negroni/v3.HandlerFunc.ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:33\ngithub.com/urfave/negroni/v3.middleware.ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngithub.com/runatlantis/atlantis/server.(*RequestLogger).ServeHTTP\n\tgithub.com/runatlantis/atlantis/server/middleware.go:70\ngithub.com/urfave/negroni/v3.middleware.ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngithub.com/urfave/negroni/v3.(*Recovery).ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/recovery.go:210\ngithub.com/urfave/negroni/v3.middleware.ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngithub.com/urfave/negroni/v3.(*Negroni).ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:111\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2936\nnet/http.(*conn).serve\n\tnet/http/server.go:1995"}

Environment details

  • Atlantis version: v0.23.2 & v0.23.4
  • Deployment method: ecs
  • If not running the latest Atlantis version have you tried to reproduce this issue on the latest version: no
  • Gitlab 16.4.0

Atlantis server-side config file:

{
    "taskDefinitionArn": "arn:aws:ecs:XXX:XXX",
    "containerDefinitions": [
        {
            "name": "atlantis",
            "image": "XXX/atlantis:v0.24.3-0bcdeaa1",
            "cpu": 2048,
            "memory": 4096,
            "memoryReservation": 128,
            "portMappings": [
                {
                    "containerPort": 4141,
                    "hostPort": 4141,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "environment": [
                {
                    "name": "ATLANTIS_ALLOW_REPO_CONFIG",
                    "value": "false"
                },
                {
                    "name": "ATLANTIS_HIDE_PREV_PLAN_COMMENTS",
                    "value": "false"
                },
                {
                    "name": "ATLANTIS_WRITE_GIT_CREDS",
                    "value": "true"
                },
                {
                    "name": "ATLANTIS_SILENCE_NO_PROJECTS",
                    "value": "false"
                },
                {
                    "name": "ATLANTIS_GITLAB_USER",
                    "value": "XXX"
                },
                {
                    "name": "ATLANTIS_LOG_LEVEL",
                    "value": "debug"
                },
                {
                    "name": "ATLANTIS_AUTOMERGE",
                    "value": "true"
                },
                {
                    "name": "ATLANTIS_BITBUCKET_USER",
                    "value": ""
                },
                {
                    "name": "ATLANTIS_REPO_CONFIG_JSON",
                    "value": "{\"repos\":[{\"allow_custom_workflows\":true,\"allowed_overrides\":[\"workflow\"],\"apply_requirements\":[\"undiverged\",\"mergeable\",\"approved\"],\"id\":\"/.*/\",\"repo_config_file\":\"atlantis.yaml\"}],\"workflows\":{\"default\":{\"apply\":{\"steps\":[\"apply\",{\"run\":\"[ ! -z \\\"$PROJECT_NAME\\\" ] \\u0026\\u0026 export TAG=$PROJECT_NAME || export TAG=last-applied \\u0026\\u0026 git config --global user.name XXXn \\u0026\\u0026 git config --global user.email XXX \\u0026\\u0026 git fetch --tags -f \\u0026\\u0026 git fetch --all --tags \\u0026\\u0026 (git tag --delete $TAG || true) \\u0026\\u0026 git tag -a $TAG -m \\\"Tagged automatically by atlantis\\\" \\u0026\\u0026 (git push origin --delete $TAG || true) \\u0026\\u0026 git push origin $TAG\"}]},\"plan\":{\"steps\":[\"init\",\"plan\"]}}}}"
                },
                {
                    "name": "ATLANTIS_PARALLEL_POOL_SIZE",
                    "value": "50"
                },
                {
                    "name": "ATLANTIS_REPO_ALLOWLIST",
                    "value": "XXX"
                },
                {
                    "name": "ATLANTIS_GITLAB_HOSTNAME",
                    "value": "gitlab.com"
                },
                {
                    "name": "ATLANTIS_DEFAULT_TF_VERSION",
                    "value": "v0.13.4"
                },
                {
                    "name": "ATLANTIS_GH_APP_ID",
                    "value": ""
                },
                {
                    "name": "ATLANTIS_BITBUCKET_BASE_URL",
                    "value": ""
                },
                {
                    "name": "ATLANTIS_PORT",
                    "value": "4141"
                },
                {
                    "name": "ATLANTIS_GH_USER",
                    "value": ""
                },
                {
                    "name": "ATLANTIS_ATLANTIS_URL",
                    "value": "https://XXX"
                }
            ],
            "mountPoints": [],
            "volumesFrom": [],
            "secrets": [
XXX
]

Repo atlantis.yaml file:

version: 3
projects:

  - name: platform
    dir: XXX/platform
    autoplan:
      when_modified: [ "*.tf" ]
      enabled: true
...

Additional Context

Issue started happening on Atlantis v0.23.2. Still happening after upgrading to v0.23.4.

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 13
  • Comments: 25 (14 by maintainers)

Most upvoted comments

I work with @syst0m and this is still an issue for us. I have been doing some debugging when I have ran in to this.

I noticed that when this occurs, if you look at the Pipelines tab of the MR it shows that the status of the atlantis plan is still in a Running state although this is clearly not the case the plan has been posted back to the MR as a comment and the main MR page shows the status of the same atlantis plan as Success.

I have an example of this below, and the steps leading up to this was just our standard workflow with no complications:

  • Create feature branch
  • Push commit
  • Raise MR
  • Atlantis plan is executed automatically and the comment/status all looks correct on MR summary page
  • MR is approved
  • atlantis apply comment
  • Apply Failed: Pull request must be mergeable before running apply. error message posted back to MR

MR Summary: atlantis-mr-summary

MR Pipeline tab: atlantis-mr-pipeline

In the above case running subsequent atlantis plan / atlantis apply commands on the MR does not resolve the issue (people are generally working around this by pushing new commits).

With this in mind I was exploring the gitlab commit status API and have written a script to manually update the commit status of the “stuck” atlantis plan job to “success” and this allows me to fix a stuck MR to work around the issue, so it feels like the issue is caused by atlantis not be posting the commit status back correctly, either the wrong request is being sent or the request is failing for some reason (e.g network / gitlab transient issues / rate limiting etc).

I thought I might be able to do the reverse this workaround of this to reliably re-produce the problem (raise an MR, then manually set the atlantis plan status to “Running”) but the behaviour wasn’t identical - it still blocked the MR with the Pull request must be mergeable before running apply. message but unlike the above example we could work around the issue by issuing atlantis plan / atlantis apply commands so I feel like I am missing something in terms of where in gitlab that atlantis pushes the statuses to.

ohhhh Gitlab…is still better than Bitbucket

Let’s see what they say in your ticket, if you can link it here @saraangelmurphy that will be useful.

Given the critical vulnerability in previous versions, I hope that you guys prioritize the upgrade

https://nvd.nist.gov/vuln/detail/CVE-2023-5009