up: Deployment Fails with "ResourceConflictException" in Lambda

Prerequisites

  • [ v] I am running the latest version. (up upgrade)
  • [ v] I searched to see if the issue already exists.
  • [ v] I inspected the verbose debug output with the -v, --verbose flag.
  • [ v] Are you an Up Pro subscriber?

Description

Please see: https://apex-dev.slack.com/archives/C65P0GAV8/p1631749067003000 … and: https://aws.amazon.com/blogs/compute/coming-soon-expansion-of-aws-lambda-states-to-all-functions/

I have this issue too, but it was first reported by Ben Nichols on the Slack #up channel.

Whether via CLI up staging and up production, or, via in my case Github Actions, you get an error like:

Error: deploying: <region>: updating function code: ResourceConflictException: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:<region>:<arn_id>:function:my_func

… and the deployment fails.

Steps to Reproduce

Make a visible change in one of your branches and do up staging or up production as appropriate, or git push to the branch and have your CI run it. Either way, you get an error like the above.

As Ben Nichols mentioned, you can add aws:states:opt-out as the lambda description, to bypass the problem, but it’s reportedly going to stop working as of 1st Oct 2021.

This feels like something other up users are suddenly going to experience, so it’s my hope that someone can figure out how to change the code to fix this problem urgently.

Slack

Join us on Slack https://chat.apex.sh/

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 3
  • Comments: 32 (8 by maintainers)

Most upvoted comments

Confirmed I get the latest version and it works on the site that was failing. Thanks!

Edit: I mean I got the latest version automatically when deploying via GH actions. Also, running up upgrade from my $HOME upgraded showing a progress bar, then gave a message “Updated 1.7.0 Pro to 1.7.1 Pro”.

Ok if you up upgrade you should get v1.7.1-pro now with 0b09440, and if you run with -v you should see a bunch of logs mentioning checking and waiting for the state to change, curious to know how long it’s actually stuck in a pending state if that is what’s going on

yikes so I guess you need to poll/wait before UpdateFunctionCode, UpdateFunctionConfiguration, and PublishVersion by the looks of it haha… good old AWS, making things slow and difficult. I’ll have to add some reasonable limit for now when it comes to the wait so it doesn’t hang forever, but ideally it’s configurable

we are also seeing this issue. setting aws:states:opt-out as the function description seems to have gotten us going again but its definitely a temporary fix that will break once AWS decides to force lifecycles on everyone

I’m not having any luck reproducing it actually, I’m still able to deploy my apps with 1.7.0-pro and I tried doing a few fresh application stacks as well. Are you guys seeing any particular pattern or is it across all of your apps?

I was about to make the same post.

I even tried a fresh app from the README, it deploys the first time but after that I can’t do any more deployments. I can’t figure out why it would suddenly stop working,

~/Workspace/my-app$ up

     build: 5 files, 6.8 MB (678ms)
     deploy: staging (version 1) (24.576s)
     stack: complete (20.248s)
     endpoint: https://bawza8mlwc.execute-api.eu-west-1.amazonaws.com/staging/
~/Workspace/my-app$ up deploy -v staging
     4ms     DEBU up version 1.7.0-pro (os: linux, arch: amd64)
     0s      DEBU inferred runtime type=node
   ⠋ 0s      DEBU 1 regions from config
     4.329s  DEBU 1 regions from config
     0s      DEBU event deploy map[commit: stage:staging]
     0s      DEBU event platform.build map[]
     0s      DEBU hook prebuild is not defined
     0s      DEBU event hook map[hook:[] name:build]
     1ms     DEBU hook "build" command ""
     0s      DEBU event hook.complete map[duration:1.521584ms hook:[] name:build]
     0s      DEBU injecting proxy
     237ms   DEBU loading env vars
     166ms   DEBU loaded env vars duration=237
     0s      DEBU open
     0s      DEBU filtered .git – 4096
     0s      DEBU add _proxy.js: size=3609 mode=-rwxr-xr-x
     0s      DEBU add app.js: size=100 mode=-rwxrwxr-x
     259ms   DEBU add main: size=13813177 mode=-rwxrwxr-x
     1ms     DEBU add up-env.json: size=2 mode=-rwxr-xr-x
     0s      DEBU add up.json: size=86 mode=-rwxr-xr-x
     0s      DEBU stats dirs_filtered=1 files_added=5 files_filtered=0 size_uncompressed=14 MB
     14ms    DEBU close
     0s      DEBU event platform.build.zip map[duration:677.298639ms files:5 size_compressed:6827405 size_uncompressed:13816974]
     5ms     DEBU removing proxy
     0s      DEBU hook postbuild is not defined
     0s      DEBU event platform.build.complete map[duration:683.539072ms]
     0s      DEBU hook predeploy is not defined
     0s      DEBU hook deploy is not defined
     4.528s  DEBU checking for role
     0s      DEBU found existing role
     337ms   DEBU updating role policy
     0s      DEBU set role to arn:aws:iam::***:role/my-app-function
     0s      DEBU event platform.deploy map[commit: region:eu-west-1 stage:staging]
     4.504s  DEBU fetching function config region=eu-west-1
     5.574s  DEBU ensuring s3 bucket exists name=up-***-eu-west-1
     6.05s   DEBU uploading function to bucket up-***-eu-west-1 key my-app/staging/1631861611-ndAEJ3t5oTlWTDEw.zip
     288ms   DEBU updating function
     319ms   DEBU updating function code
     0s      DEBU event platform.function.update map[commit: region:eu-west-1 stage:staging]
     0s      DEBU event platform.deploy.complete map[commit: duration:16.735614612s region:eu-west-1 stage:staging version:]
 DEBU event platform.deploy.complete map[commit: duration:16.735614612s region:eu-west-1 stage:staging version:]
   ⠦ 0s      DEBU event deploy.complete map[commit: duration:22.285139868s stage:staging]
     Error: deploying: eu-west-1: updating function code: ResourceConflictException: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:eu-west-1:***:function:my-app
{
  RespMetadata: {
    StatusCode: 409,
    RequestID: "cab54633-8476-459b-ba54-cad7362b8dd5"
  },
  Message_: "The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:eu-west-1:***:function:my-app",
  Type: "User"
}