amplify-cli: Resource is not in the state stackUpdateComplete

Before opening, please confirm:

I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
I have searched for duplicate or closed issues.
I have read the guide for submitting bug reports.
I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

v15.3.0

Amplify CLI Version

6.3.0

What operating system are you using?

Mac

Amplify Categories

Not applicable

Amplify Commands

push

Describe the bug

After doing an amplify pull of an existing project I did 6 months ago i’m having problems pushing a change to a function. pull the project I run:

resulting error : UPDATE_FAILED UpdateRolesWithIDPFunction AWS::Lambda::Function Tue Oct 12 2021 16:09:53 GMT+0100 (British Summer Time) Template error: IAM role amplify-podcastcontextapp-dev-151617-authRole-idp doesn’t exist

✖ An error occurred when pushing the resources to the cloud 🛑 Resource is not in the state stackUpdateComplete An error occurred during the push operation: Resource is not in the state stackUpdateComplete

Expected behavior

I’d expect the push to be sucessfull and return a response

Reproduction steps

amplify pull --appId xxxxxx --envName dev
Then make my change in the python file (add in a comment line so amplify picks up the change)
amplify status, result conforms my function has an update
amplify push

GraphQL schema(s)

# Put schemas below this line

Log output

UPDATE_IN_PROGRESS functionpostToODC          AWS::CloudFormation::Stack Tue Oct 12 2021 16:09:53 GMT+0100 (British Summer Time)                                                                                         
UPDATE_IN_PROGRESS functionrssFeedProducer    AWS::CloudFormation::Stack Tue Oct 12 2021 16:09:53 GMT+0100 (British Summer Time)                                                                                         
UPDATE_IN_PROGRESS functionspreakerAPI        AWS::CloudFormation::Stack Tue Oct 12 2021 16:09:53 GMT+0100 (British Summer Time)                                                                                         
UPDATE_IN_PROGRESS functionrssURLAdd          AWS::CloudFormation::Stack Tue Oct 12 2021 16:09:53 GMT+0100 (British Summer Time)                                                                                         
UPDATE_COMPLETE    functionpostToODC          AWS::CloudFormation::Stack Tue Oct 12 2021 16:09:53 GMT+0100 (British Summer Time)                                                                                         
UPDATE_FAILED      UpdateRolesWithIDPFunction AWS::Lambda::Function      Tue Oct 12 2021 16:09:53 GMT+0100 (British Summer Time) Template error: IAM role amplify-podcastcontextapp-dev-xxxxx-authRole-idp doesn't exist
UPDATE_COMPLETE    functionrssFeedProducer    AWS::CloudFormation::Stack Tue Oct 12 2021 16:09:54 GMT+0100 (British Summer Time)                                                                                         
UPDATE_COMPLETE    functionspreakerAPI        AWS::CloudFormation::Stack Tue Oct 12 2021 16:09:54 GMT+0100 (British Summer Time)                                                                                         
UPDATE_COMPLETE    functionrssURLAdd          AWS::CloudFormation::Stack Tue Oct 12 2021 16:09:54 GMT+0100 (British Summer Time)                                                                                         
⠴ Updating resources in the cloud. This may take a few minutes...

UPDATE_ROLLBACK_IN_PROGRESS amplify-podcastcontextapp-dev-xxxxxxx-functionuSlicerAPIAdrequests-1ITOROC82BEXN AWS::CloudFormation::Stack Tue Oct 12 2021 16:09:54 GMT+0100 (British Summer Time) Initiated by parent stack
⠴ Updating resources in the cloud. This may take a few minutes...

UPDATE_FAILED               functionuSlicerAPIAdrequests         AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:04 GMT+0100 (British Summer Time) Resource update cancelled                                                                               
UPDATE_ROLLBACK_IN_PROGRESS amplify-podcastcontextapp-dev-xxxxxx AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:06 GMT+0100 (British Summer Time) The following resource(s) failed to update: [UpdateRolesWithIDPFunction, functionuSlicerAPIAdrequests]. 
⠴ Updating resources in the cloud. This may take a few minutes...

UPDATE_IN_PROGRESS functionuSlicerAPIAdrequests AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_IN_PROGRESS queueRssDLQ                  AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_IN_PROGRESS authcognitoc36aa6a0          AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_IN_PROGRESS hostingS3AndCloudFront       AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_IN_PROGRESS storageepisodeDB             AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_IN_PROGRESS queueRssQueue                AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_IN_PROGRESS eventsPodcastEventbus        AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_IN_PROGRESS storagespreakerPublisherDB   AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_IN_PROGRESS storageoraclelogs            AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_IN_PROGRESS storagerssFeedDB             AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_IN_PROGRESS queueRssQueueDLQ             AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE    queueRssDLQ                  AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE    authcognitoc36aa6a0          AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE    storageepisodeDB             AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE    eventsPodcastEventbus        AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE    queueRssQueue                AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE    storagespreakerPublisherDB   AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE    storageoraclelogs            AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE    storagerssFeedDB             AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE    queueRssQueueDLQ             AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE    hostingS3AndCloudFront       AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:14 GMT+0100 (British Summer Time) 
UPDATE_IN_PROGRESS functionpostToODC            AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:15 GMT+0100 (British Summer Time) 
UPDATE_IN_PROGRESS functionrssURLAdd            AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:15 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE    UpdateRolesWithIDPFunction   AWS::Lambda::Function      Tue Oct 12 2021 16:10:15 GMT+0100 (British Summer Time) 
UPDATE_IN_PROGRESS functionrssFeedProducer      AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:15 GMT+0100 (British Summer Time) 
UPDATE_IN_PROGRESS functionspreakerAPI          AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:15 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE    functionrssURLAdd            AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:15 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE    functionspreakerAPI          AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:15 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE    functionrssFeedProducer      AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:15 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE    functionpostToODC            AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:15 GMT+0100 (British Summer Time) 
⠇ Updating resources in the cloud. This may take a few minutes...

UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS amplify-podcastcontextapp-dev-xxxxxxx-functionuSlicerAPIAdrequests-1xxxxxx AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:19 GMT+0100 (British Summer Time) 
⠦ Updating resources in the cloud. This may take a few minutes...

UPDATE_COMPLETE                              functionuSlicerAPIAdrequests         AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:24 GMT+0100 (British Summer Time) 
UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS amplify-podcastcontextapp-dev-xxxxxx AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:25 GMT+0100 (British Summer Time) 
⠧ Updating resources in the cloud. This may take a few minutes...

UPDATE_COMPLETE queueRssQueueDLQ           AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:27 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE eventsPodcastEventbus      AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:27 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE hostingS3AndCloudFront     AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:27 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE functionrssFeedProducer    AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:27 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE functionrssURLAdd          AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:27 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE functionpostToODC          AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:27 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE queueRssQueue              AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:27 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE queueRssDLQ                AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:27 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE storageoraclelogs          AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:27 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE authcognitoc36aa6a0        AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:27 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE functionspreakerAPI        AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:27 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE storageepisodeDB           AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:28 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE storagerssFeedDB           AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:28 GMT+0100 (British Summer Time) 
UPDATE_COMPLETE storagespreakerPublisherDB AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:28 GMT+0100 (British Summer Time) 
⠇ Updating resources in the cloud. This may take a few minutes...

UPDATE_COMPLETE          functionuSlicerAPIAdrequests         AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:37 GMT+0100 (British Summer Time) 
UPDATE_ROLLBACK_COMPLETE amplify-podcastcontextapp-dev-xxxxxx AWS::CloudFormation::Stack Tue Oct 12 2021 16:10:37 GMT+0100 (British Summer Time) 
⠏ Updating resources in the cloud. This may take a few minutes...

Following resources failed

Resource Name: amplify-podcastcontextapp-UpdateRolesWithIDPFuncti-xxxxxx (AWS::Lambda::Function)
Event Type: update
Reason: Template error: IAM role amplify-podcastcontextapp-dev-xxxxx-authRole-idp doesn't exist
URL: https://console.aws.amazon.com/cloudformation/home?region=eu-west-2xxxxxx/events


✖ An error occurred when pushing the resources to the cloud
🛑 
Resource is not in the state stackUpdateComplete
An error occurred during the push operation: Resource is not in the state stackUpdateComplete

Additional information

There are several issues related to this that have already been published. I’ve tried what I believe to be most of them and still not working. I’ve tried to pull the back, then make my change to the local lambda function, then push again. I’ve tried to remove the function then add back again. I’ve tired to do a force push after i’ve pulled the project aswell. I have DB’s so I don’t want to rebuild the whole stack again as its in production.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 15 (9 by maintainers)

Most upvoted comments

Hi all,

I stumbled upon this thread while struggling with a similar issue, although my lambdas are auth triggers (and I understand @chelto is dealing with a lambda that is not an auth trigger).

I had a PreTokenGeneration lambda that I was trying to update in an existing env. When I did an amplify push to push the updated lambda, I would see the auth trigger completely disappear from the console! And yet, when I added a brand new amplify env, the updated lambda and associated trigger were pushed perfectly. So something was behaving strangely when updating my lambda in an existing env.

What I observed in the CF Stack console was that after pushing the updated lambda to the existing env, my CustomAuthTriggerResource was being DELETED immediately after it was UPDATED.

Screen Shot 2021-10-21 at 12 05 33 PM2

I assume it’s trying to do some kind of clean up, but the result was that the amplify-generated authTrigger* lambda that updates the UserPool config was getting a “delete” request that wiped out the lambdaConfig (and hence the auth triggers).

Screen Shot 2021-10-21 at 12 21 50 PM

Here are the CloudWatch logs for that lambda, and you can seen the “Delete” is happening right after the “Update”:

Screen Shot 2021-10-21 at 12 24 16 PM

Finally, I found in auth-trigger-cloudformation-template.json there was a setting that seemed relevant: “UpdateReplacePolicy”. It was set to “Delete”:

Screen Shot 2021-10-21 at 12 28 12 PM

When I changed that single line to “Retain” everything started working again. I was worried that I might end up with some orphaned resources that would have otherwise been cleaned up, but after looking for a bit I couldn’t find any.

I realize this is a different issue than @chelto is experiencing, but I thought I’d describe my findings, just in case they are useful. The symptoms were similar in some ways, so the issue might be similar (i.e., related to “UpdateReplacePolicy” or something similar.)

Also, I wanted to ask @marcvberg and @edwardfoyle whether what I’m describing is a possible bug with the auth trigger lambdas. I looked through the amplify-cli code and aws-cdk code and saw that the “CustomAuthTriggerResource” was being created with the default RemovalPolicy (i.e., delete) but since the name of the resource is constant, it basically ends up trying to delete the resource right after the resource is updated.

https://github.com/aws-amplify/amplify-cli/blob/master/packages/amplify-category-auth/src/provider-utils/awscloudformation/utils/generate-auth-trigger-template.ts#L111

https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/core/lib/custom-resource.ts#L128

Am I going to cause more headaches for myself by manually changing the “UpdateReplacePolicy” in auth-trigger-cloudformation-template.json to “Retain”?

Thanks for any info. I hope some of this is helpful.

nickwilldoit on Oct 21, 2021