serverless: Cannot deploy/remove stack because S3 bucket is gone

I just ran into a situation where sls remove (1.17.0) failed to remove my service, because a CloudFormation resource was still updated by Amazon and the stack could not be removed. This totally makes sense, but my problem is that serverless/CloudFormation already deleted everything from S3 and now I’m stuck with a service that neither can be deployed nor removed because the S3 bucket does not exist anymore.

I’m fine if this is just based on my weird edge-case or on the custom CloudFormation resources and there is nothing you can do to prevent this. But a more specific error message than The specified bucket does not exist might be useful 😉

After deleting the Stack manually using the AWS Console I can deploy the service again. Maybe this is something that could be added doing with sls remove --force ?

Console Outputs

$ > $(npm bin)/sls deploy
…
Serverless: Checking Stack update progress...
Serverless: Stack update finished...
$ > $(npm bin)/sls remove
Serverless: Getting all objects in S3 bucket...
Serverless: Removing objects in S3 bucket...
Serverless: Removing Stack...
Serverless: Checking Stack removal progress...
Serverless: Deployment failed!
 
  Serverless Error ---------------------------------------
 
  An error occurred while provisioning your stack: CustomTableA - Attempt to change a resource which is still in use: Table: custom-table-A is in the process of being updated..

State where neither deploy nor remove work anymore

$ > $(npm bin)/sls remove
Serverless: Getting all objects in S3 bucket...
 
  Serverless Error ---------------------------------------
 
  The specified bucket does not exist
$ > $(npm bin)/sls deploy
Serverless: Packaging service...

  Serverless Error ---------------------------------------
 
  The specified bucket does not exist

Solution implementation proposal

(Added by maintainers)

Ensure that bucket does not exist error does not trigger exception. Instead removal process should continue with stack removal

It can be fixed here: https://github.com/serverless/serverless/blob/c864fbd4826de27d2796e394b0a100c8d3add33e/lib/plugins/aws/remove/lib/bucket.js#L52-L53 If either this.listObjects or this.deleteObjects fails with bucket does not exist error. ignore such error silently

It might be that it’s not sufficient for some cases. Ideally then we should force removal of a stack and eventually list all resources that could not have been removed

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 25
  • Comments: 30 (6 by maintainers)

Most upvoted comments

vaya puta mierda de sls y plugins

Here is a workaround that’s been working for me without needing --force:

  1. Run sls deploy to generate an error message like: Serverless Error --------------------------------------- The serverless deployment bucket “myapp-#########” does not exist. Create it manually if you want to reuse the CloudFormation stack “myapp”, or delete the stack if it is no longer required.
  2. Manually create an S3 bucket called “myapp-#########” - all settings default are fine.
  3. Run sls remove (which should now complete successfully)

Will be nice to have a proper fix soon… =)

IMO, it would chage the error message something like this: The specified bucket has been deleted. Please remove your stack with sls remove --force. --force option has not been added yet, we should add this to sls core though.

I had the same problem. Yesterday, I modified a Role through AWS Console and when I did sls remove something didn’t go according to the plan and I’ve got stuck with: The specified bucket does not exist. Neither sls remove or sls deploy were working. In the end I went back to AWS Console and in Cloud Formation I found an associated stack with status “DELETE_FAILED” - manually deleted it and now I can redeploy. Moreover, you can try to go to S3 and look for ServerlessDeploymentBucket and delete that too.

Any updates? This is a much-needed feature for a number of us, it sounds like.

I just ran into this as well. A few unmentioned tips to get things reset:

  • The AWS Console CloudFormation “delete” action may refuse to delete your stack the first time. If this happens you (of course) just try to delete it again. The second time it gives you an option to skip the offending resource. In my case, it was an IAM role. I proceeded with the delete, then deleted the role.
  • You may have to delete the CloudWatch log group, too. Fortunately the Serverless CLI outputs this group which can be searched for in the CloudWatch UI.

I skimmed some of the discussion on the on the --force flag. To make this slightly safer, maybe it could ask for user confirmation of each resource it is force-deleting as it goes? Having the user go directly to AWS is a leaky abstraction of Serverless magic.

I’m just encountering this. How hard can it be to address this that it has been opened for 3 years?

+1 on this. I was hoping sls remove + sls deploy would fix some weird issues I’ve been having, but now that I’ve removed I can’t redeploy since the bucket doesn’t exist. Is there a way around this without needing to create an entirely new instance? Recreate the bucket by hand?

Hi there.

I’m having the same problem when trying remove

Serverless: Getting all objects in S3 bucket...
 
  Serverless Error ---------------------------------------
 
  The specified bucket does not exist
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information ---------------------------
     OS:                     darwin
     Node Version:           8.10.0
     Serverless Version:     1.40.0

Any progress here? I ran into this issue and was really surprised to see that you can’t re-run the “remove” command once the system is in a broken state. This is a violation of idempodency and should definitely be fixed. A force flag is only one way around this issue – in my opinion if the S3 bucket doesn’t exist, the program should move on and attempt to tear down the other resources.

I have an app created by Serverless framework and after some manipulations I couldn’t delete it’s stack in CloudFormation which was in UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state permanently. sls deploy and sls remove didn’t work either.

Here is my solution: I just deleted the .serverless folder from my app and tried sls remove and sls deploy again

Hope this will help to somebody 🚀

TLDR: Manually delete the cloud formation stack in AWS console, and then redeploy.

Also getting stuck with the “The specified bucket does not exist” problem. Something somewhere obviously expects a specific bucket to exist, where is that, so I can either create the bucket or remove the reference please?

Here is a workaround that’s been working for me without needing --force:

  1. Run sls deploy to generate an error message like: Serverless Error --------------------------------------- The serverless deployment bucket “myapp-#########” does not exist. Create it manually if you want to reuse the CloudFormation stack “myapp”, or delete the stack if it is no longer required.
  2. Manually create an S3 bucket called “myapp-#########” - all settings default are fine.
  3. Run sls remove (which should now complete successfully)

Will be nice to have a proper fix soon… =)

In my case it did not tell me the name of the bucket 😅


Serverless Error ---------------------------------------
 
  The specified bucket does not exist
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           10.19.0
     Serverless Version:     1.32.0

Thanks for opening @sbstjn 👍

Yes, that’s definitely an annoying issue (I too stumbled upon this one a couple of times).

Great proposal @horike37 👍

BTW the --force flag usage / stack removal enforcement is discussed in https://github.com/serverless/serverless/issues/3928.

Thanks for all the comments. Note the help wanted label. PR’s welcome, and implementation proposal is included in issue description

Here is a workaround that’s been working for me without needing --force:

  1. Run sls deploy to generate an error message like: Serverless Error --------------------------------------- The serverless deployment bucket “myapp-#########” does not exist. Create it manually if you want to reuse the CloudFormation stack “myapp”, or delete the stack if it is no longer required.
  2. Manually create an S3 bucket called “myapp-#########” - all settings default are fine.
  3. Run sls remove (which should now complete successfully)

Will be nice to have a proper fix soon… =)

When I tried creating a bucket on AWS console, I get this error:

An unexpected error occurred.
A conflicting conditional operation is currently in progress against this resource. Please try again.

Ok solved using SLS_DEBUG=*, in this way the name of the bucket appeared in the logs

Thanks for commenting @TimCoates 👍

The resource for the deployment bucket (where all the artifacts are stored) will be automatically created by Serverless and is called ServerlessDeploymentBucket (if you’re not using your own deployment bucket). You can manipulate this resource and overwrite it with something custom if you want to.

Serverless will remove all the objects automatically if you use the S3 deployment bucket managed by Serverless. However if you have a custom bucket you might need to remove the objects / the bucket manually in the AWS console.

Serverless won’t auto-remove / manage items in your custom buckets since this could introduce unintended behavior.