serverless: Cannot deploy/remove stack because S3 bucket is gone
I just ran into a situation where sls remove (1.17.0) failed to remove my service, because a CloudFormation resource was still updated by Amazon and the stack could not be removed. This totally makes sense, but my problem is that serverless/CloudFormation already deleted everything from S3 and now I’m stuck with a service that neither can be deployed nor removed because the S3 bucket does not exist anymore.
I’m fine if this is just based on my weird edge-case or on the custom CloudFormation resources and there is nothing you can do to prevent this. But a more specific error message than The specified bucket does not exist might be useful 😉
After deleting the Stack manually using the AWS Console I can deploy the service again. Maybe this is something that could be added doing with sls remove --force ?
Console Outputs
$ > $(npm bin)/sls deploy
…
Serverless: Checking Stack update progress...
Serverless: Stack update finished...
$ > $(npm bin)/sls remove
Serverless: Getting all objects in S3 bucket...
Serverless: Removing objects in S3 bucket...
Serverless: Removing Stack...
Serverless: Checking Stack removal progress...
Serverless: Deployment failed!
Serverless Error ---------------------------------------
An error occurred while provisioning your stack: CustomTableA - Attempt to change a resource which is still in use: Table: custom-table-A is in the process of being updated..
State where neither deploy nor remove work anymore
$ > $(npm bin)/sls remove
Serverless: Getting all objects in S3 bucket...
Serverless Error ---------------------------------------
The specified bucket does not exist
$ > $(npm bin)/sls deploy
Serverless: Packaging service...
Serverless Error ---------------------------------------
The specified bucket does not exist
Solution implementation proposal
(Added by maintainers)
Ensure that bucket does not exist error does not trigger exception. Instead removal process should continue with stack removal
It can be fixed here: https://github.com/serverless/serverless/blob/c864fbd4826de27d2796e394b0a100c8d3add33e/lib/plugins/aws/remove/lib/bucket.js#L52-L53
If either this.listObjects or this.deleteObjects fails with bucket does not exist error. ignore such error silently
It might be that it’s not sufficient for some cases. Ideally then we should force removal of a stack and eventually list all resources that could not have been removed
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 25
- Comments: 30 (6 by maintainers)
vaya puta mierda de sls y plugins
Here is a workaround that’s been working for me without needing --force:
sls deployto generate an error message like: Serverless Error --------------------------------------- The serverless deployment bucket “myapp-#########” does not exist. Create it manually if you want to reuse the CloudFormation stack “myapp”, or delete the stack if it is no longer required.sls remove(which should now complete successfully)Will be nice to have a proper fix soon… =)
IMO, it would chage the error message something like this:
The specified bucket has been deleted. Please remove your stack with sls remove --force.--forceoption has not been added yet, we should add this to sls core though.I had the same problem. Yesterday, I modified a Role through AWS Console and when I did
sls removesomething didn’t go according to the plan and I’ve got stuck with:The specified bucket does not exist. Neithersls removeorsls deploywere working. In the end I went back to AWS Console and in Cloud Formation I found an associated stack with status “DELETE_FAILED” - manually deleted it and now I can redeploy. Moreover, you can try to go to S3 and look forServerlessDeploymentBucketand delete that too.Any updates? This is a much-needed feature for a number of us, it sounds like.
I just ran into this as well. A few unmentioned tips to get things reset:
I skimmed some of the discussion on the on the
--forceflag. To make this slightly safer, maybe it could ask for user confirmation of each resource it is force-deleting as it goes? Having the user go directly to AWS is a leaky abstraction of Serverless magic.I’m just encountering this. How hard can it be to address this that it has been opened for 3 years?
+1 on this. I was hoping
sls remove+sls deploywould fix some weird issues I’ve been having, but now that I’veremoved I can’t redeploy since the bucket doesn’t exist. Is there a way around this without needing to create an entirely new instance? Recreate the bucket by hand?Hi there.
I’m having the same problem when trying
removeAny progress here? I ran into this issue and was really surprised to see that you can’t re-run the “remove” command once the system is in a broken state. This is a violation of idempodency and should definitely be fixed. A force flag is only one way around this issue – in my opinion if the S3 bucket doesn’t exist, the program should move on and attempt to tear down the other resources.
I have an app created by Serverless framework and after some manipulations I couldn’t delete it’s stack in CloudFormation which was in
UPDATE_COMPLETE_CLEANUP_IN_PROGRESSstate permanently.sls deployandsls removedidn’t work either.Here is my solution: I just deleted the
.serverlessfolder from my app and triedsls removeandsls deployagainHope this will help to somebody 🚀
TLDR: Manually delete the cloud formation stack in AWS console, and then redeploy.
Also getting stuck with the “The specified bucket does not exist” problem. Something somewhere obviously expects a specific bucket to exist, where is that, so I can either create the bucket or remove the reference please?
In my case it did not tell me the name of the bucket 😅
Thanks for opening @sbstjn 👍
Yes, that’s definitely an annoying issue (I too stumbled upon this one a couple of times).
Great proposal @horike37 👍
BTW the
--forceflag usage / stack removal enforcement is discussed in https://github.com/serverless/serverless/issues/3928.Thanks for all the comments. Note the
help wantedlabel. PR’s welcome, and implementation proposal is included in issue descriptionWhen I tried creating a bucket on AWS console, I get this error:
Ok solved using
SLS_DEBUG=*, in this way the name of the bucket appeared in the logsThanks for commenting @TimCoates 👍
The resource for the deployment bucket (where all the artifacts are stored) will be automatically created by Serverless and is called
ServerlessDeploymentBucket(if you’re not using your own deployment bucket). You can manipulate this resource and overwrite it with something custom if you want to.Serverless will remove all the objects automatically if you use the S3 deployment bucket managed by Serverless. However if you have a custom bucket you might need to remove the objects / the bucket manually in the AWS console.
Serverless won’t auto-remove / manage items in your custom buckets since this could introduce unintended behavior.