aws-codedeploy-agent: CodeDeploy fails at "DownloadBundle" step

After the recent AWS CodeDeploy Agent update (version 1.1.1), all of my CodeDeploy deployments have failed at the “DownloadBundle” step.

In the console, this shows an error similar to:

Destination '/opt/codedeploy-agent/deployment-root/abcde123-4567-89edc-baabc-de1234567/d-ABCD1234/deployment-archive/appspec.yml' already exists

This is occurring even when an instance is newly launched, with a fresh install of the CodeDeploy agent. Therefore, when Scaling Out an ASG, the scale-outs fail, if using the new version of the agent. Then, this causes a “scaling loop” to occur.

To rectify this, we have rolled out a downgrade of the agent to use version 1.1.0. This appears to resolve the issue.

From what I can see, this appears to be affecting Linux-based instances (Amazon Linux 1 and 2 from my testing) however, others may be affected as well.

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 4
Comments: 16 (6 by maintainers)

Most upvoted comments

@brblck - I certainly would be interested in testing the pre-release of v1.1.2, yes.

In regards to your comment, and @AnandarajuCS, the .zip files in this case were/are generated by Atlassian Bamboo so, it’s interesting to hear that the deployment bundles could be malformed/non-standard.

Aside from this, I’m certainly happy to test!

davidreid on Jul 15, 2020

@siarzukpiatrouski @davidreid @leytonreed thanks for your help in assisting our diagnosis of the issue.

After much analysis, while it looks similar to a previous CodeDeploy agent issue, its actually a much more nuanced edge case that’s resulting in a similar behavior.

The core issue is that the agent is running into malformed / non-standard zip binaries during the DownloadBundle step and its resulting in a partial unpack or full failure of the unpack. When the scenario occurs, we have some failover logic that attempts to salvage the situation and that logic was actually generating a failure under these specific conditions.

We believe we have a good solution to the issue and we’re working to prepare a v1.1.2 release of the agent that addresses it.

Would any of you be interested in helping us test a pre-release of v1.1.2 to validate that it addresses the issue you from your perspective?

brndnblck on Jul 15, 2020