aws-codedeploy-agent: InstanceAgent::LinuxUtil: Error extracting zip archive: 1
codedeploy first deployment DownloadBundle UnknownError Destination ‘/opt/codedeploy-agent/deployment-root/c0f1c34f-4c73-43c5-a70e-88ea4a78b2e1/d-5SHGGURRB/deployment-archive/deploy/codedeploy/start.sh’ already exists
/var/log/aws/codedeploy-agent/codedeploy-agent.log Prompt the following error
2019-07-24 15:42:12 INFO [codedeploy-agent(5669)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.0-1.1758_rpm. 2019-07-24 15:42:12 INFO [codedeploy-agent(5669)]: [Aws::CodeDeployCommand::Client 200 0.024945 0 retries] get_deployment_specification(deployment_execution_id:“CodeDeploy/us-east-1/prod/orpheus:public001/arn:aws:sds:us-east-1:821863646904:deployment/d-5SHGGURRB”,host_identifier:“arn:aws:ec2:us-east-1:821863646904:instance/i-0be94b81205326aae”)
2019-07-24 15:42:12 ERROR [codedeploy-agent(5669)]: InstanceAgent::LinuxUtil: Error extracting zip archive: 1 2019-07-24 15:42:12 INFO [codedeploy-agent(5669)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.0-1.1758_rpm. 2019-07-24 15:42:12 INFO [codedeploy-agent(5669)]: [Aws::CodeDeployCommand::Client 200 0.06706 0 retries] put_host_command_complete(command_status:“Failed”,diagnostics:{format:“JSON”,payload:“{"error_code":5,"script_name":"","message":"Destination ‘/opt/codedeploy-agent/deployment-root/c0f1c34f-4c73-43c5-a70e-88ea4a78b2e1/d-5SHGGURRB/deployment-archive/deploy/codedeploy/start.sh’ already exists","log":""}”},host_command_identifier:“eyJiYXRjaElkIjoiYmI5MTljNzUwNzVkNWY0YjM5NTA4OGUwMzgzN2M2NmUvcHVibGljMDAxIiwiZGVwbG95bWVudElkIjoiQ29kZURlcGxveS91cy1lYXN0LTEvcHJvZC9vcnBoZXVzOnB1YmxpYzAwMS9hcm46YXdzOnNkczp1cy1lYXN0LTE6ODIxODYzNjQ2OTA0OmRlcGxveW1lbnQvZC01U0hHR1VSUkIiLCJob3N0SWQiOiJhcm46YXdzOmVjMjp1cy1lYXN0LTE6ODIxODYzNjQ2OTA0Omluc3RhbmNlL2ktMGJlOTRiODEyMDUzMjZhYWUiLCJjb21tYW5kSWQiOiJBcG9sbG9EZXBsb3lDb250cm9sU2VydmljZXxhcm46YXdzOmVjMjp1cy1lYXN0LTE6ODIxODYzNjQ2OTA0Omluc3RhbmNlL2ktMGJlOTRiODEyMDUzMjZhYWV8MnwwIiwiY29tbWFuZE5hbWUiOiJEb3dubG9hZEJ1bmRsZSIsImNvbW1hbmRJbmRleCI6MiwiYXR0ZW1wdEluZGV4IjoxfQ==”)
2019-07-24 15:42:12 ERROR [codedeploy-agent(5669)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during perform: Zip::DestinationFileExistsError - Destination ‘/opt/codedeploy-agent/deployment-root/c0f1c34f-4c73-43c5-a70e-88ea4a78b2e1/d-5SHGGURRB/deployment-archive/deploy/codedeploy/start.sh’ already exists - /opt/codedeploy-agent/vendor/gems/rubyzip-1.1.7/lib/zip/entry.rb:579:in create_file' /opt/codedeploy-agent/vendor/gems/rubyzip-1.1.7/lib/zip/entry.rb:154:in extract’
/opt/codedeploy-agent/vendor/gems/rubyzip-1.1.7/lib/zip/file.rb:301:in extract' /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:400:in block (2 levels) in unpack_bundle’
/opt/codedeploy-agent/vendor/gems/rubyzip-1.1.7/lib/zip/entry_set.rb:42:in call' /opt/codedeploy-agent/vendor/gems/rubyzip-1.1.7/lib/zip/entry_set.rb:42:in block in each’
/opt/codedeploy-agent/vendor/gems/rubyzip-1.1.7/lib/zip/entry_set.rb:41:in each' /opt/codedeploy-agent/vendor/gems/rubyzip-1.1.7/lib/zip/entry_set.rb:41:in each’
/opt/codedeploy-agent/vendor/gems/rubyzip-1.1.7/lib/zip/central_directory.rb:182:in each' /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:397:in block in unpack_bundle’
/opt/codedeploy-agent/vendor/gems/rubyzip-1.1.7/lib/zip/file.rb:99:in open' /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:396:in rescue in unpack_bundle’
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:393:in unpack_bundle' /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:113:in block in class:CommandExecutor’
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:68:in execute_command' /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:114:in process_command’
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:96:in acknowledge_and_process_command' /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:75:in block in perform’
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in call' /opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in run_task’
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in block (3 levels) in create_worker' /opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in loop’
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in block (2 levels) in create_worker' /opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in catch’
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in block in create_worker' /opt/codedeploy-agent/vendor/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in call’
/opt/codedeploy-agent/vendor/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context’
2019-07-24 15:42:12 WARN [codedeploy-agent(5669)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PutHostCommandComplete: “Code Error”
2019-07-24 15:42:12 INFO [codedeploy-agent(5669)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.0-1.1758_rpm.
2019-07-24 15:42:12 INFO [codedeploy-agent(5669)]: [Aws::CodeDeployCommand::Client 200 0.016842 0 retries] put_host_command_complete(command_status:“Failed”,diagnostics:{format:“JSON”,payload:“{"error_code":5,"script_name":"","message":"Destination ‘/opt/codedeploy-agent/deployment-root/c0f1c34f-4c73-43c5-a70e-88ea4a78b2e1/d-5SHGGURRB/deployment-archive/deploy/codedeploy/start.sh’ already exists","log":""}”},host_command_identifier:“eyJiYXRjaElkIjoiYmI5MTljNzUwNzVkNWY0YjM5NTA4OGUwMzgzN2M2NmUvcHVibGljMDAxIiwiZGVwbG95bWVudElkIjoiQ29kZURlcGxveS91cy1lYXN0LTEvcHJvZC9vcnBoZXVzOnB1YmxpYzAwMS9hcm46YXdzOnNkczp1cy1lYXN0LTE6ODIxODYzNjQ2OTA0OmRlcGxveW1lbnQvZC01U0hHR1VSUkIiLCJob3N0SWQiOiJhcm46YXdzOmVjMjp1cy1lYXN0LTE6ODIxODYzNjQ2OTA0Omluc3RhbmNlL2ktMGJlOTRiODEyMDUzMjZhYWUiLCJjb21tYW5kSWQiOiJBcG9sbG9EZXBsb3lDb250cm9sU2VydmljZXxhcm46YXdzOmVjMjp1cy1lYXN0LTE6ODIxODYzNjQ2OTA0Omluc3RhbmNlL2ktMGJlOTRiODEyMDUzMjZhYWV8MnwwIiwiY29tbWFuZE5hbWUiOiJEb3dubG9hZEJ1bmRsZSIsImNvbW1hbmRJbmRleCI6MiwiYXR0ZW1wdEluZGV4IjoxfQ==”)
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 4
- Comments: 31 (10 by maintainers)
This took down our business in the morning, since we auto scaled in new hosts (which installed the agent via the “latest” script) and they were unable to do anything. We hit a wall at AWS support too.
PLEASE provide us a way to install a particular version, since we can no longer trust that you release stable versions of the agent. Maybe release the agent via
snapdor some other strategy - but overall improve your release mechanism; this script is not enough, you need to give customers an easy way to rollback the agent.Furthermore, when I was troubleshooting this issue, I googled “codedeploy agent releases” and the first result was https://github.com/aws/aws-codedeploy-agent/releases -> this had NO REFERENCE that anything had changed. So I spent hours troubleshooting what happened to our zip file only to find this. So much went wrong here, AWS 😕 Regressions in your data plane affect customers much harder than those in your control plane!
We have identified multiple possible causes for the unzip warning which is causing failed deployments. We are still testing internally to find any additional edge cases before we release an update publicly.
On Jul 18, we released a new version of the CodeDeploy agent in the SA-EAST-1 region. After 4 days with no reported issues, we released this agent in the US-EAST-1 region on Jul 23 then US-WEST-2 on Jul 24. We received reports the morning of the 24th of unzip issues impacting a small number of customers, and rolled back our release at 12pm PDT and are still investigating the root cause.
If you are impacted and the automated rollback has not yet resolved the issue, you can manually install the previous version of the agent by running this command:
sudo /opt/codedeploy-agent/bin/update autoWe will post here before we begin this release again. In addition to the unzip issue, we’re investigating moving automatic updates to an “opt-in” model rather than being “on-by-default”, and implementing notification mechanisms (ex. SNS topics) for new agent releases. In the meantime, you can disable automatic updates yourself by removing this cron job:
/etc/cron.d/codedeploy-agent-update@rohkat-aws
We follow standard US business hours, meaning that we start scaling up in anticipation for 6AM Pacific / 9AM Eastern. As recently as Jul 24, 2019 5:11AM Pacific the toxic agent was still being installed. At that point, we stopped scaling and instead tried other mechanisms, because we didn’t know a new version was released, and our system failed over due to insufficient scale soon thereafter. If you rolled back the agent after that, it doesn’t help us, because we already installed the agent that didn’t work.
Please help to answer the following questions, as the communication thus far is not helping with Earn Trust here…
I know that you cannot comment on particular features, but getting a clear path from your team is critical. Please escalate this.