aws-codedeploy-agent: Intermittent CodeDeploy Agent SSL_connect error

We are running the latest version of CodeDeploy Agent on Windows Server 2019 EC2 instances. Occasionally, the agent fails, and the logs are full of NetworkingError SSL_connect. Restarting the Agent resolves the problem until the next such occurrence.

This happens across multiple regions/accounts.

`2019-04-24T19:10:47 DEBUG [codedeploy-agent(2428)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: PollHostCommand: Host Command = nil 2019-04-24T19:10:48 DEBUG [codedeploy-agent(2428)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand: 2019-04-24T19:10:48 INFO [codedeploy-agent(2428)]: Version file found in C:/ProgramData/Amazon/CodeDeploy/.version with agent version OFFICIAL_1.0.1.1597_msi. 2019-04-24T19:11:49 INFO [codedeploy-agent(2428)]: [Aws::CodeDeployCommand::Client 200 60.093372 0 retries] poll_host_command(host_identifier:“instance_arn”)

2019-04-24T19:11:49 DEBUG [codedeploy-agent(2428)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: PollHostCommand: Host Command = nil 2019-04-24T19:11:50 DEBUG [codedeploy-agent(2428)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand: 2019-04-24T19:11:50 INFO [codedeploy-agent(2428)]: Version file found in C:/ProgramData/Amazon/CodeDeploy/.version with agent version OFFICIAL_1.0.1.1597_msi. 2019-04-24T19:11:52 INFO [codedeploy-agent(2428)]: [Aws::CodeDeployCommand::Client 0 2.197781 3 retries] poll_host_command(host_identifier:“instance_arn”) Seahorse::Client::NetworkingError SSL_connect returned=1 errno=0 state=error: certificate verify failed `

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 2
Comments: 31 (5 by maintainers)

Most upvoted comments

Is there any update on this issue? This is preventing us from reliably deploying to our instances, as from time to time we have to restart the AWS CodeDeploy Agent service to make it work again.

advanw on Oct 31, 2019

I was in contact with AWS support about this. The reply was very interesting/useful so I’ll share it here.

I have done some research internally and found that this error is caused by Windows SilentCleanup deleting temp folders which in turn causes the CodeDeploy agent to encounter errors. The CodeDeploy agent is not yet officially supported on Windows Server 2019 [2], however I found a workaround for this that may work here:

You can prevent the temp folder from being deleted by disabling SilentCleanup on the instance, to do this you can run the following command [3,4]:

schtasks.exe /change /TN “\Microsoft\Windows\DiskCleanup\SilentCleanup” /Disable

However I cannot guarantee that this will fully fix the error, in my tests I was not able to reproduce the error once I disabled SilentCleanup however as the error is intermittent I cannot guarantee it. The CodeDeploy team is aware of this issue and they are working on finding a long term resolution, I have added your case to the internal ticket regarding this. Currently there is no ETA for when this will be resolved, you can keep an eye on the GitHub issue ticket [1] and the CodeDeploy release history [5] for updates.

[1] https://github.com/aws/aws-codedeploy-agent/issues/208 [2] https://docs.aws.amazon.com/codedeploy/latest/userguide/codedeploy-agent.html#codedeploy-agent-supported-operating-systems-ec2 [3] https://support.microsoft.com/en-us/help/4015218/activex-is-automatically-deleted-in-the-windows-10-x86-environment#3-3 [4] https://pupuweb.com/windows-server-2019-bug-silentcleanup-delete-temp-tmp-folder/ [5] https://docs.aws.amazon.com/codedeploy/latest/userguide/document-history.html

pieterdw on Nov 27, 2019

Reboot didn’t fix it for me. Neither reinstalling did. The only way I could fix it was to set a cron task to restart the agent every night - which unfortunately causes a few minutes downtime.

Would be great if someone in the agent team (or maybe it’s mainly related to the API) could take a look at the issue.

tbronchain on Nov 12, 2019