aws-codedeploy-agent: Agent starts to fail and unable to correct self.
OS: Amazon Linux Version: OFFICIAL_1.0-1.1231_rpm Issue: Agent will be unresponsive and unable to receive deployments. Reboot / restarting the agent does not fix. Logs:
2017-09-19 14:47:51 INFO [codedeploy-agent(4128)]: master 4128: Received CHLD - cleaning dead child process
2017-09-19 14:47:51 INFO [codedeploy-agent(4128)]: master 4128: been told to replace child 4383
2017-09-19 14:47:51 INFO [codedeploy-agent(4128)]: master 4128: not enough child processes running - missing at least 1 - respawning
2017-09-19 14:47:56 INFO [codedeploy-agent(4128)]: master 4128: Spawned child 1/1
2017-09-19 14:47:56 ERROR [codedeploy-agent(4396)]: booting child: error during start or run: NoMethodError - undefined method `underscore' for Seahorse::Util:Module - /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:45:in `command'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:70:in `<class:CommandExecutor>'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:14:in `<module:CodeDeployPlugin>'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:12:in `<module:Plugins>'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:11:in `<module:InstanceAgent>'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:10:in `<top (required)>'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/register_plugin.rb:2:in `require_relative'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/register_plugin.rb:2:in `<top (required)>'
/usr/local/rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:120:in `require'
/usr/local/rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:120:in `require'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:16:in `block in load_plugins'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:12:in `each'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:12:in `load_plugins'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:28:in `prepare_run'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:64:in `block in prepare_run_with_error_handling'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:55:in `with_error_handling'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:63:in `prepare_run_with_error_handling'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:20:in `start'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:206:in `block in spawn_child'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:in `fork'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:in `spawn_child'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:283:in `block (2 levels) in replace_terminated_children'
/opt/codedeploy-agent/vendor/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
Solution is to remove all locally installed gems via:
gem uninstall -aIx
Uninstall the codedeploy agent, remove everything from /opt/codedeploy and reinstall.
We are using the puppet module to install which has been working great.
The one thing that I think that might be happening is our rails apps are installing a local version of the aws-sdk that is older or newer than what the agent supports. But that’s just a hunch.
This issue has been randomly popping up during deployments and has been causing a great amount of grief lately. To the point we are looking to switch to blue / green just to get around it. Unfortunately that deployment strategy has it’s own issues to deal with.
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 5
- Comments: 35 (8 by maintainers)
I solved this issue by modifying the init script this way: Replace:
By:
This will force ruby to use gem dependencies from the codedeploy directory and so will not conflict with system installed gem.
@sc-chad for us, it was something with global dependencies installed on the box. In our narrow scope case, our app’s dependency had the same dependent as the CodeDeploy agent, but with different versions. Wiping, installing CodeDeploy first, then our app’s works but only by luck (nothing broke in our app’s dependency with the mismatched peer dependency).
This CodeDeploy agent really needs to be in a container or bundled application where code is already rolled up.
I’ve implemented the change suggested by @marcmillien in PR #175
We’re experiencing the same problem. For us, the cause that we installed the latest AWS SDK for another Ruby script. It appears that the latest CodeDeploy agent is not compatible with the latest Ruby AWS SDK.
So basically, steps to reproduce:
gem install aws-sdkAny chance this can be fixed? The original ticket was opened more than 2 years ago…
Any solution on this?
2017-12-11 15:43:27 ERROR [codedeploy-agent(16258)]: booting child: error during start or run: NoMethodError - undefined method
underscore' for Seahorse::Util:Module - /root/aws-codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:51:incommand’ /root/aws-codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:76:in<class:CommandExecutor>' /root/aws-codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:20:inmodule:CodeDeployPlugin’ /root/aws-codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:18:in<module:Plugins>' /root/aws-codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:17:inmodule:InstanceAgent’ /root/aws-codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:16:in<top (required)>' /root/aws-codedeploy-agent/lib/instance_agent/plugins/codedeploy/register_plugin.rb:2:inrequire_relative’ /root/aws-codedeploy-agent/lib/instance_agent/plugins/codedeploy/register_plugin.rb:2:in<top (required)>' /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:120:inrequire’ /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:120:inrequire' /root/aws-codedeploy-agent/lib/instance_agent/runner/child.rb:16:inblock in load_plugins’ /root/aws-codedeploy-agent/lib/instance_agent/runner/child.rb:12:ineach' /root/aws-codedeploy-agent/lib/instance_agent/runner/child.rb:12:inload_plugins’ /root/aws-codedeploy-agent/lib/instance_agent/runner/child.rb:28:inprepare_run' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:64:inblock in prepare_run_with_error_handling’ /root/aws-codedeploy-agent/lib/instance_agent/runner/child.rb:55:inwith_error_handling' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:63:inprepare_run_with_error_handling’ /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:20:instart' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:206:inblock in spawn_child’ /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:infork' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:inspawn_child’ /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:283:inblock (2 levels) in replace_terminated_children' /var/lib/gems/2.3.0/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:inblock in create_with_logging_context’ 2017-12-11 15:43:27 INFO [codedeploy-agent(15390)]: master 15390: Received CHLD - cleaning dead child process 2017-12-11 15:43:27 INFO [codedeploy-agent(15390)]: master 15390: been told to replace child 16258 2017-12-11 15:43:27 INFO [codedeploy-agent(15390)]: master 15390: not enough child processes running - missing at least 1 - respawning 2017-12-11 15:43:32 INFO [codedeploy-agent(15390)]: master 15390: Spawned child 1/1 2017-12-11 15:43:32 ERROR [codedeploy-agent(16262)]: booting child: error during start or run: NoMethodError - undefined methodunderscore' for Seahorse::Util:Module - /root/aws-codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:51:incommand’ /root/aws-codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:76:in<class:CommandExecutor>' /root/aws-codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:20:inmodule:CodeDeployPlugin’ /root/aws-codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:18:in<module:Plugins>' /root/aws-codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:17:inmodule:InstanceAgent’ /root/aws-codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:16:in<top (required)>' /root/aws-codedeploy-agent/lib/instance_agent/plugins/codedeploy/register_plugin.rb:2:inrequire_relative’ /root/aws-codedeploy-agent/lib/instance_agent/plugins/codedeploy/register_plugin.rb:2:in<top (required)>' /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:120:inrequire’ /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:120:inrequire' /root/aws-codedeploy-agent/lib/instance_agent/runner/child.rb:16:inblock in load_plugins’ /root/aws-codedeploy-agent/lib/instance_agent/runner/child.rb:12:ineach' /root/aws-codedeploy-agent/lib/instance_agent/runner/child.rb:12:inload_plugins’ /root/aws-codedeploy-agent/lib/instance_agent/runner/child.rb:28:inprepare_run' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:64:inblock in prepare_run_with_error_handling’ /root/aws-codedeploy-agent/lib/instance_agent/runner/child.rb:55:inwith_error_handling' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:63:inprepare_run_with_error_handling’ /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:20:instart' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:206:inblock in spawn_child’ /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:infork' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:inspawn_child’ /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:283:inblock (2 levels) in replace_terminated_children' /var/lib/gems/2.3.0/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:inblock in create_with_logging_context’ 2017-12-11 15:43:32 INFO [codedeploy-agent(15390)]: master 15390: Received CHLD - cleaning dead child process 2017-12-11 15:43:32 INFO [codedeploy-agent(15390)]: master 15390: been told to replace child 16262 2017-12-11 15:43:32 INFO [codedeploy-agent(15390)]: master 15390: not enough child processes running - missing at least 1 - respawning ^CThis happened to us last night and seems to correlate with the code-deploy agent getting updated to a newer version. The above workaround seems to have resolved things for us.