cloudstack: The cloudstack-agent can not work 4.18.1.0

ISSUE TYPE

  • Bug Report

COMPONENT NAME

  • 4.18.1.0

CLOUDSTACK VERSION

  1. Cloudstack management (fresh install)
  • OS: Ubuntu 22.04
  • Cloudstack-management: 4.18.1.0
  1. Cloudstack agent (fresh install)
  • OS: Ubuntu 22.04
  • Cloudstack-agent: 4.18.1.0

SUMMARY

After installing cloudstack-agent it won’t start.

Nov 12 22:25:13 dg42a-csrv-3 systemd[1]: Stopped CloudStack Agent.
░░ Subject: A stop job for unit cloudstack-agent.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A stop job for unit cloudstack-agent.service has finished.
░░
░░ The job identifier is 28655 and the job result is done.
Nov 12 22:25:13 dg42a-csrv-3 systemd[1]: cloudstack-agent.service: Consumed 2.244s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit cloudstack-agent.service completed and consumed the indicated resources.
Nov 12 22:25:13 dg42a-csrv-3 systemd[1]: Started CloudStack Agent.
░░ Subject: A start job for unit cloudstack-agent.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit cloudstack-agent.service has finished successfully.
░░
░░ The job identifier is 28655.
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: log4j:WARN No appenders could be found for logger (com.cloud.agent.AgentShell).
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: log4j:WARN Please initialize the log4j system properly.
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: INFO  [cloud.agent.AgentShell] (main:) (logid:) Agent started
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: INFO  [cloud.agent.AgentShell] (main:) (logid:) Implementation Version is 4.18.1.0
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: INFO  [cloud.agent.AgentShell] (main:) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: SLF4J: Defaulting to no-operation (NOP) logger implementation
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: Unable to find the guid
Nov 12 22:25:14 dg42a-csrv-3 systemd[1]: cloudstack-agent.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit cloudstack-agent.service has successfully entered the 'dead' state.
Nov 12 22:25:14 dg42a-csrv-3 systemd[1]: cloudstack-agent.service: Consumed 2.239s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit cloudstack-agent.service completed and consumed the indicated resources.

Error log from cloudstack-management server

2023-11-13 00:34:31,480 WARN  [o.a.c.a.c.a.h.AddHostCmd] (qtp1278852808-536:ctx-a43770e7 ctx-30b97ea5) (logid:4785c123) Exception:
com.cloud.exception.DiscoveryException: Could not add host at [http://10.64.11.34] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34].
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:823)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:649)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy200.discoverHosts(Unknown Source)
        at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:782)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:603)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:347)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:154)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:151)
        at com.cloud.api.ApiServlet.doPost(ApiServlet.java:110)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:376)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:649)
        ... 59 more
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)
        ... 60 more
2023-11-13 00:34:31,482 INFO  [c.c.a.ApiServer] (qtp1278852808-536:ctx-a43770e7 ctx-30b97ea5) (logid:4785c123) Could not add host at [http://10.64.11.34] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34].

About this issue

  • Original URL
  • State: open
  • Created 8 months ago
  • Comments: 15 (5 by maintainers)

Most upvoted comments

There are few caveats in installing KVM on Ubuntu 22.04. Those are not yet covered in the project documentation. One is an ability for an agent host to register itself within the mgmt host, which is done somewhat counter intuitively. To resolve your issue, do this:

  1. Update libvirt configs, following the doc
  2. On the agent host: sudo nano /etc/cloudstack/agent/agent.properties and set proper values for UUID, host and bridges
  3. Copy cloudstack generated ssh key to the authorized_keys on a mgmt host:
cat id_rsa.pub.cloud >> /root/.ssh/authorized_keys
  1. Restart libvirt and agent and make sure they started properly:
sudo systemctl restart libvirtd.service
sudo systemctl restart cloudstack-agent.service
  1. If everything is ok, switch to the mgmt service UI config. When you get to adding the host, specify for root user and click the SSH Public Key Access to login with the autogenerated key.

NOTE: You’ll probably face another errors, which you would report here or open another issue 😏😏😏

yes @tampler If you do not want to use root password, ssh key is a good alternative. it just works.

If you use a non-root user, /etc/sudoers should be configured. I suspect the following configuration is correct for non-root users.

cloudstack ALL=NOPASSWD: /usr/bin/cloudstack-setup-agent
Defaults:cloudstack !requiretty

https://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/kvm.html#install-and-configure-the-agent

+2
weizhouapache on Dec 1, 2023
Read more comments on GitHub
← cloudstack: SSO fails with error "Expired session, missing signature, or missing apiKey"
cloudstack: The System VMs Agent state is not Up after upgrade to CloudStack 4.17.1.0 →