cloudstack: Ceph RGW Object Storage couldn't be added (Invalid credentials or URL)
ISSUE TYPE
- Bug Report
- Improvement Request
- Enhancement Request
- Feature Idea
- Documentation Report
- Other
COMPONENT NAME
Infrastructure > Object Store
CLOUDSTACK VERSION
4.20.0.0
CONFIGURATION
OS / ENVIRONMENT
Ceph version: 19.2.0 squid (stable)
SUMMARY
STEPS TO REPRODUCE
- Create a Object Gateway user on Ceph Dashboard. The user is added under a Tenant.
- Try to add object store using the Ceph provider.
EXPECTED RESULTS
Being able to add a Ceph object store to my infrastructure.
ACTUAL RESULTS
Failed with error:
Request failed. (530)
Failed to add object store: Error while initializing Ceph RGW Object Store. Invalid credentials or URL
ERROR LOG
2024-12-05 05:54:03,154 DEBUG [c.c.a.ApiServlet] (qtp2038105753-397:[ctx-d6792894]) (logid:de1e0941) ===START=== 10.0.113.1 -- GET name=Ceph+Test+1&provider=Ceph&url=https:%2F%2F<URL_HERE>%2F&command=addObjectStoragePool&response=json&sessionkey=_IgdzVMijOi6635kzbViqZtkR2A
2024-12-05 05:54:03,154 DEBUG [c.c.a.ApiServlet] (qtp2038105753-397:[ctx-d6792894]) (logid:de1e0941) Two factor authentication is already verified for the user 2, so skipping
2024-12-05 05:54:03,161 DEBUG [c.c.a.ApiServer] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"c2a81095-5469-11ef-8cfa-005056a0391c"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-12-05 05:54:03,163 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-12-05 05:54:03,163 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-12-05 05:54:03,176 INFO [o.a.c.s.d.l.CephObjectStoreLifeCycleImpl] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Attempting to connect to Ceph RGW at https://<URL_HERE>/ with access key <ACCESS_KEY_HERE>
2024-12-05 05:54:03,199 DEBUG [o.a.c.s.d.l.CephObjectStoreLifeCycleImpl] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Error while initializing Ceph RGW Object Store: Cannot invoke "java.util.List.stream()" because the return value of "org.twonote.rgwadmin4j.impl.RgwAdminImpl.listBucketInfo()" is null
2024-12-05 05:54:03,199 DEBUG [c.c.s.StorageManagerImpl] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Failed to add object store: Error while initializing Ceph RGW Object Store. Invalid credentials or URL java.lang.RuntimeException: Error while initializing Ceph RGW Object Store. Invalid credentials or URL
at org.apache.cloudstack.storage.datastore.lifecycle.CephObjectStoreLifeCycleImpl.initialize(CephObjectStoreLifeCycleImpl.java:88)
at com.cloud.storage.StorageManagerImpl.discoverObjectStore(StorageManagerImpl.java:4140)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:105)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy130.discoverObjectStore(Unknown Source)
at org.apache.cloudstack.api.command.admin.storage.AddObjectStoragePoolCmd.execute(AddObjectStoragePoolCmd.java:115)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:831)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:652)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:354)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:157)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:154)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:108)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:840)
2024-12-05 05:54:03,200 WARN [o.a.c.m.w.WebhookServiceImpl] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Skipping delivering event [ID: null, description: {"event":"OBJECT.STORE.CREATE","status":"Completed"}] to any webhook as account ID is missing
2024-12-05 05:54:03,202 WARN [o.a.c.f.e.EventDistributorImpl] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Failed to publish event [category: ActionEvent, type: OBJECT.STORE.CREATE] on bus webhookEventBus
2024-12-05 05:54:03,213 ERROR [o.a.c.a.c.a.s.AddObjectStoragePoolCmd] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Exception: com.cloud.utils.exception.CloudRuntimeException: Failed to add object store: Error while initializing Ceph RGW Object Store. Invalid credentials or URL
at com.cloud.storage.StorageManagerImpl.discoverObjectStore(StorageManagerImpl.java:4145)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:105)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy130.discoverObjectStore(Unknown Source)
at org.apache.cloudstack.api.command.admin.storage.AddObjectStoragePoolCmd.execute(AddObjectStoragePoolCmd.java:115)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:831)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:652)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:354)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:157)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:154)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:108)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.RuntimeException: Error while initializing Ceph RGW Object Store. Invalid credentials or URL
at org.apache.cloudstack.storage.datastore.lifecycle.CephObjectStoreLifeCycleImpl.initialize(CephObjectStoreLifeCycleImpl.java:88)
at com.cloud.storage.StorageManagerImpl.discoverObjectStore(StorageManagerImpl.java:4140)
... 67 more
2024-12-05 05:54:03,213 INFO [c.c.a.ApiServer] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Failed to add object store: Error while initializing Ceph RGW Object Store. Invalid credentials or URL
2024-12-05 05:54:03,214 INFO [c.c.a.ApiServlet] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) (userId=2 accountId=2 sessionId=node0a1m8rqhb8vwl17sdi3x5937fp3) 10.0.113.1 -- GET name=Ceph+Test+1&provider=Ceph&url=https:%2F%2F<URL_HERE>%2F&command=addObjectStoragePool&response=json&sessionkey=_IgdzVMijOi6635kzbViqZtkR2A 530 Failed to add object store: Error while initializing Ceph RGW Object Store. Invalid credentials or URL
2024-12-05 05:54:03,214 DEBUG [c.c.a.ApiServlet] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) ===END=== 10.0.113.1 -- GET name=Ceph+Test+1&provider=Ceph&url=https:%2F%2F<URL_HERE>%2F&command=addObjectStoragePool&response=json&sessionkey=_IgdzVMijOi6635kzbViqZtkR2A
It might be something related to org.twonote.rgwadmin4j.impl.RgwAdminImpl.listBucketInfo(). Note that I’m inserting the base URL of my Ceph RGW endpoint. The endpoint should support both path and virtual host based URL (tested with WinSCP). No bucket created at all for the user.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 28 (6 by maintainers)
@hanisirfan @abh1sar i have exactly the same issue and was able to track down the issue to Cloudstacks usage of the S3 library being used.
The library is sending the request including the following tag:
<CreateBucketConfiguration xmlns=#22http://s3.amazonaws.com/doc/2006-03-01/#22><LocationConstraint>stepbee</LocationConstraint></CreateBucketConfiguration>the
<LocationContraint>tag, or actually the content, is not what (at least my version of) Ceph is expecting/accepting. It comes down to the following line of code in CloudStack https://github.com/apache/cloudstack/blob/864751d5f990d2848b3b00559202a4a590193c78/plugins/storage/object/ceph/src/main/java/org/apache/cloudstack/storage/datastore/driver/CephObjectStoreDriverImpl.java#L351 where the “auto” is instructing the S3 library to construct some s3 region string, which is not accepted by (at least my version of) Ceph.@abh1sar unfortunately not yet. As we are running a reverse proxy in front of our S3 backend, i tried to replace the LocationContraint on the fly but have not been successful without spending too much time on the approach.
So far, i am not using the Object Storage feature because of this issue.
If your Ceph S3 Storage is a fresh installation and used for this purpose only, you could try to add a zonegroup in ceph rgw matching the name, Cloudstack is providing in the LocationContraint tag as described in https://docs.ceph.com/en/latest/radosgw/placement/