rook: Object and Smoke suites are failing with object error on latest devel images

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior: The latest ceph image quay.io/ceph/daemon-base:latest-squid-devel is causing the smoke and object test suites to fail with the following errors, starting today, for example see this job:

=== RUN   TestCephSmokeSuite/TestObjectStorage_SmokeTest/check_if_the_dashboard-admin_user_exists_in_all_existing_object_stores
2024-10-31 13:50:27.346450 D | exec: Running command: kubectl exec -i rook-ceph-tools-8896d8f44-jtb5n -n smoke-ns -- timeout 15 radosgw-admin user info --uid=dashboard-admin --rgw-realm=lite-store
2024-10-31 13:50:28.241354 W | installer: Error executing command "radosgw-admin": <exit status 22>
2024-10-31 13:50:28.241400 I | integrationTest: output: . could not fetch user info: no user info saved
command terminated with exit code 22
    ceph_base_object_test.go:182: 
        	Error Trace:	/home/runner/work/rook/rook/tests/integration/ceph_base_object_test.go:182
        	Error:      	Received unexpected error:
        	            	exit status 22
        	Test:       	TestCephSmokeSuite/TestObjectStorage_SmokeTest/check_if_the_dashboard-admin_user_exists_in_all_existing_object_stores

And from the object store suite:

=== RUN   TestCephObjectSuite/TestWithTLS/check_if_the_dashboard-admin_user_exists_in_all_existing_object_stores
2024-10-31 13:41:22.974266 D | exec: Running command: kubectl exec -i rook-ceph-tools-8896d8f44-k7mj5 -n object-ns -- timeout 15 radosgw-admin user info --uid=dashboard-admin --rgw-realm=tls-test-store
2024-10-31 13:41:24.214460 W | installer: Error executing command "radosgw-admin": <exit status 22>
2024-10-31 13:41:24.214482 I | integrationTest: output: . could not fetch user info: no user info saved
command terminated with exit code 22
    ceph_base_object_test.go:182: 
        	Error Trace:	/home/runner/work/rook/rook/tests/integration/ceph_base_object_test.go:182
        	Error:      	Received unexpected error:
        	            	exit status 22
        	Test:       	TestCephObjectSuite/TestWithTLS/check_if_the_dashboard-admin_user_exists_in_all_existing_object_stores
2024-10-31 13:41:24.214611 I | integrationTest: Testing Object Operations on tls-test-store
2024-10-31 13:41:24.217758 I | integrationTest: Create Object Store "bucket-notification-tls-test-store" with replica count 1
=== NAME  TestCephObjectSuite/TestWithTLS
    ceph_bucket_notification_test.go:56: 
        	Error Trace:	/home/runner/work/rook/rook/tests/integration/ceph_bucket_notification_test.go:56
        	            				/home/runner/work/rook/rook/tests/integration/ceph_object_test.go:153
        	            				/home/runner/work/rook/rook/tests/integration/ceph_object_test.go:102
        	Error:      	Expected nil, but got: &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"cephobjectstores.ceph.rook.io \"bucket-notification-tls-test-store\" not found", Reason:"NotFound", Details:(*v1.StatusDetails)(0xc0007d9020), Code:[404](https://github.com/rook/rook/actions/runs/11612886250/job/32337370190#step:4:405)}}
        	Test:       	TestCephObjectSuite/TestWithTLS
    ceph_bucket_notification_test.go:58: 
        	Error Trace:	/home/runner/work/rook/rook/tests/integration/ceph_bucket_notification_test.go:58
        	            				/home/runner/work/rook/rook/tests/integration/ceph_object_test.go:153
        	            				/home/runner/work/rook/rook/tests/integration/ceph_object_test.go:102
        	Error:      	Expected nil, but got: failed to get advertise endpoint for object store "/": At least one of Port or SecurePort should be non-zero
        	Test:       	TestCephObjectSuite/TestWithTLS

Expected behavior: Tests should pass.

How to reproduce it (minimal and precise):

See the CI in the latest PRs that have run today.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (18 by maintainers)

Most upvoted comments

Re-running the tests passed now. A newer latest-devel-reef tag was generated since the latest daily tests were last run, so I believe the tag from yesterday just hadn’t picked up the changes yet for some reason. It all looks good now, thanks!

thank you @subhamkrai . I can see from the log its the same error. dashboard and prometheus is not starting.

debug 2024-10-31T13:39:26.638+0000 7fd6fdb580c0 -1 mgr[py] Module not found: 'prometheus'
debug 2024-10-31T13:39:26.638+0000 7fd6fdb580c0 -1 mgr[py] Traceback (most recent call last):
  File "/usr/share/ceph/mgr/prometheus/__init__.py", line 2, in <module>
    from .module import Module, StandbyModule
  File "/usr/share/ceph/mgr/prometheus/module.py", line 38, in <module>
    v = Version(cherrypy.__version__)
  File "/usr/lib/python3.9/site-packages/packaging/version.py", line 298, in __init__
    raise InvalidVersion("Invalid version: '{0}'".format(version))
packaging.version.InvalidVersion: Invalid version: 'unknown'

debug 2024-10-31T13:39:26.638+0000 7fd6fdb580c0 -1 mgr[py] Class not found in module 'prometheus'
debug 2024-10-31T13:39:26.638+0000 7fd6fdb580c0 -1 mgr[py] Error loading module 'prometheus': (2) No such file or directory
debug 2024-10-31T13:39:26.638+0000 7fd6fdb580c0  1 mgr[py] Loading python module 'dashboard'
debug 2024-10-31T13:39:27.054+0000 7fd6fdb580c0 -1 mgr[py] Module not found: 'dashboard'
debug 2024-10-31T13:39:27.054+0000 7fd6fdb580c0 -1 mgr[py] Traceback (most recent call last):
  File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in <module>
    from .module import Module, StandbyModule  # noqa: F401
  File "/usr/share/ceph/mgr/dashboard/module.py", line 52, in <module>
    patch_cherrypy(cherrypy.__version__)
  File "/usr/share/ceph/mgr/dashboard/cherrypy_backports.py", line 197, in patch_cherrypy
    accept_exceptions_from_builtin_ssl(ver)
  File "/usr/share/ceph/mgr/dashboard/cherrypy_backports.py", line 113, in accept_exceptions_from_builtin_ssl
    patch_builtin_ssl_wrap(v, accept_ssl_errors)
  File "/usr/share/ceph/mgr/dashboard/cherrypy_backports.py", line 75, in patch_builtin_ssl_wrap
    from cherrypy.wsgiserver.ssl_builtin import BuiltinSSLAdapter as builtin_ssl
ModuleNotFoundError: No module named 'cherrypy.wsgiserver'

debug 2024-10-31T13:39:27.054+0000 7fd6fdb580c0 -1 mgr[py] Class not found in module 'dashboard'

Should be fixed by https://github.com/ceph/ceph/pull/60602