rook: Manager prometheus module failed to construct - ImportError: Module not found
Hi,
After modified the cluster spec with this settings:
mgr:
...
modules:
- name: rook
enabled: true
- name: pg_autoscaler
enabled: true
- name: prometheus
enabled: true
...
monitoring:
enabled: true
The prometheus module does not starting.
debug 2024-01-08T10:02:03.658+0000 7fb850722200 0 set uid:gid to 167:167 (ceph:ceph)
debug 2024-01-08T10:02:03.658+0000 7fb850722200 0 ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable), process ceph-mgr, pid 13
debug 2024-01-08T10:02:03.658+0000 7fb850722200 0 pidfile_write: ignore empty --pid-file
debug 2024-01-08T10:02:03.694+0000 7fb850722200 1 mgr[py] Loading python module 'restful'
debug 2024-01-08T10:02:04.142+0000 7fb850722200 1 mgr[py] Loading python module 'progress'
debug 2024-01-08T10:02:04.282+0000 7fb850722200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:04.282+0000 7fb850722200 1 mgr[py] Loading python module 'telegraf'
debug 2024-01-08T10:02:04.394+0000 7fb850722200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:04.394+0000 7fb850722200 1 mgr[py] Loading python module 'cephadm'
debug 2024-01-08T10:02:05.262+0000 7fb850722200 1 mgr[py] Loading python module 'osd_support'
debug 2024-01-08T10:02:05.458+0000 7fb850722200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:05.458+0000 7fb850722200 1 mgr[py] Loading python module 'pg_autoscaler'
debug 2024-01-08T10:02:05.606+0000 7fb850722200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:05.606+0000 7fb850722200 1 mgr[py] Loading python module 'selftest'
debug 2024-01-08T10:02:05.754+0000 7fb850722200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:05.754+0000 7fb850722200 1 mgr[py] Loading python module 'prometheus'
debug 2024-01-08T10:02:06.326+0000 7fb850722200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:06.326+0000 7fb850722200 1 mgr[py] Loading python module 'localpool'
debug 2024-01-08T10:02:06.446+0000 7fb850722200 1 mgr[py] Loading python module 'balancer'
debug 2024-01-08T10:02:06.666+0000 7fb850722200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:06.666+0000 7fb850722200 1 mgr[py] Loading python module 'iostat'
debug 2024-01-08T10:02:06.802+0000 7fb850722200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:06.802+0000 7fb850722200 1 mgr[py] Loading python module 'snap_schedule'
debug 2024-01-08T10:02:06.982+0000 7fb850722200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:06.982+0000 7fb850722200 1 mgr[py] Loading python module 'orchestrator'
debug 2024-01-08T10:02:07.250+0000 7fb850722200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:07.250+0000 7fb850722200 1 mgr[py] Loading python module 'rook'
debug 2024-01-08T10:02:08.538+0000 7fb850722200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:08.538+0000 7fb850722200 1 mgr[py] Loading python module 'crash'
debug 2024-01-08T10:02:08.814+0000 7fb850722200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:08.814+0000 7fb850722200 1 mgr[py] Loading python module 'k8sevents'
debug 2024-01-08T10:02:09.298+0000 7fb850722200 1 mgr[py] Loading python module 'nfs'
debug 2024-01-08T10:02:09.638+0000 7fb850722200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:09.638+0000 7fb850722200 1 mgr[py] Loading python module 'status'
debug 2024-01-08T10:02:09.938+0000 7fb850722200 -1 mgr[py] Module status has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:09.938+0000 7fb850722200 1 mgr[py] Loading python module 'stats'
debug 2024-01-08T10:02:10.082+0000 7fb850722200 1 mgr[py] Loading python module 'mirroring'
debug 2024-01-08T10:02:10.234+0000 7fb850722200 1 mgr[py] Loading python module 'rbd_support'
debug 2024-01-08T10:02:10.398+0000 7fb850722200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:10.398+0000 7fb850722200 1 mgr[py] Loading python module 'alerts'
debug 2024-01-08T10:02:10.546+0000 7fb850722200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:10.546+0000 7fb850722200 1 mgr[py] Loading python module 'diskprediction_local'
debug 2024-01-08T10:02:10.754+0000 7fb850722200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:10.754+0000 7fb850722200 1 mgr[py] Loading python module 'volumes'
debug 2024-01-08T10:02:11.098+0000 7fb850722200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:11.098+0000 7fb850722200 1 mgr[py] Loading python module 'mds_autoscaler'
debug 2024-01-08T10:02:11.578+0000 7fb850722200 1 mgr[py] Loading python module 'telemetry'
debug 2024-01-08T10:02:11.794+0000 7fb850722200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:11.794+0000 7fb850722200 1 mgr[py] Loading python module 'osd_perf_query'
debug 2024-01-08T10:02:11.942+0000 7fb850722200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:11.942+0000 7fb850722200 1 mgr[py] Loading python module 'test_orchestrator'
debug 2024-01-08T10:02:12.222+0000 7fb850722200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:12.222+0000 7fb850722200 1 mgr[py] Loading python module 'zabbix'
debug 2024-01-08T10:02:12.354+0000 7fb850722200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:12.354+0000 7fb850722200 1 mgr[py] Loading python module 'rgw'
debug 2024-01-08T10:02:12.654+0000 7fb850722200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:12.654+0000 7fb850722200 1 mgr[py] Loading python module 'devicehealth'
debug 2024-01-08T10:02:12.842+0000 7fb850722200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:12.842+0000 7fb850722200 1 mgr[py] Loading python module 'insights'
debug 2024-01-08T10:02:13.030+0000 7fb850722200 1 mgr[py] Loading python module 'dashboard'
debug 2024-01-08T10:02:14.134+0000 7fb850722200 1 mgr[py] Loading python module 'influx'
debug 2024-01-08T10:02:14.282+0000 7fb850722200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member
debug 2024-01-08T10:02:14.286+0000 7fb843dd0700 0 ms_deliver_dispatch: unhandled message 0x557908acb600 mon_map magic: 0 v1 from mon.0 v2:172.31.16.198:3300/0
debug 2024-01-08T10:02:14.294+0000 7fb843dd0700 1 mgr handle_mgr_map Activating!
debug 2024-01-08T10:02:14.294+0000 7fb843dd0700 1 mgr handle_mgr_map I am now activating
debug 2024-01-08T10:02:14.310+0000 7fb7ff998700 0 [balancer DEBUG root] setting log level based on debug_mgr: INFO (2/5)
debug 2024-01-08T10:02:14.310+0000 7fb7ff998700 1 mgr load Constructed class from module: balancer
debug 2024-01-08T10:02:14.310+0000 7fb7e045d700 0 [balancer INFO root] Starting
debug 2024-01-08T10:02:14.310+0000 7fb7ff998700 0 [crash DEBUG root] setting log level based on debug_mgr: INFO (2/5)
debug 2024-01-08T10:02:14.310+0000 7fb7ff998700 1 mgr load Constructed class from module: crash
debug 2024-01-08T10:02:14.314+0000 7fb7ff998700 0 [dashboard DEBUG root] setting log level based on debug_mgr: INFO (2/5)
debug 2024-01-08T10:02:14.314+0000 7fb7ff998700 1 mgr load Constructed class from module: dashboard
debug 2024-01-08T10:02:14.314+0000 7fb7ff998700 0 [devicehealth DEBUG root] setting log level based on debug_mgr: INFO (2/5)
debug 2024-01-08T10:02:14.314+0000 7fb7ff998700 1 mgr load Constructed class from module: devicehealth
debug 2024-01-08T10:02:14.314+0000 7fb7de459700 0 [dashboard INFO access_control] Loading user roles DB version=2
debug 2024-01-08T10:02:14.314+0000 7fb7de459700 0 [dashboard INFO sso] Loading SSO DB version=1
debug 2024-01-08T10:02:14.314+0000 7fb7de459700 0 [dashboard INFO root] server: ssl=no host=:: port=8080
debug 2024-01-08T10:02:14.314+0000 7fb7de459700 0 [dashboard INFO root] Configured CherryPy, starting engine...
debug 2024-01-08T10:02:14.318+0000 7fb7dd457700 0 [devicehealth INFO root] Starting
debug 2024-01-08T10:02:14.318+0000 7fb7ff998700 0 [iostat DEBUG root] setting log level based on debug_mgr: INFO (2/5)
debug 2024-01-08T10:02:14.318+0000 7fb7ff998700 1 mgr load Constructed class from module: iostat
debug 2024-01-08T10:02:14.322+0000 7fb7ff998700 0 [nfs DEBUG root] setting log level based on debug_mgr: INFO (2/5)
debug 2024-01-08T10:02:14.322+0000 7fb7ff998700 1 mgr load Constructed class from module: nfs
debug 2024-01-08T10:02:14.322+0000 7fb7ff998700 0 [orchestrator DEBUG root] setting log level based on debug_mgr: INFO (2/5)
debug 2024-01-08T10:02:14.322+0000 7fb7ff998700 1 mgr load Constructed class from module: orchestrator
debug 2024-01-08T10:02:14.326+0000 7fb7e045d700 0 [balancer INFO root] Optimize plan auto_2024-01-08_10:02:14
debug 2024-01-08T10:02:14.326+0000 7fb7e045d700 0 [balancer INFO root] Mode upmap, max misplaced 0.050000
debug 2024-01-08T10:02:14.326+0000 7fb7e045d700 0 [balancer INFO root] Some PGs (1.000000) are unknown; try again later
debug 2024-01-08T10:02:14.326+0000 7fb7ff998700 0 [pg_autoscaler DEBUG root] setting log level based on debug_mgr: INFO (2/5)
debug 2024-01-08T10:02:14.326+0000 7fb7ff998700 1 mgr load Constructed class from module: pg_autoscaler
debug 2024-01-08T10:02:14.330+0000 7fb7ff998700 0 [progress DEBUG root] setting log level based on debug_mgr: INFO (2/5)
debug 2024-01-08T10:02:14.330+0000 7fb7ff998700 1 mgr load Constructed class from module: progress
debug 2024-01-08T10:02:14.334+0000 7fb7d944f700 0 [pg_autoscaler INFO root] _maybe_adjust
debug 2024-01-08T10:02:14.338+0000 7fb7d844d700 0 [progress INFO root] Loading...
debug 2024-01-08T10:02:14.338+0000 7fb7ff998700 0 [prometheus DEBUG root] setting log level based on debug_mgr: INFO (2/5)
debug 2024-01-08T10:02:14.346+0000 7fb7d844d700 0 [progress INFO root] Loaded [<progress.module.GhostEvent object at 0x7fb7e460bbe0>, <progress.module.GhostEvent object at 0x7fb7e460b7b8>, <progress.modu
le.GhostEvent object at 0x7fb7e460bac8>, <progress.module.GhostEvent object at 0x7fb7e460bf28>, <progress.module.GhostEvent object at 0x7fb7e460b080>, <progress.module.GhostEvent object at 0x7fb7e460b438>
, <progress.module.GhostEvent object at 0x7fb7e460b208>, <progress.module.GhostEvent object at 0x7fb7e460bf60>, <progress.module.GhostEvent object at 0x7fb7e460bcf8>, <progress.module.GhostEvent object at
0x7fb7e460bc18>, <progress.module.GhostEvent object at 0x7fb7e460b8d0>, <progress.module.GhostEvent object at 0x7fb7e460b048>, <progress.module.GhostEvent object at 0x7fb7e460b9b0>, <progress.module.Ghos
tEvent object at 0x7fb7e460ba58>, <progress.module.GhostEvent object at 0x7fb7e460b550>, <progress.module.GhostEvent object at 0x7fb7e460ba90>, <progress.module.GhostEvent object at 0x7fb7e460b940>, <prog
ress.module.GhostEvent object at 0x7fb7e460be48>, <progress.module.GhostEvent object at 0x7fb7e460b7f0>, <progress.module.GhostEvent object at 0x7fb7e460b668>, <progress.module.GhostEvent object at 0x7fb7
e460b160>, <progress.module.GhostEvent object at 0x7fb7e460bda0>, <progress.module.GhostEvent object at 0x7fb7e460b128>, <progress.module.GhostEvent object at 0x7fb7e460bba8>, <progress.module.GhostEvent
object at 0x7fb7e460b6d8>, <progress.module.GhostEvent object at 0x7fb7e460b320>, <progress.module.GhostEvent object at 0x7fb7e460bd30>, <progress.module.GhostEvent object at 0x7fb7e460bc88>, <progress.mo
dule.GhostEvent object at 0x7fb7e460b748>, <progress.module.GhostEvent object at 0x7fb7e460b5c0>, <progress.module.GhostEvent object at 0x7fb7e460bf98>, <progress.module.GhostEvent object at 0x7fb7e460b4e
0>] historic events
debug 2024-01-08T10:02:14.346+0000 7fb7d844d700 0 [progress INFO root] Loaded OSDMap, ready.
debug 2024-01-08T10:02:14.346+0000 7fb7ff998700 -1 no module 'rook'
debug 2024-01-08T10:02:14.346+0000 7fb7ff998700 -1 no module 'rook'
debug 2024-01-08T10:02:14.346+0000 7fb7ff998700 -1 mgr load Failed to construct class in 'prometheus'
debug 2024-01-08T10:02:14.358+0000 7fb81919b700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
debug 2024-01-08T10:02:14.346+0000 7fb7ff998700 -1 mgr load Traceback (most recent call last):
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1657, in _oremote
return mgr.remote(o, meth, *args, **kwargs)
File "/usr/share/ceph/mgr/mgr_module.py", line 2228, in remote
args, kwargs)
ImportError: Module not found
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/share/ceph/mgr/prometheus/module.py", line 649, in __init__
self.modify_instance_id = self.get_orch_status() and self.get_module_option(
File "/usr/share/ceph/mgr/prometheus/module.py", line 869, in get_orch_status
return self.available()[0]
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1586, in inner
completion = self._oremote(method_name, args, kwargs)
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1661, in _oremote
f_set = self.get_feature_set()
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1586, in inner
completion = self._oremote(method_name, args, kwargs)
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1657, in _oremote
return mgr.remote(o, meth, *args, **kwargs)
File "/usr/share/ceph/mgr/mgr_module.py", line 2228, in remote
args, kwargs)
ImportError: Module not found
debug 2024-01-08T10:02:14.358+0000 7fb7ff998700 -1 mgr operator() Failed to run module in active mode ('prometheus')
ceph mgr module ls
MODULE
balancer on (always on)
crash on (always on)
devicehealth on (always on)
orchestrator on (always on)
pg_autoscaler on (always on)
progress on (always on)
rbd_support on (always on)
status on (always on)
telemetry on (always on)
volumes on (always on)
dashboard on
iostat on
nfs on
prometheus on
restful on
rook on
...
The CRD is in Ready status, everything seems to work except the dasboard “performance” tabs.
What can I check to debug this issue? Thanks!
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 2
- Comments: 44 (20 by maintainers)
@ahgraber It’s a ceph problem. pr has been merged: https://github.com/ceph/ceph/pull/55149 it will be part of 18.2.2 https://github.com/ceph/ceph/milestone/19
@rspier thanks for deploying and testing the first part of the fix (the one related to the prometheus module crash). As for the new issue I’m sure that it has to do with some mismatch between how
daemon_idsare called in cephadm based deployments vs rook. I created a new imagedocker.io/rkachach/ceph:v18.2.1_patched_v1with a small change hoping it fixes this new issue.As of building an image you have basically to pick the changes from my PR https://github.com/ceph/ceph/pull/55149/files and use them to update the base image of v18.2.1. A simple docker file like:
This issue is fixed by updating the Ceph version to v18.2.2. See the Ceph upgrade guide to make this change. This will become the default in Rook v1.13.7 when released in the next few days, but no need to wait for that release before applying v18.2.2.
Hi! I’ve tested with the new image, there is no error about the prometheus module on the mgr log. The mgr serve on the metrics port, thanks!
@bhuism i have build multi arch image, a public gitlab repository is available on https://gitlab.onlineterroir.com/ceph/ceph with opreationnal CI. Multi arch image is availbe on
registry.gitlab.onlineterroir.com/ceph/ceph:v18.2.1_patched_v1I confirm the v1 fix the issue also for me (using rook without helm charts).
new image tag should be now available on
quay.io/ceph/ceph:v18.2.2Not sure if this could help or not, but on my local env I was able to enable rook and prometheus by using the following setup:
modulessection from the cluster yamlFollowing these steps the mgr seems to start without issues but I can’t guarantee that same will happen in your env. Besides, plz keep in mind that any restart on the mgr can lead to the problem again.
Thank you very much @matthewpi. Just FYI I tried several times to reproduce the issue locally but with no success.
Analyzing these logs and the ones provided originally by @pasztorl the common pattern is the order of modules loading: first prometheus is loaded then rook (we can see this in the following snippet from your active mgr logs):
I think the problem is a dependency between promethues and orchestrator (rook in this case) modules introduced recently by the changes from PR https://github.com/ceph/ceph/pull/52191. Specifically the orch status check.
In the error stracktrace we can see how the mgr fails to load prometheus (
Module not found) because rook has not yet been loaded:That also explains why I wasn’t able to reproduce in my local environment because in my setup rook is loaded always before prometheus.
I am also able to reproduce this. My config only explicitly configures the
rookmodule though, monitoring is enabled.Something interesting to note is that Prometheus works on a standby mgr, but not the active one.
Logs from active mgr: (some lines are omitted, indicated by
...)Environment:
NixOS 24.05 (Uakari)Linux 6.1.71 #1-NixOS SMP PREEMPT_DYNAMIC Fri Jan 5 14:18:41 UTC 2024 x86_64 GNU/Linuxv1.13.1quay.io/ceph/ceph:v18.2.1-20231215v1.28.5@guillaumetorresani thanks for the arm image!
@rkachach works like a charm!
@guillaumetorresani thank you for providing the multi-arch image.
Was having this issue in the discussion area. Hopefully this makes it into the rook-ceph release. Nice work!
I confirm the v1 fix the issue also for me (using rook helm charts).
Also can confirm that the
v1patch is working for meThe
v1patch fixes the issue for me. podip:9223/metrics returns metrics again!Thank you!
@xavi-clovr Unfortunately loading process is not deterministic and right now there’s a race condition between loading prometheus and the orchestrator (rook) which could lead to a crash of prometheus module in case it’s loaded before rook. So the proposed work around is not 100% sure to work.
@rkachach Looks that the workaround works in our setup as well. Thanks for that. Please keep us informed when a permanent fix is available. Thanks.
Thanks for the info!
@kpoos I have confirmed that the issue is related with the changes from the PR I post above. I have already opened a ticket on the ceph project to track the issue. https://tracker.ceph.com/issues/63992 Unfortunately at this moment there’s no work around to this problem.
This issue also affects our setup. It started to coming immediately after upgrading to 1.13 from 1.12. MGR Pod is constantly crashing since then. [rook@rook-ceph-tools-68644848b9-flk4x /]$ ceph status cluster: id: e64af4e2-a50d-408b-943a-89cc8539a3ca health: HEALTH_WARN 125 mgr modules have recently crashed
[rook@rook-ceph-tools-68644848b9-flk4x /]$ ceph crash info 2024-01-10T09:17:44.340271Z_c16a026b-2df4-4857-b64a-ea73c41a5493 { “backtrace”: [ " File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1657, in _oremote\n return mgr.remote(o, meth, *args, **kwargs)“, " File "/usr/share/ceph/mgr/mgr_module.py", line 2228, in remote\n args, kwargs)”, “ImportError: Module not found”, “\nDuring handling of the above exception, another exception occurred:\n”, “Traceback (most recent call last):”, " File "/usr/share/ceph/mgr/prometheus/module.py", line 649, in init\n self.modify_instance_id = self.get_orch_status() and self.get_module_option(“, " File "/usr/share/ceph/mgr/prometheus/module.py", line 869, in get_orch_status\n return self.available()[0]”, " File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1586, in inner\n completion = self._oremote(method_name, args, kwargs)“, " File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1661, in _oremote\n f_set = self.get_feature_set()”, " File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1586, in inner\n completion = self._oremote(method_name, args, kwargs)“, " File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1657, in _oremote\n return mgr.remote(o, meth, *args, **kwargs)”, " File "/usr/share/ceph/mgr/mgr_module.py", line 2228, in remote\n args, kwargs)", “ImportError: Module not found” ], “ceph_version”: “18.2.1”, “crash_id”: “2024-01-10T09:17:44.340271Z_c16a026b-2df4-4857-b64a-ea73c41a5493”, “entity_name”: “mgr.b”, “mgr_module”: “prometheus”, “mgr_module_caller”: “ActivePyModule::load”, “mgr_python_exception”: “ImportError”, “os_id”: “centos”, “os_name”: “CentOS Stream”, “os_version”: “8”, “os_version_id”: “8”, “process_name”: “ceph-mgr”, “stack_sig”: “6f4033c8739c625d3935380854c7945f8bd28267f6a4d03fc2017bc8815c257c”, “timestamp”: “2024-01-10T09:17:44.340271Z”, “utsname_hostname”: “rook-ceph-mgr-b-6d58d88fd9-j5bw6”, “utsname_machine”: “x86_64”, “utsname_release”: “5.15.0-53-generic”, “utsname_sysname”: “Linux”, “utsname_version”: “#59~20.04.1-Ubuntu SMP Thu Oct 20 15:10:22 UTC 2022” } [rook@rook-ceph-tools-68644848b9-flk4x /]$
Here are the logs from startup. mgr-active.log mgr-standby.log
rook module seems working because I can see the cluster -> Physical Disks
ceph mgr module enable rook && ceph orch set backend rook && ceph orch status
output:
module ‘rook’ is already enabled Backend: rook Available: Yes
In the manager logs there was nothing about that, so I restarted the mgr pod then the startup log is the same as above.