rook: ceph mgr dashboard service response very slow and offen fail.
I install rook of the version v1.12.3 in k8s in the version 1.24.16 .
I found the mgr service occupies a high CPU resources during operation. And the dashboard service response very slowly and offen faild like this.
I try to increase mgr’s cpu and memory limits .
resources:
limits:
cpu: "4"
memory: 6Gi
requests:
cpu: "2"
memory: 1Gi
But the effect for the response speed was not obvious .
Of course, in some cases, the dashboard service work normally with a slow response speed.
Expected behavior: I expect the mgr dashboard service wouldn’t fail to response,and rise up the response speed.
Environment:
- OS (e.g. from /etc/os-release):
[root@sc-neutral-2 ~]# cat /etc/os-release
NAME="Rocky Linux"
VERSION="9.2 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.2"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.2 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.2"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.2"
- Kernel (e.g.
uname -a):
[root@sc-neutral-2 ~]# uname -a
Linux sc-neutral-2 5.14.0-284.11.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Tue May 9 17:09:15 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
- Cloud provider or hardware configuration:
- Rook version (use
rook versioninside of a Rook Pod): 1.12.3 - Storage backend version (e.g. for ceph do
ceph -v):
bash-4.4$ ceph -v
ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
- Kubernetes version (use
kubectl version):
[root@sc-neutral-2 ~]# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.16", GitCommit:"51e33fadff13065ae5518db94e84598293965939", GitTreeState:"clean", BuildDate:"2023-07-19T12:26:21Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.16", GitCommit:"51e33fadff13065ae5518db94e84598293965939", GitTreeState:"clean", BuildDate:"2023-07-19T12:19:24Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
- Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): kubernetes
- Storage backend status (e.g. for Ceph use
ceph healthin the Rook Ceph toolbox):
bash-4.4$ ceph health detail
HEALTH_OK
bash-4.4$ ceph -s
cluster:
id: aa0e7bed-d3e3-49axxxxxxxxxxxxxxxxxxxxxxxxxxxx
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,c,d (age 4d)
mgr: b(active, since 8h), standbys: a
mds: 1/1 daemons up, 1 hot standby
osd: 36 osds: 36 up (since 74m), 36 in (since 3d)
rgw: 1 daemon active (1 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 12 pools, 265 pgs
objects: 985.44k objects, 2.6 TiB
usage: 7.8 TiB used, 123 TiB / 131 TiB avail
pgs: 265 active+clean
io:
client: 852 B/s rd, 4.0 KiB/s wr, 1 op/s rd, 0 op/s wr
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 20 (10 by maintainers)
@travisn, sorry for the late response, I didn’t see your question. As I understood from the Dashboard team, this fix in fact comes with el9 (that PR wasn’t even necessary). That’s said, any future minor release based on Reef should include the fix automatically. To check if a ceph version has the fix or not it’s as simple as checking the version of cheroot, it should be > 10.0.1:
I’m glad that the community can resolve this issue ,and waiting for the further solution. There are another good thing:the cause that the mgr service occupies a high CPU resources during operation seems to have been found . I solved the problem by setting the daemon of containerd 's param for LimitNOFILE=1048576. This is due to two issues from the rook community : ISSUE 11253 and ISSUE 10110 I change the param like this:
Then I restart the daemon of continaerd for all the node of k8s:
I restart the mgr pod and the osd pod .Comparing the CPU usage before and after the modification,It is obviously for these processes .

@rkachach hi,very happy about your answer. I changed the https of the dashboard to http in the your way. I haven’t find a surge in the number of socket connections inside the dashboard pod for almost one hour. The number of socket connections remained stable. And there is no 504 error httpcode in the front end. Then, the description above can basically verify your solution that the problem occurs when SSL is turned on.