rook: ceph mgr dashboard service response very slow and offen fail.

I install rook of the version v1.12.3 in k8s in the version 1.24.16 . I found the mgr service occupies a high CPU resources during operation. And the dashboard service response very slowly and offen faild like this. 截图 2023-09-11 11-02-27 截图 2023-09-11 10-48-00 I try to increase mgr’s cpu and memory limits .

        resources:
          limits:
            cpu: "4"
            memory: 6Gi
          requests:
            cpu: "2"
            memory: 1Gi

But the effect for the response speed was not obvious . Of course, in some cases, the dashboard service work normally with a slow response speed. 截图 2023-09-11 10-50-39

Expected behavior: I expect the mgr dashboard service wouldn’t fail to response,and rise up the response speed.

Environment:

  • OS (e.g. from /etc/os-release):
[root@sc-neutral-2 ~]# cat  /etc/os-release
NAME="Rocky Linux"
VERSION="9.2 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.2"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.2 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.2"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.2"
  • Kernel (e.g. uname -a):
[root@sc-neutral-2 ~]# uname -a
Linux sc-neutral-2 5.14.0-284.11.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Tue May 9 17:09:15 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
  • Cloud provider or hardware configuration:
  • Rook version (use rook version inside of a Rook Pod): 1.12.3
  • Storage backend version (e.g. for ceph do ceph -v):
bash-4.4$ ceph -v
ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
  • Kubernetes version (use kubectl version):
[root@sc-neutral-2 ~]# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.16", GitCommit:"51e33fadff13065ae5518db94e84598293965939", GitTreeState:"clean", BuildDate:"2023-07-19T12:26:21Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.16", GitCommit:"51e33fadff13065ae5518db94e84598293965939", GitTreeState:"clean", BuildDate:"2023-07-19T12:19:24Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): kubernetes
  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):
bash-4.4$ ceph health detail
HEALTH_OK
bash-4.4$ ceph -s
  cluster:
    id:     aa0e7bed-d3e3-49axxxxxxxxxxxxxxxxxxxxxxxxxxxx
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum a,c,d (age 4d)
    mgr: b(active, since 8h), standbys: a
    mds: 1/1 daemons up, 1 hot standby
    osd: 36 osds: 36 up (since 74m), 36 in (since 3d)
    rgw: 1 daemon active (1 hosts, 1 zones)
 
  data:
    volumes: 1/1 healthy
    pools:   12 pools, 265 pgs
    objects: 985.44k objects, 2.6 TiB
    usage:   7.8 TiB used, 123 TiB / 131 TiB avail
    pgs:     265 active+clean
 
  io:
    client:   852 B/s rd, 4.0 KiB/s wr, 1 op/s rd, 0 op/s wr

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 20 (10 by maintainers)

Most upvoted comments

@rkachach Great, thanks for the fix! I see there is a backport for Squid. Any plans to backport to Reef also, or just Squid?

@travisn, sorry for the late response, I didn’t see your question. As I understood from the Dashboard team, this fix in fact comes with el9 (that PR wasn’t even necessary). That’s said, any future minor release based on Reef should include the fix automatically. To check if a ceph version has the fix or not it’s as simple as checking the version of cheroot, it should be > 10.0.1:

[root@4aa747c19248 /]# python3
Python 3.9.20 (main, Sep  9 2024, 00:00:00) 
[GCC 11.5.0 20240719 (Red Hat 11.5.0-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cheroot
>>> print(cheroot.__version__)
10.0.1

@cyxinda reopening as dashboard team is trying to address the ssl issue by this PR ceph/ceph#55108

I’m glad that the community can resolve this issue ,and waiting for the further solution. There are another good thing:the cause that the mgr service occupies a high CPU resources during operation seems to have been found . I solved the problem by setting the daemon of containerd 's param for LimitNOFILE=1048576. This is due to two issues from the rook community : ISSUE 11253 and ISSUE 10110 I change the param like this:

[root@sc-midware-2 ~]# vim /usr/local/lib/systemd/system/containerd.service;
# Copyright The containerd Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
#uncomment to enable the experimental sbservice (sandboxed) version of containerd/cri integration
#Environment="ENABLE_CRI_SANDBOXES=sandboxed"
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
#LimitNOFILE=infinity
LimitNOFILE=1048576
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

Then I restart the daemon of continaerd for all the node of k8s:

[root@sc-midware-2 ~]# systemctl daemon-reload && systemctl restart containerd && sleep 2s && systemctl status containerd
● containerd.service - containerd container runtime
     Loaded: loaded (/usr/local/lib/systemd/system/containerd.service; enabled; preset: disabled)
     Active: active (running) since Tue 2024-01-23 02:45:15 EST; 2s ago
       Docs: https://containerd.io
    Process: 2664780 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 2664781 (containerd)
      Tasks: 893
     Memory: 4.1G
        CPU: 2.751s
     CGroup: /system.slice/containerd.service

I restart the mgr pod and the osd pod .Comparing the CPU usage before and after the modification,It is obviously for these processes . 截图 2024-01-23 14-56-53 截图 2024-01-23 15-38-00

@rkachach hi,very happy about your answer. I changed the https of the dashboard to http in the your way. I haven’t find a surge in the number of socket connections inside the dashboard pod for almost one hour. The number of socket connections remained stable. And there is no 504 error httpcode in the front end. Then, the description above can basically verify your solution that the problem occurs when SSL is turned on. 截图 2023-11-14 18-57-28