rook: Rook Dashboard hangs/times out

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior: When accessing the Rook Dashboard from the Ingress (or even directly to the active mgr pod), the HTTP request times out. Sometimes it serves the content once but then it continuously times out with the ingress reporting 504.

Expected behavior: Rook Dashboard is properly served

How to reproduce it (minimal and precise):

Deploy an operator that then deploys the CephCluster

File(s) to submit:

  • Cluster CR (custom resource), typically called cluster.yaml, if necessary
  values:
    cephBlockPools:
      - name: ceph-blockpool
        spec:
          failureDomain: host
          replicated:
            size: 3
          enableRBDStats: true
        storageClass:
          enabled: true
          name: ceph-block
          isDefault: false
          reclaimPolicy: Delete
          allowVolumeExpansion: true
          volumeBindingMode: "Immediate"
          mountOptions: []
          allowedTopologies: []
          parameters:
            imageFormat: "2"
            imageFeatures: layering
            csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
            csi.storage.k8s.io/provisioner-secret-namespace: "{{ .Release.Namespace }}"
            csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
            csi.storage.k8s.io/controller-expand-secret-namespace: "{{ .Release.Namespace }}"
            csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
            csi.storage.k8s.io/node-stage-secret-namespace: "{{ .Release.Namespace }}"
            csi.storage.k8s.io/fstype: ext4
    cephClusterSpec:
      dataDirHostPath: /var/lib/rook
      dashboard:
        enabled: true
        ssl: false
      monitoring:
        enabled: true
      disruptionManagement:
        managePodBudgets: true
      mon:
        count: 3
        allowMultiplePerNode: false
        volumeClaimTemplate:
          spec:
            storageClassName: standard-rwo
            resources:
              requests:
                storage: 10Gi
      mgr:
        count: 3
      storage:
        useAllNodes: true
        useAllDevices: false
        storageClassDeviceSets:
          - name: rook
            count: 3
            placement:
              podAntiAffinity:
                requiredDuringSchedulingIgnoredDuringExecution:
                  - labelSelector:
                      matchExpressions:
                        - key: app
                          operator: In
                          values:
                            - rook-ceph-osd
                            - rook-ceph-osd-prepare
                    topologyKey: topology.kubernetes.io/zone
            portable: true
            tuneDeviceClass: false
            volumeClaimTemplates:
              - metadata:
                  name: data
                spec:
                  resources:
                    requests:
                      storage: 200Gi
                  storageClassName: standard-rwo
                  volumeMode: Block
                  accessModes:
                    - ReadWriteOnce
      placement:
        mon:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - rook-ceph-mon
                topologyKey: topology.kubernetes.io/zone
        storage:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - rook-ceph-osd
                        - rook-ceph-osd-prepare
                topologyKey: topology.kubernetes.io/zone
      resources:
        crashcollector:
          limits:
            cpu: 500m
            memory: 60Mi
          requests:
            cpu: 10m
            memory: 60Mi
        mgr:
          limits:
            cpu: 500m
            memory: 1Gi
          requests:
            cpu: 50m
            memory: 512Mi
        mon:
          limits:
            cpu: 500m
            memory: 2Gi
          requests:
            cpu: 100m
            memory: 1Gi
        osd:
          limits:
            cpu: 500m
            memory: 6Gi
          requests:
            cpu: 100m
            memory: 2Gi
    cephFileSystems: {}
    cephObjectStores: {}
    ingress:
      dashboard:
        annotations:
          cert-manager.io/cluster-issuer: dns-issuer
          kubernetes.io/tls-acme: "true"
          nginx.ingress.kubernetes.io/backend-protocol: HTTP
          nginx.ingress.kubernetes.io/ssl-passthrough: "false"
          nginx.ingress.kubernetes.io/ssl-redirect: "true"
        ingressClassName: nginx-internal
    monitoring:
      enabled: true
      createPrometheusRules: true
      rulesNamespaceOverride: logging-monitoring
    toolbox:
      enabled: true
      containerSecurityContext:
        runAsNonRoot: true
        runAsUser: 2016
        runAsGroup: 2016
        capabilities:
          drop: ["ALL"]

Logs to submit: No logs related to accessing the dashboard from the mgr pods

  • Operator’s logs, if necessary
  • Crashing pod(s) logs, if necessary No crashing pods

Cluster Status to submit:

[root@rook-ceph-tools-6595b6bb4-gw68d /]# ceph status
  cluster:
    id:     113e9a1d-f5bd-4c95-b3b6-ce79136c679b
    health: HEALTH_WARN
            2 nearfull osd(s)
            2 pool(s) nearfull
            4 daemons have recently crashed

  services:
    mon: 3 daemons, quorum g,j,k (age 3h)
    mgr: a(active, since 3h), standbys: c, b
    osd: 3 osds: 3 up (since 26h), 3 in (since 3M)

  data:
    pools:   2 pools, 33 pgs
    objects: 20.13k objects, 78 GiB
    usage:   436 GiB used, 164 GiB / 600 GiB avail
    pgs:     33 active+clean

  io:
    client:   2.0 MiB/s wr, 0 op/s rd, 164 op/s wr
  • Output of kubectl commands, if necessary

Environment:

  • OS (e.g. from /etc/os-release): Ubuntu 22.04.3 LTS
  • Kernel (e.g. uname -a): 5.15.0-1039-gke
  • Cloud provider or hardware configuration: GCP
  • Rook version (use rook version inside of a Rook Pod):
  • Storage backend version (e.g. for ceph do ceph -v): ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
  • Kubernetes version (use kubectl version): v1.26.10-gke.1101000
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): GKE
  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): HEALTH_WARN 2 nearfull osd(s); 2 pool(s) nearfull; 4 daemons have recently crashed (crashed pods probably correspond with spot instances terminating)

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 17 (8 by maintainers)

Most upvoted comments

Thanks @rkachach, closing this as the original issue was resolved.

@Nickmman Thanks for the feedback, I’m glad that your setup is working after the upgrade. The patch has not been released yet and it would take some time as it depends on the Ceph next reef release (v18.2.2).

@rkachach I have fixed the osd/pool usage last night by recreating each of the osds, since it was not freeing up space from deleted block images. I hope that I don’t have to do this frequently but I haven’t been able to come across any documentation about cleaning up the osds, aside from recreating them.

Regarding the original issue, I have also increased the CPU limit to double (or more than double) what was assigned to the mgr pods. This seems to have resolved the issue, though it is still unclear why it needs so much processing power to present a dashboard. It is already reaching the CPU limit I have assigned.

Should this happen again, I will try to obtain the mgr logs (nothing jumps out aside from constant DBG messages about the PGs). I did run top however, and the process that is consuming near 100% (and sometimes 100%) is dashboard image