rook: Rook Dashboard hangs/times out
Is this a bug report or feature request?
- Bug Report
Deviation from expected behavior: When accessing the Rook Dashboard from the Ingress (or even directly to the active mgr pod), the HTTP request times out. Sometimes it serves the content once but then it continuously times out with the ingress reporting 504.
Expected behavior: Rook Dashboard is properly served
How to reproduce it (minimal and precise):
Deploy an operator that then deploys the CephCluster
File(s) to submit:
- Cluster CR (custom resource), typically called
cluster.yaml, if necessary
values:
cephBlockPools:
- name: ceph-blockpool
spec:
failureDomain: host
replicated:
size: 3
enableRBDStats: true
storageClass:
enabled: true
name: ceph-block
isDefault: false
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: "Immediate"
mountOptions: []
allowedTopologies: []
parameters:
imageFormat: "2"
imageFeatures: layering
csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: "{{ .Release.Namespace }}"
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: "{{ .Release.Namespace }}"
csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
csi.storage.k8s.io/node-stage-secret-namespace: "{{ .Release.Namespace }}"
csi.storage.k8s.io/fstype: ext4
cephClusterSpec:
dataDirHostPath: /var/lib/rook
dashboard:
enabled: true
ssl: false
monitoring:
enabled: true
disruptionManagement:
managePodBudgets: true
mon:
count: 3
allowMultiplePerNode: false
volumeClaimTemplate:
spec:
storageClassName: standard-rwo
resources:
requests:
storage: 10Gi
mgr:
count: 3
storage:
useAllNodes: true
useAllDevices: false
storageClassDeviceSets:
- name: rook
count: 3
placement:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- rook-ceph-osd
- rook-ceph-osd-prepare
topologyKey: topology.kubernetes.io/zone
portable: true
tuneDeviceClass: false
volumeClaimTemplates:
- metadata:
name: data
spec:
resources:
requests:
storage: 200Gi
storageClassName: standard-rwo
volumeMode: Block
accessModes:
- ReadWriteOnce
placement:
mon:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- rook-ceph-mon
topologyKey: topology.kubernetes.io/zone
storage:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- rook-ceph-osd
- rook-ceph-osd-prepare
topologyKey: topology.kubernetes.io/zone
resources:
crashcollector:
limits:
cpu: 500m
memory: 60Mi
requests:
cpu: 10m
memory: 60Mi
mgr:
limits:
cpu: 500m
memory: 1Gi
requests:
cpu: 50m
memory: 512Mi
mon:
limits:
cpu: 500m
memory: 2Gi
requests:
cpu: 100m
memory: 1Gi
osd:
limits:
cpu: 500m
memory: 6Gi
requests:
cpu: 100m
memory: 2Gi
cephFileSystems: {}
cephObjectStores: {}
ingress:
dashboard:
annotations:
cert-manager.io/cluster-issuer: dns-issuer
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/backend-protocol: HTTP
nginx.ingress.kubernetes.io/ssl-passthrough: "false"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
ingressClassName: nginx-internal
monitoring:
enabled: true
createPrometheusRules: true
rulesNamespaceOverride: logging-monitoring
toolbox:
enabled: true
containerSecurityContext:
runAsNonRoot: true
runAsUser: 2016
runAsGroup: 2016
capabilities:
drop: ["ALL"]
Logs to submit: No logs related to accessing the dashboard from the mgr pods
- Operator’s logs, if necessary
- Crashing pod(s) logs, if necessary No crashing pods
Cluster Status to submit:
[root@rook-ceph-tools-6595b6bb4-gw68d /]# ceph status
cluster:
id: 113e9a1d-f5bd-4c95-b3b6-ce79136c679b
health: HEALTH_WARN
2 nearfull osd(s)
2 pool(s) nearfull
4 daemons have recently crashed
services:
mon: 3 daemons, quorum g,j,k (age 3h)
mgr: a(active, since 3h), standbys: c, b
osd: 3 osds: 3 up (since 26h), 3 in (since 3M)
data:
pools: 2 pools, 33 pgs
objects: 20.13k objects, 78 GiB
usage: 436 GiB used, 164 GiB / 600 GiB avail
pgs: 33 active+clean
io:
client: 2.0 MiB/s wr, 0 op/s rd, 164 op/s wr
- Output of kubectl commands, if necessary
Environment:
- OS (e.g. from /etc/os-release): Ubuntu 22.04.3 LTS
- Kernel (e.g.
uname -a): 5.15.0-1039-gke - Cloud provider or hardware configuration: GCP
- Rook version (use
rook versioninside of a Rook Pod): - Storage backend version (e.g. for ceph do
ceph -v): ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) - Kubernetes version (use
kubectl version): v1.26.10-gke.1101000 - Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): GKE
- Storage backend status (e.g. for Ceph use
ceph healthin the Rook Ceph toolbox):HEALTH_WARN 2 nearfull osd(s); 2 pool(s) nearfull; 4 daemons have recently crashed(crashed pods probably correspond with spot instances terminating)
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 17 (8 by maintainers)
Thanks @rkachach, closing this as the original issue was resolved.
@Nickmman Thanks for the feedback, I’m glad that your setup is working after the upgrade. The patch has not been released yet and it would take some time as it depends on the Ceph next reef release (v18.2.2).
@rkachach I have fixed the osd/pool usage last night by recreating each of the osds, since it was not freeing up space from deleted block images. I hope that I don’t have to do this frequently but I haven’t been able to come across any documentation about cleaning up the osds, aside from recreating them.
Regarding the original issue, I have also increased the CPU limit to double (or more than double) what was assigned to the mgr pods. This seems to have resolved the issue, though it is still unclear why it needs so much processing power to present a dashboard. It is already reaching the CPU limit I have assigned.
Should this happen again, I will try to obtain the mgr logs (nothing jumps out aside from constant DBG messages about the PGs). I did run top however, and the process that is consuming near 100% (and sometimes 100%) is
dashboard