rook: Data distribution on OSDs is not effective
Is this a bug report or feature request?
- Bug Report
Deviation from expected behavior:
I’ve created an 8TB PVC There is absolutely nothing else on the SSDs, apart from the single 8TB PVC. After copying about 3.4TB of data onto the PVC, I get the following error message :
But I see that only the 4TB SSDs have been used, not the 2TB SSDs.
Expected behavior:
I was expecting the PVC to be spread over all the disks, not just two.
How to reproduce it (minimal and precise):
File(s) to submit: CephCluster
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
name: cluster-global
namespace: rook-cluster-global # namespace:cluster
spec:
cephVersion:
image: quay.io/ceph/ceph:v18.2.4
allowUnsupported: false
dataDirHostPath: /var/lib/rook/global
skipUpgradeChecks: false
continueUpgradeAfterChecksEvenIfNotHealthy: false
waitTimeoutForHealthyOSDInMinutes: 10
upgradeOSDRequiresHealthyPGs: false
mon:
count: 3
allowMultiplePerNode: false
mgr:
count: 2
allowMultiplePerNode: false
modules:
- name: rook
enabled: true
dashboard:
enabled: true
ssl: false
monitoring:
enabled: false
metricsDisabled: false
network:
connections:
volumes.
encryption:
enabled: false
compression:
enabled: false
requireMsgr2: false
crashCollector:
disable: false
logCollector:
enabled: true
periodicity: daily # one of: hourly, daily, weekly, monthly
maxLogSize: 500M # SUFFIX may be 'M' or 'G'. Must be at least 1M.
cleanupPolicy:
confirmation: ""
sanitizeDisks:
method: quick
dataSource: zero
iteration: 1
allowUninstallWithVolumes: false
placement:
all:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: rook-cluster-global-storage-node
operator: In
values:
- "true"
podAffinity:
podAntiAffinity:
topologySpreadConstraints:
tolerations:
- key: rook-cluster-global-storage-node
operator: Exists
labels:
resources:
removeOSDsIfOutAndSafeToRemove: false
priorityClassNames:
mon: system-node-critical
osd: system-node-critical
mgr: system-cluster-critical
storage:
useAllNodes: false
useAllDevices: false
devices:
nodes:
- name: "kube-worker-1"
devices:
- name: /dev/disk/by-id/..........
- name: /dev/disk/by-id/..........
- .......
- name: "kube-worker-2"
devices:
- name: /dev/disk/by-id/..........
- name: /dev/disk/by-id/..........
- .......
- name: "kube-worker-3"
- name: /dev/disk/by-id/..........
- name: /dev/disk/by-id/..........
- .......
config:
allowDeviceClassUpdate: false
allowOsdCrushWeightUpdate: false
onlyApplyOSDPlacement: false
disruptionManagement:
managePodBudgets: true
osdMaintenanceTimeout: 30
pgHealthCheckTimeout: 0
csi:
readAffinity:
enabled: true
cephfs:
healthCheck:
daemonHealth:
mon:
disabled: false
interval: 45s
osd:
disabled: false
interval: 60s
status:
disabled: false
interval: 60s
livenessProbe:
mon:
disabled: false
mgr:
disabled: false
osd:
disabled: false
startupProbe:
mon:
disabled: false
mgr:
disabled: false
osd:
disabled: false
CephBlockPool
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
name: standard-ssd
namespace: rook-cluster-global
spec:
failureDomain: host
enableRBDStats: true
replicated:
size: 2
deviceClass: ssd
StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: standard-r2-rbd
annotations:
storageclass.kubernetes.io/is-default-class: "false"
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
clusterID: rook-cluster-global
pool: standard-ssd
imageFormat: "2"
imageFeatures: "layering,exclusive-lock,object-map,fast-diff,deep-flatten"
csi.storage.k8s.io/fstype: ext4
csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: rook-cluster-global
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: rook-cluster-global
csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
csi.storage.k8s.io/node-stage-secret-namespace: rook-cluster-global
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
- discard
Logs to submit:
-
Operator’s logs, if necessary
-
Crashing pod(s) logs, if necessary
-
Manager pod(s) logs
To get logs, use
kubectl -n <namespace> logs <pod name>When pasting logs, always surround them with backticks or use theinsert codebutton from the Github UI. Read GitHub documentation if you need help.
Cluster Status to submit:
-
Output of kubectl commands, if necessary
To get the health of the cluster, use
kubectl rook-ceph healthTo get the status of the cluster, use
kubectl rook-ceph ceph statusFor more details, see the Rook kubectl Plugin
Environment:
-
OS (e.g. from /etc/os-release):
-
Kernel (e.g.
uname -a): -
Cloud provider or hardware configuration:
- I have an Bare Metal K8S cluster with 3 masters and 3 workers.
- All nodes (masters and workers) have Talos as OS
- Each worker has the following configuration:
- 3 x 2 TB NVME
- 1 x 2 TB SDD
- 1 x 4 TB SDD
- 1 x 12TB HDD
-
Rook version (use
rook versioninside of a Rook Pod): -
Storage backend version (e.g. for ceph do
ceph -v): -
Kubernetes version (use
kubectl version): -
Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):
Bare metal -
Storage backend status (e.g. for Ceph use
ceph healthin the Rook Ceph toolbox):
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 19 (7 by maintainers)
You don’t need to set a different crush root. Ceph automatically generates those for the purposes of using device classes for the pools. If you specify a device class for a pool, ceph will automatically use the correct crush root, and no need to define the crushRoot property separately on the pool.
That error about overlapping roots is caused when device classes are set on some pools, but not all pools. You’ll need to ensure that the deviceClass property is set on all pools, including the .mgr pool. After all pools have the deviceClass set, and rook reconciles/updates them, it should resolve the errors of overlapping roots, and then the PG autoscaler can work properly.
Did you set
enableCrushUpdates: truein the.mgrCephBlockPool CR? This is a new property, set to false by default.You can create the
.mgrCephBlockPool at any point, and the device class or other settings (e.g.replicated.size) would be then configured by Rook. With a new cluster, you can go ahead and create it at cluster creation.Glad to hear you got it working. Alternatively, sounds like you could have set the following property on the .mgr pool and rook would have fixed the crush rule for you:
I have no idea why so many crush root tree, it seems to be related to device class. cc @travisn you’d better to custom crush roots according to device class, then create pool with specific crush root, make sure not overlap. https://docs.ceph.com/en/quincy/rados/operations/crush-map/#modifying-the-crush-map https://github.com/rook/rook/blob/afe72ffd8c80eabe13118173d486d8e7b8999529/deploy/examples/pool.yaml#L30