rook: Data distribution on OSDs is not effective

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior:

I’ve created an 8TB PVC There is absolutely nothing else on the SSDs, apart from the single 8TB PVC. After copying about 3.4TB of data onto the PVC, I get the following error message :

image

But I see that only the 4TB SSDs have been used, not the 2TB SSDs. image

Expected behavior:

I was expecting the PVC to be spread over all the disks, not just two.

How to reproduce it (minimal and precise):

File(s) to submit: CephCluster

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: cluster-global
  namespace: rook-cluster-global # namespace:cluster
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v18.2.4
    allowUnsupported: false
  dataDirHostPath: /var/lib/rook/global
  skipUpgradeChecks: false
  continueUpgradeAfterChecksEvenIfNotHealthy: false
  waitTimeoutForHealthyOSDInMinutes: 10
  upgradeOSDRequiresHealthyPGs: false
  mon:
    count: 3
    allowMultiplePerNode: false
  mgr:
    count: 2
    allowMultiplePerNode: false
    modules:
      - name: rook
        enabled: true
  dashboard:
    enabled: true
    ssl: false
  monitoring:
    enabled: false
    metricsDisabled: false
  network:
    connections:
volumes.
      encryption:
        enabled: false
      compression:
        enabled: false
      requireMsgr2: false
  crashCollector:
    disable: false
  logCollector:
    enabled: true
    periodicity: daily # one of: hourly, daily, weekly, monthly
    maxLogSize: 500M # SUFFIX may be 'M' or 'G'. Must be at least 1M.
  cleanupPolicy:
    confirmation: ""
    sanitizeDisks:
      method: quick
      dataSource: zero
      iteration: 1
    allowUninstallWithVolumes: false
  placement:
    all:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: rook-cluster-global-storage-node
              operator: In
              values:
              - "true"
      podAffinity:
      podAntiAffinity:
      topologySpreadConstraints:
      tolerations:
      - key: rook-cluster-global-storage-node
        operator: Exists
  labels:
  resources:
  removeOSDsIfOutAndSafeToRemove: false
  priorityClassNames:
    mon: system-node-critical
    osd: system-node-critical
    mgr: system-cluster-critical
  storage:
    useAllNodes: false
    useAllDevices: false
    devices:
    nodes:
      - name: "kube-worker-1"
        devices:
          - name: /dev/disk/by-id/..........
          - name: /dev/disk/by-id/..........
          - .......
      - name: "kube-worker-2"
        devices:
          - name: /dev/disk/by-id/..........
          - name: /dev/disk/by-id/..........
          - .......
      - name: "kube-worker-3"
          - name: /dev/disk/by-id/..........
          - name: /dev/disk/by-id/..........
          - .......
    config:
    allowDeviceClassUpdate: false
    allowOsdCrushWeightUpdate: false
    onlyApplyOSDPlacement: false
  disruptionManagement:
    managePodBudgets: true
    osdMaintenanceTimeout: 30
    pgHealthCheckTimeout: 0
  csi:
    readAffinity:
      enabled: true
    cephfs:
  healthCheck:
    daemonHealth:
      mon:
        disabled: false
        interval: 45s
      osd:
        disabled: false
        interval: 60s
      status:
        disabled: false
        interval: 60s
    livenessProbe:
      mon:
        disabled: false
      mgr:
        disabled: false
      osd:
        disabled: false
    startupProbe:
      mon:
        disabled: false
      mgr:
        disabled: false
      osd:
        disabled: false

CephBlockPool

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: standard-ssd
  namespace: rook-cluster-global
spec:
  failureDomain: host
  enableRBDStats: true
  replicated:
    size: 2
  deviceClass: ssd

StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard-r2-rbd
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  clusterID: rook-cluster-global
  pool: standard-ssd
  imageFormat: "2"
  imageFeatures: "layering,exclusive-lock,object-map,fast-diff,deep-flatten"
  csi.storage.k8s.io/fstype: ext4
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-cluster-global
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-cluster-global
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-cluster-global
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - discard

Logs to submit:

Cluster Status to submit:

  • Output of kubectl commands, if necessary

    To get the health of the cluster, use kubectl rook-ceph health image

    To get the status of the cluster, use kubectl rook-ceph ceph status image

    For more details, see the Rook kubectl Plugin

Environment:

  • OS (e.g. from /etc/os-release): image

  • Kernel (e.g. uname -a): image

  • Cloud provider or hardware configuration:

    • I have an Bare Metal K8S cluster with 3 masters and 3 workers.
    • All nodes (masters and workers) have Talos as OS
    • Each worker has the following configuration:
      • 3 x 2 TB NVME
      • 1 x 2 TB SDD
      • 1 x 4 TB SDD
      • 1 x 12TB HDD
  • Rook version (use rook version inside of a Rook Pod): image

  • Storage backend version (e.g. for ceph do ceph -v): image

  • Kubernetes version (use kubectl version): image

  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): Bare metal

  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): image

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (7 by maintainers)

Most upvoted comments

You don’t need to set a different crush root. Ceph automatically generates those for the purposes of using device classes for the pools. If you specify a device class for a pool, ceph will automatically use the correct crush root, and no need to define the crushRoot property separately on the pool.

That error about overlapping roots is caused when device classes are set on some pools, but not all pools. You’ll need to ensure that the deviceClass property is set on all pools, including the .mgr pool. After all pools have the deviceClass set, and rook reconciles/updates them, it should resolve the errors of overlapping roots, and then the PG autoscaler can work properly.

Did you set enableCrushUpdates: true in the .mgr CephBlockPool CR? This is a new property, set to false by default.

You can create the .mgr CephBlockPool at any point, and the device class or other settings (e.g. replicated.size) would be then configured by Rook. With a new cluster, you can go ahead and create it at cluster creation.

Glad to hear you got it working. Alternatively, sounds like you could have set the following property on the .mgr pool and rook would have fixed the crush rule for you:

deviceClass: nvme

I have no idea why so many crush root tree, it seems to be related to device class. cc @travisn you’d better to custom crush roots according to device class, then create pool with specific crush root, make sure not overlap. https://docs.ceph.com/en/quincy/rados/operations/crush-map/#modifying-the-crush-map https://github.com/rook/rook/blob/afe72ffd8c80eabe13118173d486d8e7b8999529/deploy/examples/pool.yaml#L30