rook: Disk ignored because of id_bus

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior:

I added 3 hard drives in one on each worker. On two workers, the operation went smoothly and the hard disks were successfully added to the Ceph.

On the third worker my hard disk is excluded from rook-ceph-osd-prepare processing. The error displayed is as follows.

cephosd: skipping device "sdc": ["id_bus"].

I don’t understand why I’m getting this error message.

I did the same thing on my other 2 Workers and it worked without a hitch.

image

Expected behavior:

Normally, the disk should be added in Ceph on Woker n°3.

How to reproduce it (minimal and precise):

File(s) to submit: CephCluster

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: cluster-global
  namespace: rook-cluster-global # namespace:cluster
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v18.2.4
    allowUnsupported: false
  dataDirHostPath: /var/lib/rook/global
  skipUpgradeChecks: false
  continueUpgradeAfterChecksEvenIfNotHealthy: false
  waitTimeoutForHealthyOSDInMinutes: 10
  upgradeOSDRequiresHealthyPGs: false
  mon:
    count: 3
    allowMultiplePerNode: false
  mgr:
    count: 2
    allowMultiplePerNode: false
    modules:
      - name: rook
        enabled: true
  dashboard:
    enabled: true
    ssl: false
  monitoring:
    enabled: false
    metricsDisabled: false
  network:
    connections:
volumes.
      encryption:
        enabled: false
      compression:
        enabled: false
      requireMsgr2: false
  crashCollector:
    disable: false
  logCollector:
    enabled: true
    periodicity: daily # one of: hourly, daily, weekly, monthly
    maxLogSize: 500M # SUFFIX may be 'M' or 'G'. Must be at least 1M.
  cleanupPolicy:
    confirmation: ""
    sanitizeDisks:
      method: quick
      dataSource: zero
      iteration: 1
    allowUninstallWithVolumes: false
  placement:
    all:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: rook-cluster-global-storage-node
              operator: In
              values:
              - "true"
      podAffinity:
      podAntiAffinity:
      topologySpreadConstraints:
      tolerations:
      - key: rook-cluster-global-storage-node
        operator: Exists
  labels:
  resources:
  removeOSDsIfOutAndSafeToRemove: false
  priorityClassNames:
    mon: system-node-critical
    osd: system-node-critical
    mgr: system-cluster-critical
  storage:
    useAllNodes: false
    useAllDevices: false
    devices:
    nodes:
      - name: "kube-worker-1"
        devices:
          - name: /dev/disk/by-id/..........
          - name: /dev/disk/by-id/..........
          - .......
      - name: "kube-worker-2"
        devices:
          - name: /dev/disk/by-id/..........
          - name: /dev/disk/by-id/..........
          - .......
      - name: "kube-worker-3"
          - name: /dev/disk/by-id/..........
          - name: /dev/disk/by-id/..........
          - .......
    config:
    allowDeviceClassUpdate: false
    allowOsdCrushWeightUpdate: false
    onlyApplyOSDPlacement: false
  disruptionManagement:
    managePodBudgets: true
    osdMaintenanceTimeout: 30
    pgHealthCheckTimeout: 0
  csi:
    readAffinity:
      enabled: true
    cephfs:
  healthCheck:
    daemonHealth:
      mon:
        disabled: false
        interval: 45s
      osd:
        disabled: false
        interval: 60s
      status:
        disabled: false
        interval: 60s
    livenessProbe:
      mon:
        disabled: false
      mgr:
        disabled: false
      osd:
        disabled: false
    startupProbe:
      mon:
        disabled: false
      mgr:
        disabled: false
      osd:
        disabled: false
  • Cluster CR (custom resource), typically called cluster.yaml, if necessary

Logs to submit:

image

worker_1_rook-ceph-osd-prepare.log OK worker_2_rook-ceph-osd-prepare.log OK worker_3_rook-ceph-osd-prepare.log KO

  • Operator’s logs, if necessary

  • Crashing pod(s) logs, if necessary

    To get logs, use kubectl -n <namespace> logs <pod name> When pasting logs, always surround them with backticks or use the insert code button from the Github UI. Read GitHub documentation if you need help.

Cluster Status to submit:

  • Output of kubectl commands, if necessary

    To get the health of the cluster, use kubectl rook-ceph health To get the status of the cluster, use kubectl rook-ceph ceph status For more details, see the Rook kubectl Plugin

Environment:

  • OS (e.g. from /etc/os-release): image

  • Kernel (e.g. uname -a): image

  • Cloud provider or hardware configuration:

    • I have an Bare Metal K8S cluster with 3 masters and 3 workers.
    • All nodes (masters and workers) have Talos as OS
    • Each worker has the following configuration:
      • 3 x 2 TB NVME
      • 1 x 2 TB SDD
      • 1 x 4 TB SDD
      • 1 x 12TB HDD
  • Rook version (use rook version inside of a Rook Pod): image

  • Storage backend version (e.g. for ceph do ceph -v): image

  • Kubernetes version (use kubectl version): image

  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): Bare metal

  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): image

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 4
  • Comments: 33 (9 by maintainers)

Most upvoted comments

hi @cdalvaro, not sure if this is one of the possible solutions. ceph/ceph#49954 (comment)

Hi @microyahoo, thank you for your reply!

So the solution could be something like creating a file: /etc/udev/rules.d/99-usb-to-scsi.rules like the following and then applying the rule?

ACTION=="add", ENV{ID_TYPE}=="disk", ENV{ID_BUS}=="usb", ENV{ID_BUS}="scsi"
ACTION=="change", ENV{ID_TYPE}=="disk", ENV{ID_BUS}=="usb", ENV{ID_BUS}="scsi"
ACTION=="online", ENV{ID_TYPE}=="disk", ENV{ID_BUS}=="usb", ENV{ID_BUS}="scsi"

(I don’t really know about udev)

Edit: Apparently doing that, and then executing:

udevadm control --reload-rules && udevadm trigger

Changes the id_bus type:

# before
udevadm info --query=property /dev/sda | grep -i id_bus
ID_BUS=usb

# after
udevadm info --query=property /dev/sda | grep -i id_bus
ID_BUS=scsi

Much easier than using a custom ceph version 😳

Hi @cdalvaro, tried with Talos, but I had downgraded to 18.2.2 , get my ODS’s ready and then back to 18.2.4 so no idea if that works until I try with a new disk…

But for someone who wants to try it out with 18.2.4 and Talos, here’s a snippet of my talosconfig:

  patches:
    # set all disks to no scheduler and allow usb disks to be handled by rook/ceph
    - &udev |-
      machine:
        udev:
          rules:
            # set all disks to `none` scheduler (optimal setting for Ceph and ZFS)
            - SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ATTR{queue/scheduler}="none"
            # allow GID 44 (video) to use Intel GPU
            #- SUBSYSTEM=="drm", KERNEL=="renderD*", GROUP="44", MODE="0660"
            - SUBSYSTEM=="drm", GROUP="44", MODE="0660"
            # Allow ceph to recognize usb devices as scsi via udev custom rules
            # https://github.com/rook/rook/issues/14699#issuecomment-2350953135
            - ACTION=="add", ENV{ID_TYPE}=="disk", ENV{ID_BUS}=="usb", ENV{ID_BUS}="scsi"
            - ACTION=="change", ENV{ID_TYPE}=="disk", ENV{ID_BUS}=="usb", ENV{ID_BUS}="scsi"
            - ACTION=="online", ENV{ID_TYPE}=="disk", ENV{ID_BUS}=="usb", ENV{ID_BUS}="scsi"

that talosconfig snippet seems to be working as I’ve managed to get all 3 usb disks visible now

Thank you for this thread 🙏, you saved me hours of troubleshooting the failing OSDs in my RPI cluster with USB attached SSDs

For those running Ceph v18.2.4 and having this issue, I’ve made this repo: https://github.com/cdalvaro/docker-ceph which patches https://github.com/ceph/ceph/pull/49954 in order to allow USB devices.

You have the patched image available as: ghcr.io/cdalvaro/ceph:v18.2.4

I’ve managed to replace the original image with my custom one by running the following command:

ceph orch upgrade start --image ghcr.io/cdalvaro/ceph:v18.2.4

Edit: See comment https://github.com/rook/rook/issues/14699#issuecomment-2350953135 if you prefer to not use this image and patch the ID_BUS instead.

Did you try this with talos? I was wondering If I could setup a patch to write this into my talconfig.yaml 😄

Hi @rafaribe! I’ve tried it with AlmaLinux, but not with talos.

I tried to run the following command in the Job Batch:

chown ceph:ceph /dev/sdc && ceph-volume --log-path /tmp/ceph-volume.log inventory /dev/sdc && cat /tmp/ceph-volume.log

The full logs are here : job_script_v3.log

then I tried to update the cluster. It doesn’t change anything, I still get the same error :

2024-09-10 22:12:31.292684 D | exec: Running command: ceph-volume inventory --format json /dev/sdc
2024-09-10 22:12:31.791902 I | cephosd: skipping device "sdc": ["id_bus"].

The full logs are here : worker_3_provision_v2.log

hi @VxDlH, the udevadm info of /dev/sdc has ID_BUS=usb option, which will be rejected in https://github.com/ceph/ceph/blame/v18.2.4/src/ceph-volume/ceph_volume/util/device.py#L592-L598. screenshot-20240911-212657