rook: Disk ignored because of id_bus
Is this a bug report or feature request?
- Bug Report
Deviation from expected behavior:
I added 3 hard drives in one on each worker. On two workers, the operation went smoothly and the hard disks were successfully added to the Ceph.
On the third worker my hard disk is excluded from rook-ceph-osd-prepare processing. The error displayed is as follows.
cephosd: skipping device "sdc": ["id_bus"].
I don’t understand why I’m getting this error message.
I did the same thing on my other 2 Workers and it worked without a hitch.
Expected behavior:
Normally, the disk should be added in Ceph on Woker n°3.
How to reproduce it (minimal and precise):
File(s) to submit: CephCluster
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
name: cluster-global
namespace: rook-cluster-global # namespace:cluster
spec:
cephVersion:
image: quay.io/ceph/ceph:v18.2.4
allowUnsupported: false
dataDirHostPath: /var/lib/rook/global
skipUpgradeChecks: false
continueUpgradeAfterChecksEvenIfNotHealthy: false
waitTimeoutForHealthyOSDInMinutes: 10
upgradeOSDRequiresHealthyPGs: false
mon:
count: 3
allowMultiplePerNode: false
mgr:
count: 2
allowMultiplePerNode: false
modules:
- name: rook
enabled: true
dashboard:
enabled: true
ssl: false
monitoring:
enabled: false
metricsDisabled: false
network:
connections:
volumes.
encryption:
enabled: false
compression:
enabled: false
requireMsgr2: false
crashCollector:
disable: false
logCollector:
enabled: true
periodicity: daily # one of: hourly, daily, weekly, monthly
maxLogSize: 500M # SUFFIX may be 'M' or 'G'. Must be at least 1M.
cleanupPolicy:
confirmation: ""
sanitizeDisks:
method: quick
dataSource: zero
iteration: 1
allowUninstallWithVolumes: false
placement:
all:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: rook-cluster-global-storage-node
operator: In
values:
- "true"
podAffinity:
podAntiAffinity:
topologySpreadConstraints:
tolerations:
- key: rook-cluster-global-storage-node
operator: Exists
labels:
resources:
removeOSDsIfOutAndSafeToRemove: false
priorityClassNames:
mon: system-node-critical
osd: system-node-critical
mgr: system-cluster-critical
storage:
useAllNodes: false
useAllDevices: false
devices:
nodes:
- name: "kube-worker-1"
devices:
- name: /dev/disk/by-id/..........
- name: /dev/disk/by-id/..........
- .......
- name: "kube-worker-2"
devices:
- name: /dev/disk/by-id/..........
- name: /dev/disk/by-id/..........
- .......
- name: "kube-worker-3"
- name: /dev/disk/by-id/..........
- name: /dev/disk/by-id/..........
- .......
config:
allowDeviceClassUpdate: false
allowOsdCrushWeightUpdate: false
onlyApplyOSDPlacement: false
disruptionManagement:
managePodBudgets: true
osdMaintenanceTimeout: 30
pgHealthCheckTimeout: 0
csi:
readAffinity:
enabled: true
cephfs:
healthCheck:
daemonHealth:
mon:
disabled: false
interval: 45s
osd:
disabled: false
interval: 60s
status:
disabled: false
interval: 60s
livenessProbe:
mon:
disabled: false
mgr:
disabled: false
osd:
disabled: false
startupProbe:
mon:
disabled: false
mgr:
disabled: false
osd:
disabled: false
- Cluster CR (custom resource), typically called
cluster.yaml, if necessary
Logs to submit:
worker_1_rook-ceph-osd-prepare.log OK
worker_2_rook-ceph-osd-prepare.log OK
worker_3_rook-ceph-osd-prepare.log KO
-
Operator’s logs, if necessary
-
Crashing pod(s) logs, if necessary
To get logs, use
kubectl -n <namespace> logs <pod name>When pasting logs, always surround them with backticks or use theinsert codebutton from the Github UI. Read GitHub documentation if you need help.
Cluster Status to submit:
-
Output of kubectl commands, if necessary
To get the health of the cluster, use
kubectl rook-ceph healthTo get the status of the cluster, usekubectl rook-ceph ceph statusFor more details, see the Rook kubectl Plugin
Environment:
-
OS (e.g. from /etc/os-release):
-
Kernel (e.g.
uname -a): -
Cloud provider or hardware configuration:
- I have an Bare Metal K8S cluster with 3 masters and 3 workers.
- All nodes (masters and workers) have Talos as OS
- Each worker has the following configuration:
- 3 x 2 TB NVME
- 1 x 2 TB SDD
- 1 x 4 TB SDD
- 1 x 12TB HDD
-
Rook version (use
rook versioninside of a Rook Pod): -
Storage backend version (e.g. for ceph do
ceph -v): -
Kubernetes version (use
kubectl version): -
Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):
Bare metal -
Storage backend status (e.g. for Ceph use
ceph healthin the Rook Ceph toolbox):
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 4
- Comments: 33 (9 by maintainers)
Hi @microyahoo, thank you for your reply!
So the solution could be something like creating a file:
/etc/udev/rules.d/99-usb-to-scsi.ruleslike the following and then applying the rule?(I don’t really know about
udev)Edit: Apparently doing that, and then executing:
Changes the id_bus type:
Much easier than using a custom ceph version 😳
Hi @cdalvaro, tried with Talos, but I had downgraded to
18.2.2, get my ODS’s ready and then back to18.2.4so no idea if that works until I try with a new disk…But for someone who wants to try it out with
18.2.4and Talos, here’s a snippet of my talosconfig:that talosconfig snippet seems to be working as I’ve managed to get all 3 usb disks visible now
Thank you for this thread 🙏, you saved me hours of troubleshooting the failing OSDs in my RPI cluster with USB attached SSDs
For those running Ceph v18.2.4 and having this issue, I’ve made this repo: https://github.com/cdalvaro/docker-ceph which patches https://github.com/ceph/ceph/pull/49954 in order to allow USB devices.
You have the patched image available as:
ghcr.io/cdalvaro/ceph:v18.2.4I’ve managed to replace the original image with my custom one by running the following command:
Edit: See comment https://github.com/rook/rook/issues/14699#issuecomment-2350953135 if you prefer to not use this image and patch the
ID_BUSinstead.Hi @rafaribe! I’ve tried it with AlmaLinux, but not with talos.
Hi @VxDlH , the PR https://github.com/ceph/ceph/pull/49954/files is intended to fix https://github.com/rook/rook/issues/11474
hi @VxDlH, the
udevadm infoof /dev/sdc hasID_BUS=usboption, which will be rejected in https://github.com/ceph/ceph/blame/v18.2.4/src/ceph-volume/ceph_volume/util/device.py#L592-L598.