rook: in toolbox pod, ceph tell osd.N command hangs when using Multus
- Bug Report
ceph toolbox pod cannot talk to OSDs using ceph tell when Multus is enabled, but it can when Multus is not used.
Deviation from expected behavior:
The “ceph tell osd.N” command hangs, with no errors logged by either the OSD pod or the toolbox pod. The Ceph toolbox pod does not have Multus interfaces defined, and attempts to add the annotation similar to what the OSDs and MONs have did not work. But the toolbox pod must use Multus to talk to OSDs when Multus is enabled, yes?
Expected behavior:
ceph toolbox pod works same with and without multus.
How to reproduce it (minimal and precise):
alias ocrc=“oc -n rook-ceph”
ocrc rsh $(ocrc get pod | awk ‘/tools/{print $1}’)
ceph tell osd.0 config get bluestore_default_buffered_read
File(s) to submit:
- Cluster CR (custom resource), typically called
cluster.yaml, if necessary
- Operator’s logs, if necessary
nothing unusual
- Crashing pod(s) logs, if necessary
none
Environment:
- OS (e.g. from /etc/os-release):
RHEL_VERSION=“8.2” PRETTY_NAME=“Red Hat Enterprise Linux CoreOS 45.82.202009040230-0 (Ootpa)”
- Kernel (e.g.
uname -a):
Linux master-0 4.18.0-193.14.3.el8_2.x86_64 #1 SMP Mon Jul 20 15:02:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
- Cloud provider or hardware configuration:
baremetal Alias lab Dell 740xds 25-GbE dual-port NIC/host 1 NVM device/host
- Rook version (use
rook versioninside of a Rook Pod):
sh-4.4$ rook version rook: v1.4.0-alpha.0.352.g56fa357 go: go1.13.8
- Storage backend version (e.g. for ceph do
ceph -v):
ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)
- Kubernetes version (use
kubectl version):
[kni@e23-h15-740xd rook-multus]$ oc version Client Version: 4.5.9 Server Version: 4.5.9 Kubernetes Version: v1.18.3+6c42de8
- Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):
- Storage backend status (e.g. for Ceph use
ceph healthin the Rook Ceph toolbox):
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 2
- Comments: 20 (7 by maintainers)
am still seeing this error, I think this is because the Ceph OSDs are on Multus interfaces, and the toolbox is not. What do you think? sorry this isn’t vanilla upstream rook-ceph. I got OCS 4.7,9 = ceph-version=14.2.11-147 to deploy with multus, but I had to define this before injecting the OCS CR:
Otherwise OCS ignores Multus during install. It did deploy successfully, from OSD 1 log:
Normal AddedInterface 31m multus Add eth0 [10.129.3.100/23] Normal AddedInterface 31m multus Add net1 [192.168.232.6/24] from openshift-storage/cluster Normal AddedInterface 31m multus Add net2 [192.168.0.18/24] from openshift-storage/public
when I bring up the toolbox pod, if it fails to do ceph -s, there is a warning in the pod description that the rook-ceph-mon-endpoints configmap volume was not found. But when it does work, this configmap is defined and we get this error now: