rook: in toolbox pod, ceph tell osd.N command hangs when using Multus

  • Bug Report

ceph toolbox pod cannot talk to OSDs using ceph tell when Multus is enabled, but it can when Multus is not used.

Deviation from expected behavior:

The “ceph tell osd.N” command hangs, with no errors logged by either the OSD pod or the toolbox pod. The Ceph toolbox pod does not have Multus interfaces defined, and attempts to add the annotation similar to what the OSDs and MONs have did not work. But the toolbox pod must use Multus to talk to OSDs when Multus is enabled, yes?

Expected behavior:

ceph toolbox pod works same with and without multus.

How to reproduce it (minimal and precise):

alias ocrc=“oc -n rook-ceph” ocrc rsh $(ocrc get pod | awk ‘/tools/{print $1}’)
ceph tell osd.0 config get bluestore_default_buffered_read

File(s) to submit:

  • Cluster CR (custom resource), typically called cluster.yaml, if necessary

https://pastebin.com/FfdnUkH5

  • Operator’s logs, if necessary

nothing unusual

  • Crashing pod(s) logs, if necessary

none

Environment:

  • OS (e.g. from /etc/os-release):

RHEL_VERSION=“8.2” PRETTY_NAME=“Red Hat Enterprise Linux CoreOS 45.82.202009040230-0 (Ootpa)”

  • Kernel (e.g. uname -a):

Linux master-0 4.18.0-193.14.3.el8_2.x86_64 #1 SMP Mon Jul 20 15:02:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

  • Cloud provider or hardware configuration:

baremetal Alias lab Dell 740xds 25-GbE dual-port NIC/host 1 NVM device/host

  • Rook version (use rook version inside of a Rook Pod):

sh-4.4$ rook version rook: v1.4.0-alpha.0.352.g56fa357 go: go1.13.8

  • Storage backend version (e.g. for ceph do ceph -v):

ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)

  • Kubernetes version (use kubectl version):

[kni@e23-h15-740xd rook-multus]$ oc version Client Version: 4.5.9 Server Version: 4.5.9 Kubernetes Version: v1.18.3+6c42de8

  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):
  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 2
  • Comments: 20 (7 by maintainers)

Most upvoted comments

am still seeing this error, I think this is because the Ceph OSDs are on Multus interfaces, and the toolbox is not. What do you think? sorry this isn’t vanilla upstream rook-ceph. I got OCS 4.7,9 = ceph-version=14.2.11-147 to deploy with multus, but I had to define this before injecting the OCS CR:

apiVersion: v1
kind: ConfigMap
metadata:
  name: rook-config-override
  namespace: openshift-storage
data:
  config: |
    [global]
    public network = 192.168.231.0/24
    cluster network = 192.168.232.0/24

Otherwise OCS ignores Multus during install. It did deploy successfully, from OSD 1 log:

Normal AddedInterface 31m multus Add eth0 [10.129.3.100/23] Normal AddedInterface 31m multus Add net1 [192.168.232.6/24] from openshift-storage/cluster Normal AddedInterface 31m multus Add net2 [192.168.0.18/24] from openshift-storage/public

when I bring up the toolbox pod, if it fails to do ceph -s, there is a warning in the pod description that the rook-ceph-mon-endpoints configmap volume was not found. But when it does work, this configmap is defined and we get this error now:

sh-4.4# ceph -s
  cluster:
    id:     a8850b36-45bd-4f4d-9b28-4b2d2ccc7aba
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum a,b,c (age 25m)
    mgr: a(active, since 25m)
    mds: ocs-storagecluster-cephfilesystem:1 {0=ocs-storagecluster-cephfilesystem-b=up:active} 1 up:standby-replay
    osd: 3 osds: 3 up (since 24m), 3 in (since 24m)
 
  task status:
    scrub status:
        mds.ocs-storagecluster-cephfilesystem-a: idle
        mds.ocs-storagecluster-cephfilesystem-b: idle
 
  data:
    pools:   10 pools, 272 pgs
    objects: 98 objects, 149 MiB
    usage:   3.2 GiB used, 17 TiB / 17 TiB avail
    pgs:     272 active+clean
 
  io:
    client:   852 B/s rd, 8.7 KiB/s wr, 1 op/s rd, 1 op/s wr
 
sh-4.4# ceph tell osd.* cache drop
<hangs>