rook: Rook Ceph: "failed to fetch mon config" after node reboot

Bug report:

I created a CephCluster (see this gist file) on a 3-node bare metal k8s cluster with 5 OSDs per node. I created it in namespace rook-ceph-ssd, because I couldn’t use namespace rook-ceph. Thus I first needed to deploy some RBAC resources (see this gist file). It seemed to work fine and all 15 OSDs were running (according to kubectl get pods) and up (according to ceph osd tree). However, after rebooting one of the 3 nodes, only 1 of 5 OSDs on that node came up:

ID   CLASS     WEIGHT    TYPE NAME        STATUS  REWEIGHT  PRI-AFF
 -1            22.20648  root default                              
-10             7.40216      host kube01                           
 11  ssd-nvme   0.12500          osd.11       up   1.00000  1.00000
  7  ssd-sata   1.81929          osd.7        up   1.00000  1.00000
  8  ssd-sata   1.81929          osd.8        up   1.00000  1.00000
 13  ssd-sata   1.81929          osd.13       up   1.00000  1.00000
 14  ssd-sata   1.81929          osd.14       up   1.00000  1.00000
 -7             7.40216      host kube02                           
 12  ssd-nvme   0.12500          osd.12       up   1.00000  1.00000
  1  ssd-sata   1.81929          osd.1        up   1.00000  1.00000
  2  ssd-sata   1.81929          osd.2        up   1.00000  1.00000
  4  ssd-sata   1.81929          osd.4        up   1.00000  1.00000
  9  ssd-sata   1.81929          osd.9        up   1.00000  1.00000
 -4             7.40216      host kube03                           
  6  ssd-nvme   0.12500          osd.6      down         0  1.00000
  0  ssd-sata   1.81929          osd.0      down         0  1.00000
  3  ssd-sata   1.81929          osd.3        up   1.00000  1.00000
  5  ssd-sata   1.81929          osd.5      down         0  1.00000
 10  ssd-sata   1.81929          osd.10     down         0  1.00000

How to reproduce it (minimal and precise):

  1. Deploy the operator in namespace rook-ceph
  2. Deploy abovementioned RBAC in namespace rook-ceph-ssd.
  3. Create StorageClasses rook-ceph-ssd-sata and rook-ceph-ssd-nvme, and create sufficient PersistentVolumes thereof.
  4. Deploy abovementioned CephCluster in namespace rook-ceph-ssd.
  5. Wait until all OSDs are up.
  6. Reboot a node.

Operator logs since last operator restart:

(Note that I filtered the log using grep.)

$ kubectl logs -n rook-ceph deploy/rook-ceph-operator | grep rook-ceph-ssd
2021-03-29 06:53:01.042029 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph-ssd"
2021-03-29 06:53:03.056691 I | op-mon: ceph mon status in namespace "rook-ceph-ssd" check interval "45s"
2021-03-29 06:53:03.056711 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph-ssd"
2021-03-29 06:53:03.056732 I | op-osd: ceph osd status in namespace "rook-ceph-ssd" check interval "60s"
2021-03-29 06:53:03.056745 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph-ssd"
2021-03-29 06:53:03.056777 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph-ssd"
2021-03-29 06:53:03.056785 I | op-client: start watching client resources in namespace "rook-ceph-ssd"
2021-03-29 06:53:03.645883 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph-ssd.ceph.rook.io/bucket"
I0329 06:53:03.647557       7 manager.go:118] objectbucket.io/provisioner-manager "msg"="starting provisioner"  "name"="rook-ceph-ssd.ceph.rook.io/bucket"
2021-03-29 06:53:09.245983 I | cephclient: writing config file /var/lib/rook/rook-ceph-ssd/rook-ceph-ssd.config
2021-03-29 06:53:09.246134 I | cephclient: generated admin config in /var/lib/rook/rook-ceph-ssd
2021-03-29 06:53:10.370180 I | ceph-cluster-controller: cluster "rook-ceph-ssd": version "15.2.10-0 octopus" detected for image "ceph/ceph:v15.2.10"
2021-03-29 06:53:13.053508 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph-ssd","monitors":["10.96.157.91:6789","10.96.111.158:6789","10.96.77.254:6789"]}] data:d=10.96.111.158:6789,b=10.96.77.254:6789,c=10.96.157.91:6789 mapping:{"node":{"b":null,"c":null,"d":null}} maxMonId:3]
2021-03-29 06:53:14.447764 I | cephclient: writing config file /var/lib/rook/rook-ceph-ssd/rook-ceph-ssd.config
2021-03-29 06:53:14.448068 I | cephclient: generated admin config in /var/lib/rook/rook-ceph-ssd
2021-03-29 06:53:21.248211 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph-ssd","monitors":["10.96.111.158:6789","10.96.77.254:6789","10.96.157.91:6789"]}] data:d=10.96.111.158:6789,b=10.96.77.254:6789,c=10.96.157.91:6789 mapping:{"node":{"b":null,"c":null,"d":null}} maxMonId:3]
2021-03-29 06:53:21.848683 I | cephclient: writing config file /var/lib/rook/rook-ceph-ssd/rook-ceph-ssd.config
2021-03-29 06:53:21.849011 I | cephclient: generated admin config in /var/lib/rook/rook-ceph-ssd
2021-03-29 06:53:22.446335 I | cephclient: writing config file /var/lib/rook/rook-ceph-ssd/rook-ceph-ssd.config
2021-03-29 06:53:22.446578 I | cephclient: generated admin config in /var/lib/rook/rook-ceph-ssd
2021-03-29 06:53:37.783194 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph-ssd"
2021-03-29 06:53:38.374651 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph-ssd"
2021-03-29 06:53:39.688934 I | op-osd: start running osds in namespace rook-ceph-ssd
2021-03-29 06:58:49.770074 W | op-osd: no valid nodes available to run osds on nodes in namespace rook-ceph-ssd
2021-03-29 06:58:50.860897 I | op-osd: completed running osds in namespace rook-ceph-ssd
2021-03-29 06:58:50.860916 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph-ssd"

OSD logs:

$ kubectl get pod
NAME                                                  READY   STATUS             RESTARTS   AGE
rook-ceph-crashcollector-kube01-65cc449469-j4tsc      1/1     Running            0          13h
rook-ceph-crashcollector-kube02-96468dbf8-nj6nk       1/1     Running            0          14h
rook-ceph-crashcollector-kube03-75bffcf575-b9qfl      1/1     Running            0          14h
rook-ceph-mgr-a-7749ddfc76-s8fl9                      1/1     Running            0          13h
rook-ceph-mon-b-6c6bb97dc-h9w95                       1/1     Running            0          14h
rook-ceph-mon-c-5c7c75bcd4-zqxd6                      1/1     Running            0          13h
rook-ceph-mon-d-86449646b9-qmwp8                      1/1     Running            0          14h
rook-ceph-osd-0-86c4f967c4-x82qg                      0/1     CrashLoopBackOff   165        13h
rook-ceph-osd-1-68bffc74f4-9lwbt                      1/1     Running            0          13h
rook-ceph-osd-10-6c7c649769-wv88p                     0/1     CrashLoopBackOff   167        13h
rook-ceph-osd-11-84bccf5c48-drhnq                     1/1     Running            0          13h
rook-ceph-osd-12-57879cd4b7-4xnj7                     1/1     Running            0          13h
rook-ceph-osd-13-5c668444c5-fjt7p                     1/1     Running            0          13h
rook-ceph-osd-14-595ff7f5fb-2pknw                     1/1     Running            0          13h
rook-ceph-osd-2-65f84b9b4d-75ddj                      1/1     Running            0          13h
rook-ceph-osd-3-67c84cb9c5-h2lqz                      1/1     Running            0          13h
rook-ceph-osd-4-6cd645954f-2wcrj                      1/1     Running            0          13h
rook-ceph-osd-5-6ddcc94d84-8flrk                      0/1     CrashLoopBackOff   165        13h
rook-ceph-osd-6-56554b9bc4-5pmf6                      0/1     CrashLoopBackOff   168        13h
rook-ceph-osd-7-8485889c98-7scgf                      1/1     Running            0          13h
rook-ceph-osd-8-774f98894c-kxx7w                      1/1     Running            0          13h
rook-ceph-osd-9-68cd549c5-mln6f                       1/1     Running            0          13h
rook-ceph-osd-prepare-ssd-nvme-0-data-1mrs6m-b4dfq    0/1     Completed          0          2d18h
rook-ceph-osd-prepare-ssd-nvme-0-data-2b4pv8-8msk6    0/1     Completed          0          2d18h
rook-ceph-osd-prepare-ssd-sata-0-data-10jfgh8-g2gw6   0/1     Completed          0          2d18h
rook-ceph-osd-prepare-ssd-sata-0-data-11sjdqj-j4brs   0/1     Completed          0          2d18h
rook-ceph-osd-prepare-ssd-sata-0-data-4lrmjh-4shl2    0/1     Completed          0          2d18h
rook-ceph-osd-prepare-ssd-sata-0-data-58grwx-zgf7p    0/1     Completed          0          2d18h
rook-ceph-osd-prepare-ssd-sata-0-data-67twsn-8f9kl    0/1     Completed          0          2d18h
rook-ceph-osd-prepare-ssd-sata-0-data-772pjf-pgfpm    0/1     Completed          0          2d18h
rook-ceph-osd-prepare-ssd-sata-0-data-8xddsk-mf8vc    0/1     Completed          0          2d18h
rook-ceph-osd-prepare-ssd-sata-0-data-9ktp4n-n5scq    0/1     Completed          0          2d18h
rook-ceph-tools-6f6c4f485-2jp9j                       1/1     Running            0          2d4h
$ kubectl logs rook-ceph-osd-0-86c4f967c4-x82qg
debug 2021-03-29T20:35:39.334+0000 7fca2bfc4700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2021-03-29T20:35:39.334+0000 7fca2c7c5700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2021-03-29T20:35:39.334+0000 7fca2cfc6700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
failed to fetch mon config (--no-mon-config to skip)
$ kubectl logs rook-ceph-osd-10-6c7c649769-wv88p
debug 2021-03-29T20:35:24.330+0000 7efe1146c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2021-03-29T20:35:24.330+0000 7efe1246e700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2021-03-29T20:35:24.330+0000 7efe11c6d700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
failed to fetch mon config (--no-mon-config to skip)
$ kubectl logs rook-ceph-osd-5-6ddcc94d84-8flrk
debug 2021-03-29T20:34:14.350+0000 7f6d17534700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2021-03-29T20:34:14.354+0000 7f6d16532700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2021-03-29T20:34:14.354+0000 7f6d16d33700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
failed to fetch mon config (--no-mon-config to skip)
$ kubectl logs rook-ceph-osd-6-56554b9bc4-5pmf6
debug 2021-03-29T20:34:40.414+0000 7f9f486e4700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2021-03-29T20:34:40.414+0000 7f9f48ee5700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2021-03-29T20:34:40.414+0000 7f9f47ee3700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
failed to fetch mon config (--no-mon-config to skip)

Environment:

  • OS: Debian GNU/Linux 10 (buster)
  • Kernel: 4.19.0-14-amd64
  • Cloud provider or hardware configuration: Bare-metal
  • Rook version: v1.5.6
  • Storage backend version: ceph/ceph:v15.2.10 / docker.io/ceph/ceph@sha256:030e84addad8f3a7d26ea49b180660f1a9b33ef06d691759f320a58e892fb535
  • Kubernetes version: v1.19.2
  • Kubernetes cluster type: kubeadm
  • Storage backend status: HEALTH_WARN mons b,c are low on available space

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 19 (9 by maintainers)

Most upvoted comments

For anyone else hitting this, I ended following this page to recover from a similar issue.

In my case, I was removed the OSD and then re-attempted adding it back into the CephCluster config (spec.storage.nodes[].devices).

To the OSD that’s failing to startup, I just had to update the keyring defined for the OSD (the older one persists but the newly spawned OSD generated a new one).

Can do this by (copied/pasted from the linked page):

  1. SSH onto Node for given OSD you need to recover
  2. In /var/lib/rook/rook-ceph, search for a directory containing the keyring and whoami files that have the number of the required Ceph OSD. For example:
    # cat keyring
    [osd.3]
    key = AQD2k/BlcE+YJxAA/QsD/fIAL1qPrh3hjQ7AKQ==
    
  3. Exec shell into ceph-tools Pod and do the following:
    1. execute: ceph auth get osd.<ID> -o /tmp/key
    2. update /tmp/key to reset the key to what you found in the keyring file earlier. For example
      vi /tmp/key
      # replace the key with the one from the keyring file
      [osd.3]
      key = AQD2k/BlcE+YJxAA/QsD/fIAL1qPrh3hjQ7AKQ==
      caps mgr = "allow profile osd"
      caps mon = "allow profile osd"
      caps osd = "allow *"
      
    3. import the replaced Ceph OSD keyring to the Ceph cluster: ceph auth import -i /tmp/key
  4. Finally, restart the OSD deployment: kubectl -n rook-ceph rollout restart deploy rook-ceph-osd-<ID>

I am new to Rook and met the same issue with Minikube. Just make sure spec/dataDirHostPath in cluster-test.yaml referencing a persistent data directory, for example, set it to /mnt/vda1/var/lib/rook. By default /var is mouted as tmpfs in Minikube-created VM.