rook: Rook Ceph: "failed to fetch mon config" after node reboot
Bug report:
I created a CephCluster (see this gist file) on a 3-node bare metal k8s cluster with 5 OSDs per node. I created it in namespace rook-ceph-ssd, because I couldn’t use namespace rook-ceph. Thus I first needed to deploy some RBAC resources (see this gist file). It seemed to work fine and all 15 OSDs were running (according to kubectl get pods) and up (according to ceph osd tree). However, after rebooting one of the 3 nodes, only 1 of 5 OSDs on that node came up:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 22.20648 root default
-10 7.40216 host kube01
11 ssd-nvme 0.12500 osd.11 up 1.00000 1.00000
7 ssd-sata 1.81929 osd.7 up 1.00000 1.00000
8 ssd-sata 1.81929 osd.8 up 1.00000 1.00000
13 ssd-sata 1.81929 osd.13 up 1.00000 1.00000
14 ssd-sata 1.81929 osd.14 up 1.00000 1.00000
-7 7.40216 host kube02
12 ssd-nvme 0.12500 osd.12 up 1.00000 1.00000
1 ssd-sata 1.81929 osd.1 up 1.00000 1.00000
2 ssd-sata 1.81929 osd.2 up 1.00000 1.00000
4 ssd-sata 1.81929 osd.4 up 1.00000 1.00000
9 ssd-sata 1.81929 osd.9 up 1.00000 1.00000
-4 7.40216 host kube03
6 ssd-nvme 0.12500 osd.6 down 0 1.00000
0 ssd-sata 1.81929 osd.0 down 0 1.00000
3 ssd-sata 1.81929 osd.3 up 1.00000 1.00000
5 ssd-sata 1.81929 osd.5 down 0 1.00000
10 ssd-sata 1.81929 osd.10 down 0 1.00000
How to reproduce it (minimal and precise):
- Deploy the operator in namespace
rook-ceph - Deploy abovementioned RBAC in namespace
rook-ceph-ssd. - Create StorageClasses
rook-ceph-ssd-sataandrook-ceph-ssd-nvme, and create sufficient PersistentVolumes thereof. - Deploy abovementioned CephCluster in namespace
rook-ceph-ssd. - Wait until all OSDs are up.
- Reboot a node.
Operator logs since last operator restart:
(Note that I filtered the log using grep.)
$ kubectl logs -n rook-ceph deploy/rook-ceph-operator | grep rook-ceph-ssd
2021-03-29 06:53:01.042029 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph-ssd"
2021-03-29 06:53:03.056691 I | op-mon: ceph mon status in namespace "rook-ceph-ssd" check interval "45s"
2021-03-29 06:53:03.056711 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph-ssd"
2021-03-29 06:53:03.056732 I | op-osd: ceph osd status in namespace "rook-ceph-ssd" check interval "60s"
2021-03-29 06:53:03.056745 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph-ssd"
2021-03-29 06:53:03.056777 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph-ssd"
2021-03-29 06:53:03.056785 I | op-client: start watching client resources in namespace "rook-ceph-ssd"
2021-03-29 06:53:03.645883 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph-ssd.ceph.rook.io/bucket"
I0329 06:53:03.647557 7 manager.go:118] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph-ssd.ceph.rook.io/bucket"
2021-03-29 06:53:09.245983 I | cephclient: writing config file /var/lib/rook/rook-ceph-ssd/rook-ceph-ssd.config
2021-03-29 06:53:09.246134 I | cephclient: generated admin config in /var/lib/rook/rook-ceph-ssd
2021-03-29 06:53:10.370180 I | ceph-cluster-controller: cluster "rook-ceph-ssd": version "15.2.10-0 octopus" detected for image "ceph/ceph:v15.2.10"
2021-03-29 06:53:13.053508 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph-ssd","monitors":["10.96.157.91:6789","10.96.111.158:6789","10.96.77.254:6789"]}] data:d=10.96.111.158:6789,b=10.96.77.254:6789,c=10.96.157.91:6789 mapping:{"node":{"b":null,"c":null,"d":null}} maxMonId:3]
2021-03-29 06:53:14.447764 I | cephclient: writing config file /var/lib/rook/rook-ceph-ssd/rook-ceph-ssd.config
2021-03-29 06:53:14.448068 I | cephclient: generated admin config in /var/lib/rook/rook-ceph-ssd
2021-03-29 06:53:21.248211 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph-ssd","monitors":["10.96.111.158:6789","10.96.77.254:6789","10.96.157.91:6789"]}] data:d=10.96.111.158:6789,b=10.96.77.254:6789,c=10.96.157.91:6789 mapping:{"node":{"b":null,"c":null,"d":null}} maxMonId:3]
2021-03-29 06:53:21.848683 I | cephclient: writing config file /var/lib/rook/rook-ceph-ssd/rook-ceph-ssd.config
2021-03-29 06:53:21.849011 I | cephclient: generated admin config in /var/lib/rook/rook-ceph-ssd
2021-03-29 06:53:22.446335 I | cephclient: writing config file /var/lib/rook/rook-ceph-ssd/rook-ceph-ssd.config
2021-03-29 06:53:22.446578 I | cephclient: generated admin config in /var/lib/rook/rook-ceph-ssd
2021-03-29 06:53:37.783194 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph-ssd"
2021-03-29 06:53:38.374651 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph-ssd"
2021-03-29 06:53:39.688934 I | op-osd: start running osds in namespace rook-ceph-ssd
2021-03-29 06:58:49.770074 W | op-osd: no valid nodes available to run osds on nodes in namespace rook-ceph-ssd
2021-03-29 06:58:50.860897 I | op-osd: completed running osds in namespace rook-ceph-ssd
2021-03-29 06:58:50.860916 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph-ssd"
OSD logs:
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
rook-ceph-crashcollector-kube01-65cc449469-j4tsc 1/1 Running 0 13h
rook-ceph-crashcollector-kube02-96468dbf8-nj6nk 1/1 Running 0 14h
rook-ceph-crashcollector-kube03-75bffcf575-b9qfl 1/1 Running 0 14h
rook-ceph-mgr-a-7749ddfc76-s8fl9 1/1 Running 0 13h
rook-ceph-mon-b-6c6bb97dc-h9w95 1/1 Running 0 14h
rook-ceph-mon-c-5c7c75bcd4-zqxd6 1/1 Running 0 13h
rook-ceph-mon-d-86449646b9-qmwp8 1/1 Running 0 14h
rook-ceph-osd-0-86c4f967c4-x82qg 0/1 CrashLoopBackOff 165 13h
rook-ceph-osd-1-68bffc74f4-9lwbt 1/1 Running 0 13h
rook-ceph-osd-10-6c7c649769-wv88p 0/1 CrashLoopBackOff 167 13h
rook-ceph-osd-11-84bccf5c48-drhnq 1/1 Running 0 13h
rook-ceph-osd-12-57879cd4b7-4xnj7 1/1 Running 0 13h
rook-ceph-osd-13-5c668444c5-fjt7p 1/1 Running 0 13h
rook-ceph-osd-14-595ff7f5fb-2pknw 1/1 Running 0 13h
rook-ceph-osd-2-65f84b9b4d-75ddj 1/1 Running 0 13h
rook-ceph-osd-3-67c84cb9c5-h2lqz 1/1 Running 0 13h
rook-ceph-osd-4-6cd645954f-2wcrj 1/1 Running 0 13h
rook-ceph-osd-5-6ddcc94d84-8flrk 0/1 CrashLoopBackOff 165 13h
rook-ceph-osd-6-56554b9bc4-5pmf6 0/1 CrashLoopBackOff 168 13h
rook-ceph-osd-7-8485889c98-7scgf 1/1 Running 0 13h
rook-ceph-osd-8-774f98894c-kxx7w 1/1 Running 0 13h
rook-ceph-osd-9-68cd549c5-mln6f 1/1 Running 0 13h
rook-ceph-osd-prepare-ssd-nvme-0-data-1mrs6m-b4dfq 0/1 Completed 0 2d18h
rook-ceph-osd-prepare-ssd-nvme-0-data-2b4pv8-8msk6 0/1 Completed 0 2d18h
rook-ceph-osd-prepare-ssd-sata-0-data-10jfgh8-g2gw6 0/1 Completed 0 2d18h
rook-ceph-osd-prepare-ssd-sata-0-data-11sjdqj-j4brs 0/1 Completed 0 2d18h
rook-ceph-osd-prepare-ssd-sata-0-data-4lrmjh-4shl2 0/1 Completed 0 2d18h
rook-ceph-osd-prepare-ssd-sata-0-data-58grwx-zgf7p 0/1 Completed 0 2d18h
rook-ceph-osd-prepare-ssd-sata-0-data-67twsn-8f9kl 0/1 Completed 0 2d18h
rook-ceph-osd-prepare-ssd-sata-0-data-772pjf-pgfpm 0/1 Completed 0 2d18h
rook-ceph-osd-prepare-ssd-sata-0-data-8xddsk-mf8vc 0/1 Completed 0 2d18h
rook-ceph-osd-prepare-ssd-sata-0-data-9ktp4n-n5scq 0/1 Completed 0 2d18h
rook-ceph-tools-6f6c4f485-2jp9j 1/1 Running 0 2d4h
$ kubectl logs rook-ceph-osd-0-86c4f967c4-x82qg
debug 2021-03-29T20:35:39.334+0000 7fca2bfc4700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2021-03-29T20:35:39.334+0000 7fca2c7c5700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2021-03-29T20:35:39.334+0000 7fca2cfc6700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
failed to fetch mon config (--no-mon-config to skip)
$ kubectl logs rook-ceph-osd-10-6c7c649769-wv88p
debug 2021-03-29T20:35:24.330+0000 7efe1146c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2021-03-29T20:35:24.330+0000 7efe1246e700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2021-03-29T20:35:24.330+0000 7efe11c6d700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
failed to fetch mon config (--no-mon-config to skip)
$ kubectl logs rook-ceph-osd-5-6ddcc94d84-8flrk
debug 2021-03-29T20:34:14.350+0000 7f6d17534700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2021-03-29T20:34:14.354+0000 7f6d16532700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2021-03-29T20:34:14.354+0000 7f6d16d33700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
failed to fetch mon config (--no-mon-config to skip)
$ kubectl logs rook-ceph-osd-6-56554b9bc4-5pmf6
debug 2021-03-29T20:34:40.414+0000 7f9f486e4700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2021-03-29T20:34:40.414+0000 7f9f48ee5700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2021-03-29T20:34:40.414+0000 7f9f47ee3700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
failed to fetch mon config (--no-mon-config to skip)
Environment:
- OS: Debian GNU/Linux 10 (buster)
- Kernel: 4.19.0-14-amd64
- Cloud provider or hardware configuration: Bare-metal
- Rook version: v1.5.6
- Storage backend version:
ceph/ceph:v15.2.10/docker.io/ceph/ceph@sha256:030e84addad8f3a7d26ea49b180660f1a9b33ef06d691759f320a58e892fb535 - Kubernetes version: v1.19.2
- Kubernetes cluster type: kubeadm
- Storage backend status:
HEALTH_WARN mons b,c are low on available space
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 19 (9 by maintainers)
For anyone else hitting this, I ended following this page to recover from a similar issue.
In my case, I was removed the OSD and then re-attempted adding it back into the
CephClusterconfig (spec.storage.nodes[].devices).To the OSD that’s failing to startup, I just had to update the
keyringdefined for the OSD (the older one persists but the newly spawned OSD generated a new one).Can do this by (copied/pasted from the linked page):
/var/lib/rook/rook-ceph, search for a directory containing thekeyringandwhoamifiles that have the number of the required Ceph OSD. For example:ceph-toolsPod and do the following:ceph auth get osd.<ID> -o /tmp/key/tmp/keyto reset the key to what you found in the keyring file earlier. For exampleceph auth import -i /tmp/keykubectl -n rook-ceph rollout restart deploy rook-ceph-osd-<ID>I am new to Rook and met the same issue with Minikube. Just make sure
spec/dataDirHostPathincluster-test.yamlreferencing a persistent data directory, for example, set it to/mnt/vda1/var/lib/rook. By default/varis mouted as tmpfs in Minikube-created VM.