rook: mgr pod in CrashLoop in 0.8.x

Is this a bug report or feature request? Bug Report

Deviation from expected behavior: After rescheduling the mgr pod, it goes into a CrashLoop with the following:

2018-09-30 19:26:48.956022 I | ceph-mgr: 2018-09-30 19:26:48.955809 7f0adebf4700  1 mgr send_beacon active
2018-09-30 19:26:50.970860 I | ceph-mgr: 2018-09-30 19:26:50.970649 7f0adebf4700  1 mgr send_beacon active
2018-09-30 19:26:52.985827 I | ceph-mgr: 2018-09-30 19:26:52.985611 7f0adebf4700  1 mgr send_beacon active
2018-09-30 19:26:54.004538 I | ceph-mgr: [30/Sep/2018:19:26:47] ENGINE Bus STARTING
2018-09-30 19:26:54.004566 I | ceph-mgr: CherryPy Checker:
2018-09-30 19:26:54.004575 I | ceph-mgr: The Application mounted at '' has an empty config.
2018-09-30 19:26:54.004581 I | ceph-mgr: 
2018-09-30 19:26:54.004588 I | ceph-mgr: [30/Sep/2018:19:26:47] ENGINE Started monitor thread '_TimeoutMonitor'.
2018-09-30 19:26:54.004594 I | ceph-mgr: [30/Sep/2018:19:26:47] ENGINE Bus STARTING
2018-09-30 19:26:54.004600 I | ceph-mgr: [30/Sep/2018:19:26:47] ENGINE Started monitor thread '_TimeoutMonitor'.
2018-09-30 19:26:54.004606 I | ceph-mgr: [30/Sep/2018:19:26:47] ENGINE Serving on :::7000
2018-09-30 19:26:54.004611 I | ceph-mgr: [30/Sep/2018:19:26:47] ENGINE Bus STARTED
2018-09-30 19:26:54.004624 I | ceph-mgr: [30/Sep/2018:19:26:47] ENGINE Serving on :::9283
2018-09-30 19:26:54.004630 I | ceph-mgr: [30/Sep/2018:19:26:47] ENGINE Bus STARTED
2018-09-30 19:26:54.004636 I | ceph-mgr: terminate called after throwing an instance of 'std::out_of_range'
2018-09-30 19:26:54.004644 I | ceph-mgr:   what():  map::at
failed to run mgr. failed to start mgr: Failed to complete 'ceph-mgr': signal: aborted (core dumped).

Expected behavior: No crash loop 😉

How to reproduce it (minimal and precise): Personally I’ve experienced it in several test clusters, but haven’t had time to dig into it until tonight. Another user mentioned this a week ago on slack and @galexrt pointed to this bug in Ceph that seems to be related: https://tracker.ceph.com/issues/24982

In that issue, people mention multiple RGWs and I’m running RGWs as a daemonset for these clusters. So I tried to scale the amount of RGWs down to 1, using NodeAffinity, and nor the mgr was able to start up. After it’s started, I can scale the RGWs back up to full count (5 on testing cluster) in one go and the mgr stays up. Without knowing this in depth, it seems to me the RGWs are building up a history of metrics to deliver to the mgr when they can’t reach it. When the mgr starts again, these historic metrics overwhelms it and it gets startled, not to confuse with started.

Environment:

• OS (e.g. from /etc/os-release): CoreOS
• Kernel (e.g. uname -a): Something new
• Cloud provider or hardware configuration: Bare metal
• Rook version (use rook version inside of a Rook Pod): v0.8.1 - 99% sure I saw it on 0.8.2 as well, but had to vacate that due to a different bug.
• Kubernetes version (use kubectl version): 1.11.3
• Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): Kubespray
• Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): Seems happy, but no dashboard and metrics obviously.