rook: CSI pods intermittently disappear

Is this a bug report or feature request?

• Bug Report

Deviation from expected behavior:

• The Ceph-CSI-Pods (Plugin and Provisioner) should not beeing killed if the Operator is restarted for whatever reason…

Expected behavior:

• The CSI-Pods are running normally.

How to reproduce it (minimal and precise):

• Create a CephFS-Cluster with the current images (Rook 1.2.0), I used helm for that.
• Enable the CephFS-CSI Driver and create a CephFilesystem resource (see cluster.yaml)
• The Cluster starts and works as expected.
• At some time the Operator crashes and restarts (see old-operator.log)
• The new Operator reconciles the Cluster and kills all CSI-Pods and never starts them again. (No error logs or events from kubernetes here) (see new-operator.log)
• This behavior occurs not always, but really often. Is there any way to start those pods again and get the cluster working again?

File(s) to submit: cluster.yaml

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: ceph/ceph:v14.2.5-20191210
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: false
  mgr:
    modules:
    - name: pg_autoscaler
      enabled: true
  dashboard:
    enabled: true
  monitoring:
    enabled: true
    rulesNamespace: rook-ceph
  priorityClassNames:
    all: infra-priority
  storage:
    useAllNodes: false
    useAllDevices: false
    config:
      databaseSizeMB: "1024" # this value can be removed for environments with normal sized disks (100 GB or larger)
      journalSizeMB: "1024"  # this value can be removed for environments with normal sized disks (20 GB or larger)
      osdsPerDevice: "1" # this value can be overridden at the node or device level
    # nodes below will be used as storage resources.  Each node's 'name' field should match their 'kubernetes.io/hostname' label.
    nodes:
    - name: "kubeworker1"
      directories: # specific devices to use for storage can be specified for each node
      - path: "/mnt/volume01"
    - name: "kubeworker2"
      directories: # specific devices to use for storage can be specified for each node
      - path: "/mnt/volume01"
    - name: "kubeworker3"
      directories: # specific devices to use for storage can be specified for each node
      - path: "/mnt/volume01"

---
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: ceph-fs
  namespace: rook-ceph
spec:
  metadataPool:
    failureDomain: host
    replicated:
      size: 3
  dataPools:
    - failureDomain: host
      replicated:
        size: 3
  preservePoolsOnDelete: true
  metadataServer:
    activeCount: 2
    activeStandby: true
    placement:
       podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - rook-ceph-mds
              topologyKey: kubernetes.io/hostname

old-operator.log

2019-12-23 16:38:06.849317 I | rookcmd: starting Rook v1.2.0 with arguments '/usr/local/bin/rook ceph operator'
2019-12-23 16:38:06.849479 I | rookcmd: flag values: --add_dir_header=false, --alsologtostderr=false, --csi-attacher-image=quay.io/k8scsi/csi-attacher:v1.2.0, --csi-ceph-image=quay.io/cephcsi/cephcsi:v1.2.2, --csi-cephfs-plugin-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin.yaml, --csi-cephfs-provisioner-dep-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin-provisioner-dep.yaml, --csi-cephfs-provisioner-sts-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin-provisioner-sts.yaml, --csi-driver-name-prefix=, --csi-enable-cephfs=true, --csi-enable-grpc-metrics=false, --csi-enable-rbd=false, --csi-kubelet-dir-path=/var/lib/kubelet, --csi-provisioner-image=quay.io/k8scsi/csi-provisioner:v1.4.0, --csi-rbd-plugin-template-path=/etc/ceph-csi/rbd/csi-rbdplugin.yaml, --csi-rbd-provisioner-dep-template-path=/etc/ceph-csi/rbd/csi-rbdplugin-provisioner-dep.yaml, --csi-rbd-provisioner-sts-template-path=/etc/ceph-csi/rbd/csi-rbdplugin-provisioner-sts.yaml, --csi-registrar-image=quay.io/k8scsi/csi-node-driver-registrar:v1.1.0, --csi-snapshotter-image=quay.io/k8scsi/csi-snapshotter:v1.2.2, --enable-discovery-daemon=true, --enable-flex-driver=false, --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-flush-frequency=5s, --log-level=INFO, --log_backtrace_at=:0, --log_dir=, --log_file=, --log_file_max_size=1800, --logtostderr=true, --master=, --mon-healthcheck-interval=45s, --mon-out-timeout=10m0s, --operator-image=, --service-account=, --skip_headers=false, --skip_log_headers=false, --stderrthreshold=2, --v=0, --vmodule=
2019-12-23 16:38:06.849488 I | cephcmd: starting operator
2019-12-23 16:38:07.028118 I | op-discover: rook-discover daemonset already exists, updating ...
2019-12-23 16:38:07.052427 I | operator: rook-provisioner ceph.rook.io/block started using ceph.rook.io flex vendor dir
2019-12-23 16:38:07.053166 I | operator: rook-provisioner rook.io/block started using rook.io flex vendor dir
2019-12-23 16:38:07.053187 I | operator: Watching all namespaces for cluster CRDs
2019-12-23 16:38:07.053209 I | op-cluster: start watching clusters in all namespaces
2019-12-23 16:38:07.053529 I | op-cluster: Enabling hotplug orchestration: ROOK_DISABLE_DEVICE_HOTPLUG=false
I1223 16:38:07.054396       7 leaderelection.go:217] attempting to acquire leader lease  rook-ceph/ceph.rook.io-block...
I1223 16:38:07.054793       7 leaderelection.go:217] attempting to acquire leader lease  rook-ceph/rook.io-block...
2019-12-23 16:38:07.055398 I | operator: setting up the controller-runtime manager
2019-12-23 16:38:07.080069 I | op-cluster: starting cluster in namespace rook-ceph
2019-12-23 16:38:07.452301 I | ceph-csi: CSIDriver CRD already had been registered for "rook-ceph.rbd.csi.ceph.com"
2019-12-23 16:38:07.478364 I | ceph-csi: CSIDriver CRD already had been registered for "rook-ceph.cephfs.csi.ceph.com"
2019-12-23 16:38:07.478410 I | operator: successfully started Ceph CSI driver(s)
2019-12-23 16:38:07.668011 I | operator: starting the controller-runtime manager
2019-12-23 16:38:07.972028 I | clusterdisruption-controller: clusterName is not known yet for namespace "rook-ceph"
2019-12-23 16:38:07.972129 E | clusterdisruption-controller: clusterName for this namespace not yet known
2019-12-23 16:38:13.479225 I | op-cluster: detecting the ceph image version for image ceph/ceph:v14.2.5-20191210...
2019-12-23 16:38:21.056695 I | op-cluster: Detected ceph image version: "14.2.5-0 nautilus"
2019-12-23 16:38:21.077234 I | op-mon: parsing mon endpoints: a=10.96.122.158:6789,b=10.96.116.199:6789,c=10.96.253.5:6789
2019-12-23 16:38:21.077427 I | op-mon: loaded: maxMonID=2, mons=map[a:0xc0011c5060 b:0xc0011c50a0 c:0xc0011c50e0], mapping=&{Node:map[a:0xc000a491d0 b:0xc000a49260 c:0xc000a49290]}
2019-12-23 16:38:21.078006 I | cephconfig: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2019-12-23 16:38:21.078111 I | cephconfig: generated admin config in /var/lib/rook/rook-ceph
2019-12-23 16:38:21.078258 I | exec: Running command: ceph versions --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/289733070
2019-12-23 16:38:22.338274 I | op-cluster: CephCluster "rook-ceph" status: "Creating". 
2019-12-23 16:38:22.442419 I | op-mon: start running mons
2019-12-23 16:38:22.455570 I | op-mon: parsing mon endpoints: a=10.96.122.158:6789,b=10.96.116.199:6789,c=10.96.253.5:6789
2019-12-23 16:38:22.455703 I | op-mon: loaded: maxMonID=2, mons=map[a:0xc000522d00 b:0xc000522d80 c:0xc000522e00], mapping=&{Node:map[a:0xc000b3ccc0 b:0xc000b3ccf0 c:0xc000b3cd50]}
2019-12-23 16:38:22.484681 I | op-mon: saved mon endpoints to config map map[mapping:{"node":{"a":{"Name":"kubeworker2","Hostname":"kubeworker2","Address":"88.99.121.47"},"b":{"Name":"kubeworker1","Hostname":"kubeworker1","Address":"116.202.30.255"},"c":{"Name":"kubeworker3","Hostname":"kubeworker3","Address":"116.202.30.249"}}} csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.96.122.158:6789","10.96.116.199:6789","10.96.253.5:6789"]}] data:a=10.96.122.158:6789,b=10.96.116.199:6789,c=10.96.253.5:6789 maxMonId:2]
2019-12-23 16:38:22.502212 I | cephconfig: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2019-12-23 16:38:22.502514 I | cephconfig: generated admin config in /var/lib/rook/rook-ceph
2019-12-23 16:38:23.005230 I | op-mon: targeting the mon count 3
2019-12-23 16:38:23.005575 I | exec: Running command: ceph config set global mon_allow_pool_delete true --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/002515157
I1223 16:38:23.919195       7 leaderelection.go:227] successfully acquired lease rook-ceph/rook.io-block
I1223 16:38:23.919704       7 controller.go:769] Starting provisioner controller rook.io/block_rook-ceph-operator-575b89745f-wzjjc_9907028a-25a2-11ea-a613-cafea86acf8a!
I1223 16:38:23.920055       7 event.go:209] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"rook-ceph", Name:"rook.io-block", UID:"a2fbe584-eddf-4161-a215-08fd85b60825", APIVersion:"v1", ResourceVersion:"11870", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' rook-ceph-operator-575b89745f-wzjjc_9907028a-25a2-11ea-a613-cafea86acf8a became leader
I1223 16:38:24.319959       7 controller.go:818] Started provisioner controller rook.io/block_rook-ceph-operator-575b89745f-wzjjc_9907028a-25a2-11ea-a613-cafea86acf8a!
2019-12-23 16:38:24.518901 I | exec: Running command: ceph config set global rbd_default_features 3 --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/171030576
I1223 16:38:25.742238       7 leaderelection.go:227] successfully acquired lease rook-ceph/ceph.rook.io-block
I1223 16:38:25.742887       7 controller.go:769] Starting provisioner controller ceph.rook.io/block_rook-ceph-operator-575b89745f-wzjjc_9906dc63-25a2-11ea-a613-cafea86acf8a!
I1223 16:38:25.742935       7 event.go:209] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"rook-ceph", Name:"ceph.rook.io-block", UID:"39e9eac3-7e86-43f7-91b7-6235ca9dcba8", APIVersion:"v1", ResourceVersion:"11884", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' rook-ceph-operator-575b89745f-wzjjc_9906dc63-25a2-11ea-a613-cafea86acf8a became leader
2019-12-23 16:38:25.838008 I | op-mon: checking for basic quorum with existing mons
I1223 16:38:25.843172       7 controller.go:818] Started provisioner controller ceph.rook.io/block_rook-ceph-operator-575b89745f-wzjjc_9906dc63-25a2-11ea-a613-cafea86acf8a!
2019-12-23 16:38:26.414229 I | op-mon: mon "a" endpoint are [v2:10.96.122.158:3300,v1:10.96.122.158:6789]
2019-12-23 16:38:27.408162 I | op-mon: mon "b" endpoint are [v2:10.96.116.199:3300,v1:10.96.116.199:6789]
2019-12-23 16:38:28.011048 I | op-mon: mon "c" endpoint are [v2:10.96.253.5:3300,v1:10.96.253.5:6789]
2019-12-23 16:38:28.814819 I | op-mon: saved mon endpoints to config map map[mapping:{"node":{"a":{"Name":"kubeworker2","Hostname":"kubeworker2","Address":"88.99.121.47"},"b":{"Name":"kubeworker1","Hostname":"kubeworker1","Address":"116.202.30.255"},"c":{"Name":"kubeworker3","Hostname":"kubeworker3","Address":"116.202.30.249"}}} csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.96.122.158:6789","10.96.116.199:6789","10.96.253.5:6789"]}] data:b=10.96.116.199:6789,c=10.96.253.5:6789,a=10.96.122.158:6789 maxMonId:2]
2019-12-23 16:38:29.203353 I | cephconfig: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2019-12-23 16:38:29.203721 I | cephconfig: generated admin config in /var/lib/rook/rook-ceph
2019-12-23 16:38:29.803416 I | cephconfig: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2019-12-23 16:38:29.803641 I | cephconfig: generated admin config in /var/lib/rook/rook-ceph
2019-12-23 16:38:29.866364 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed
2019-12-23 16:38:29.877551 I | op-k8sutil: updating deployment rook-ceph-mon-a
2019-12-23 16:38:31.971652 I | op-k8sutil: finished waiting for updated deployment rook-ceph-mon-a
2019-12-23 16:38:31.971737 I | op-mon: waiting for mon quorum with [a b c]
2019-12-23 16:38:32.032016 I | op-mon: mons running: [a b c]
2019-12-23 16:38:32.032425 I | exec: Running command: ceph quorum_status --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/200640463
2019-12-23 16:38:33.431621 I | op-mon: Monitors in quorum: [a b c]
2019-12-23 16:38:33.455364 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed
2019-12-23 16:38:33.462973 I | op-k8sutil: updating deployment rook-ceph-mon-b
2019-12-23 16:38:35.547302 I | op-k8sutil: finished waiting for updated deployment rook-ceph-mon-b
2019-12-23 16:38:35.547332 I | op-mon: waiting for mon quorum with [a b c]
2019-12-23 16:38:35.596664 I | op-mon: mons running: [a b c]
2019-12-23 16:38:35.596850 I | exec: Running command: ceph quorum_status --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/761644770
2019-12-23 16:38:37.064507 I | op-mon: Monitors in quorum: [a b c]
2019-12-23 16:38:37.075576 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed
2019-12-23 16:38:37.127957 I | op-k8sutil: updating deployment rook-ceph-mon-c
2019-12-23 16:38:39.187452 I | op-k8sutil: finished waiting for updated deployment rook-ceph-mon-c
2019-12-23 16:38:39.187523 I | op-mon: waiting for mon quorum with [a b c]
2019-12-23 16:38:39.249670 I | op-mon: mons running: [a b c]
2019-12-23 16:38:39.250453 I | exec: Running command: ceph quorum_status --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/587191001
2019-12-23 16:38:40.727562 I | op-mon: Monitors in quorum: [a b c]
2019-12-23 16:38:40.727627 I | op-mon: mons created: 3
2019-12-23 16:38:40.728017 I | exec: Running command: ceph versions --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/126695524
2019-12-23 16:38:42.120150 I | op-mon: waiting for mon quorum with [a b c]
2019-12-23 16:38:42.169495 I | op-mon: mons running: [a b c]
2019-12-23 16:38:42.169639 I | exec: Running command: ceph quorum_status --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/439857267
2019-12-23 16:38:43.638056 I | op-mon: Monitors in quorum: [a b c]
2019-12-23 16:38:43.638793 I | exec: Running command: ceph auth get-or-create-key client.csi-rbd-provisioner mon profile rbd mgr allow rw osd profile rbd --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/241210166
2019-12-23 16:38:44.928039 I | exec: Running command: ceph auth get-or-create-key client.csi-rbd-node mon profile rbd osd profile rbd --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/275735837
2019-12-23 16:38:46.122746 I | exec: Running command: ceph auth get-or-create-key client.csi-cephfs-provisioner mon allow r mgr allow rw osd allow rw tag cephfs metadata=* --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/786835416
2019-12-23 16:38:47.539711 I | exec: Running command: ceph auth get-or-create-key client.csi-cephfs-node mon allow r mgr allow rw osd allow rw tag cephfs *=* mds allow rw --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/907479383
2019-12-23 16:38:48.931422 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph"
2019-12-23 16:38:48.931726 I | exec: Running command: ceph auth get-or-create-key client.crash mon allow profile crash mgr allow profile crash --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/315758282
2019-12-23 16:38:50.363749 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph"
2019-12-23 16:38:50.363889 I | exec: Running command: ceph version --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/535336865
2019-12-23 16:38:51.752432 I | exec: Running command: ceph versions --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/449249932
2019-12-23 16:38:52.967290 I | exec: Running command: ceph mon enable-msgr2 --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/425495163
2019-12-23 16:38:54.247948 I | cephclient: successfully enabled msgr2 protocol
2019-12-23 16:38:54.248005 I | op-mgr: start running mgr
2019-12-23 16:38:54.248355 I | exec: Running command: ceph auth get-or-create-key mgr.a mon allow * mds allow * osd allow * --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/322053022
2019-12-23 16:38:55.528126 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed
2019-12-23 16:38:55.539777 I | op-k8sutil: updating deployment rook-ceph-mgr-a
2019-12-23 16:38:57.614371 I | op-k8sutil: finished waiting for updated deployment rook-ceph-mgr-a
2019-12-23 16:38:57.717673 I | op-mgr: dashboard service already exists
2019-12-23 16:38:57.726049 I | exec: Running command: ceph mgr module enable prometheus --force --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/914471040
2019-12-23 16:38:57.726188 I | exec: Running command: ceph mgr module enable crash --force --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/576754655
2019-12-23 16:38:57.726678 I | exec: Running command: ceph mgr module enable pg_autoscaler --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/876081586
2019-12-23 16:38:57.727459 I | exec: Running command: ceph config get mgr.a mgr/dashboard/server_addr --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/268299113
2019-12-23 16:38:57.727785 I | exec: Running command: ceph mgr module enable dashboard --force --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/070734949
2019-12-23 16:38:57.728143 I | exec: Running command: ceph mgr module enable rook --force --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/895532468
2019-12-23 16:39:04.538329 I | exec: Running command: ceph config rm mgr.a mgr/dashboard/server_addr --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/075648387
2019-12-23 16:39:06.024579 I | exec: module 'rook' is already enabled
2019-12-23 16:39:06.024972 I | exec: Running command: ceph mgr module enable orchestrator_cli --force --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/089058566
2019-12-23 16:39:06.041007 I | exec: module 'pg_autoscaler' is already enabled
2019-12-23 16:39:06.041305 I | exec: Running command: ceph config set global osd_pool_default_pg_autoscale_mode on --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/470289069
2019-12-23 16:39:06.127021 I | exec: module 'prometheus' is already enabled
2019-12-23 16:39:06.127363 I | op-mgr: successful modules: prometheus
2019-12-23 16:39:06.918619 I | exec: module 'crash' is already enabled (always-on)
2019-12-23 16:39:06.919475 I | op-mgr: successful modules: crash
2019-12-23 16:39:07.030555 I | exec: module 'dashboard' is already enabled
2019-12-23 16:39:10.129790 I | exec: Running command: ceph config get mgr.a mgr/dashboard/a/server_addr --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/805712424
2019-12-23 16:39:10.827605 I | exec: module 'orchestrator_cli' is already enabled (always-on)
2019-12-23 16:39:10.827864 I | exec: Running command: ceph orchestrator set backend rook --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/178483047
2019-12-23 16:39:11.319042 I | exec: Running command: ceph config set global mon_pg_warn_min_per_osd 0 --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/324880282
2019-12-23 16:39:12.042398 I | op-mgr: the dashboard secret was already generated
2019-12-23 16:39:12.042792 I | op-mgr: Running command: ceph dashboard set-login-credentials admin *******
2019-12-23 16:39:14.642135 I | exec: Running command: ceph config rm mgr.a mgr/dashboard/a/server_addr --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/460341212
2019-12-23 16:39:16.318455 I | op-mgr: successful modules: mgr module(s) from the spec
2019-12-23 16:39:17.535826 I | op-mgr: successful modules: orchestrator modules
2019-12-23 16:39:18.227964 I | exec: Running command: ceph config get mgr.a mgr/dashboard/url_prefix --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/908432267
2019-12-23 16:39:18.836941 I | exec: Running command: ceph config get mgr.a mgr/prometheus/server_addr --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/406566766
2019-12-23 16:39:21.418724 I | exec: Running command: ceph config rm mgr.a mgr/dashboard/url_prefix --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/951653365
2019-12-23 16:39:21.818506 I | exec: Running command: ceph config rm mgr.a mgr/prometheus/server_addr --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/007539920
2019-12-23 16:39:24.538649 I | exec: Running command: ceph config get mgr.a mgr/dashboard/ssl --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/339303919
2019-12-23 16:39:24.839240 I | exec: Running command: ceph config get mgr.a mgr/prometheus/a/server_addr --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/468722818
2019-12-23 16:39:27.441750 I | exec: Running command: ceph config set mgr.a mgr/dashboard/ssl false --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/247382009
2019-12-23 16:39:28.219045 I | exec: Running command: ceph config rm mgr.a mgr/prometheus/a/server_addr --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/066432772
2019-12-23 16:39:29.941558 I | exec: Running command: ceph config get mgr.a mgr/dashboard/server_port --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/795714707
2019-12-23 16:39:31.248571 I | op-mgr: successful modules: http bind settings
2019-12-23 16:39:31.783750 I | exec: Running command: ceph config set mgr.a mgr/dashboard/server_port 7000 --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/561366230
2019-12-23 16:39:32.959623 I | op-mgr: dashboard config has changed. restarting the dashboard module.
2019-12-23 16:39:32.959660 I | op-mgr: restarting the mgr module
2019-12-23 16:39:32.959912 I | exec: Running command: ceph mgr module disable dashboard --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/556479037
2019-12-23 16:39:34.255074 I | exec: Running command: ceph mgr module enable dashboard --force --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/995575416
2019-12-23 16:39:37.384384 I | op-mgr: successful modules: dashboard
2019-12-23 16:39:37.488417 I | op-mgr: mgr metrics service already exists
2019-12-23 16:39:37.488468 I | op-mgr: starting monitoring deployment
W1223 16:39:37.491593       7 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2019-12-23 16:39:37.501979 E | op-mgr: failed to enable service monitor. service monitor could not be enabled: failed to create servicemonitor. servicemonitors.monitoring.coreos.com is forbidden: User "system:serviceaccount:rook-ceph:rook-ceph-system" cannot create resource "servicemonitors" in API group "monitoring.coreos.com" in the namespace "rook-ceph"
W1223 16:39:37.505484       7 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2019-12-23 16:39:37.515225 E | op-mgr: failed to deploy prometheus rule. prometheus rule could not be deployed: failed to create prometheusRules. prometheusrules.monitoring.coreos.com is forbidden: User "system:serviceaccount:rook-ceph:rook-ceph-system" cannot create resource "prometheusrules" in API group "monitoring.coreos.com" in the namespace "rook-ceph"
2019-12-23 16:39:37.515288 I | op-osd: start running osds in namespace rook-ceph
2019-12-23 16:39:37.515298 I | op-osd: start provisioning the osds on pvcs, if needed
2019-12-23 16:39:37.515304 I | op-osd: no volume sources defined to configure OSDs on PVCs.
2019-12-23 16:39:37.515309 I | op-osd: start provisioning the osds on nodes, if needed
2019-12-23 16:39:37.540571 I | op-osd: 3 of the 3 storage nodes are valid
2019-12-23 16:39:37.580046 I | op-k8sutil: Removing previous job rook-ceph-osd-prepare-kubeworker1 to start a new one
2019-12-23 16:39:37.641988 I | op-k8sutil: batch job rook-ceph-osd-prepare-kubeworker1 still exists
2019-12-23 16:39:39.652246 I | op-k8sutil: batch job rook-ceph-osd-prepare-kubeworker1 deleted
2019-12-23 16:39:39.681406 I | op-osd: osd provision job started for node kubeworker1
2019-12-23 16:39:39.744939 I | op-k8sutil: Removing previous job rook-ceph-osd-prepare-kubeworker2 to start a new one
2019-12-23 16:39:39.825555 I | op-k8sutil: batch job rook-ceph-osd-prepare-kubeworker2 still exists
2019-12-23 16:39:41.872201 I | op-k8sutil: batch job rook-ceph-osd-prepare-kubeworker2 deleted
2019-12-23 16:39:41.900466 I | op-osd: osd provision job started for node kubeworker2
2019-12-23 16:39:41.974350 I | op-k8sutil: Removing previous job rook-ceph-osd-prepare-kubeworker3 to start a new one
2019-12-23 16:39:42.087557 I | op-k8sutil: batch job rook-ceph-osd-prepare-kubeworker3 still exists
2019-12-23 16:39:44.099076 I | op-k8sutil: batch job rook-ceph-osd-prepare-kubeworker3 deleted
2019-12-23 16:39:44.122846 I | op-osd: osd provision job started for node kubeworker3
2019-12-23 16:39:44.122893 I | op-osd: start osds after provisioning is completed, if needed
2019-12-23 16:39:44.132595 I | op-osd: osd orchestration status for node kubeworker1 is starting
2019-12-23 16:39:44.132662 I | op-osd: osd orchestration status for node kubeworker2 is starting
2019-12-23 16:39:44.133225 I | op-osd: osd orchestration status for node kubeworker3 is starting
2019-12-23 16:39:44.133289 I | op-osd: 0/3 node(s) completed osd provisioning, resource version 12381
2019-12-23 16:39:47.784203 I | op-osd: osd orchestration status for node kubeworker1 is computingDiff
2019-12-23 16:39:47.950634 I | op-osd: osd orchestration status for node kubeworker1 is orchestrating
2019-12-23 16:39:49.290573 I | op-osd: osd orchestration status for node kubeworker2 is computingDiff
2019-12-23 16:39:49.454561 I | op-osd: osd orchestration status for node kubeworker1 is completed
2019-12-23 16:39:49.454890 I | op-osd: starting 1 osd daemons on node kubeworker1
2019-12-23 16:39:49.455580 I | exec: Running command: ceph auth get-or-create-key osd.1 osd allow * mon allow profile osd --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/426168183
2019-12-23 16:39:50.763306 I | op-osd: deployment for osd 1 already exists. updating if needed
2019-12-23 16:39:50.816702 I | op-k8sutil: updating deployment rook-ceph-osd-1
2019-12-23 16:39:52.870399 I | op-k8sutil: finished waiting for updated deployment rook-ceph-osd-1
2019-12-23 16:39:52.870711 I | op-osd: started deployment for osd 1 (dir=true, type=)
2019-12-23 16:39:53.396006 I | op-osd: osd orchestration status for node kubeworker2 is orchestrating
2019-12-23 16:39:53.397258 I | op-osd: osd orchestration status for node kubeworker2 is completed
2019-12-23 16:39:53.397529 I | op-osd: starting 1 osd daemons on node kubeworker2
2019-12-23 16:39:53.397929 I | exec: Running command: ceph auth get-or-create-key osd.2 osd allow * mon allow profile osd --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/759703658
2019-12-23 16:39:54.726099 I | op-osd: deployment for osd 2 already exists. updating if needed
2019-12-23 16:39:54.802656 I | op-k8sutil: updating deployment rook-ceph-osd-2
2019-12-23 16:39:56.867486 I | op-k8sutil: finished waiting for updated deployment rook-ceph-osd-2
2019-12-23 16:39:56.867556 I | op-osd: started deployment for osd 2 (dir=true, type=)
2019-12-23 16:39:56.898903 I | op-osd: osd orchestration status for node kubeworker3 is computingDiff
2019-12-23 16:39:56.899346 I | op-osd: osd orchestration status for node kubeworker3 is orchestrating
2019-12-23 16:39:56.899897 I | op-osd: osd orchestration status for node kubeworker3 is completed
2019-12-23 16:39:56.899921 I | op-osd: starting 1 osd daemons on node kubeworker3
2019-12-23 16:39:56.900045 I | exec: Running command: ceph auth get-or-create-key osd.0 osd allow * mon allow profile osd --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/999111873
2019-12-23 16:39:58.280421 I | op-osd: deployment for osd 0 already exists. updating if needed
2019-12-23 16:39:58.329283 I | op-k8sutil: updating deployment rook-ceph-osd-0
2019-12-23 16:40:00.374179 I | op-k8sutil: finished waiting for updated deployment rook-ceph-osd-0
2019-12-23 16:40:00.374214 I | op-osd: started deployment for osd 0 (dir=true, type=)
2019-12-23 16:40:00.396406 I | op-osd: 3/3 node(s) completed osd provisioning
2019-12-23 16:40:00.396975 I | exec: Running command: ceph versions --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/007559980
2019-12-23 16:40:01.638305 I | exec: Running command: ceph osd require-osd-release nautilus --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/204437659
2019-12-23 16:40:02.834267 I | cephclient: successfully disallowed pre-nautilus osds and enabled all new nautilus-only functionality
2019-12-23 16:40:02.834695 I | op-osd: completed running osds in namespace rook-ceph
2019-12-23 16:40:02.834922 I | rbd-mirror: configure rbd-mirroring with 0 workers
2019-12-23 16:40:02.846456 I | rbd-mirror: no extra daemons to remove
2019-12-23 16:40:02.846497 I | op-cluster: Done creating rook instance in namespace rook-ceph
2019-12-23 16:40:02.846536 I | op-cluster: CephCluster "rook-ceph" status: "Created". 
2019-12-23 16:40:02.890222 I | op-client: start watching client resources in namespace "rook-ceph"
2019-12-23 16:40:02.890285 I | op-pool: start watching pools in namespace "rook-ceph"
2019-12-23 16:40:02.890306 I | op-object: start watching object store resources in namespace rook-ceph
2019-12-23 16:40:02.890319 I | op-object: start watching object store user resources in namespace rook-ceph
2019-12-23 16:40:02.890346 I | op-bucket-prov: Ceph Bucket Provisioner launched
2019-12-23 16:40:02.893164 I | op-file: start watching filesystem resource in namespace rook-ceph
2019-12-23 16:40:02.893212 I | op-nfs: start watching ceph nfs resource in namespace rook-ceph
2019-12-23 16:40:02.893248 I | op-cluster: ceph status check interval is 60s
I1223 16:40:02.895236       7 manager.go:98] objectbucket.io/provisioner-manager "level"=0 "msg"="starting provisioner"  "name"="ceph.rook.io/bucket"
2019-12-23 16:40:02.942749 I | op-cluster: finalizer already set on cluster rook-ceph
2019-12-23 16:40:03.080164 I | exec: Running command: ceph osd crush dump --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/476649861
2019-12-23 16:40:05.427985 I | exec: Running command: ceph osd crush dump --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/034597664
2019-12-23 16:40:06.663902 I | exec: Running command: ceph fs get ceph-fs --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/143102975
2019-12-23 16:40:08.892072 I | op-file: filesystem ceph-fs already exists
2019-12-23 16:40:08.892235 I | exec: Running command: ceph fs get ceph-fs --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/276131154
2019-12-23 16:40:10.126386 I | exec: Running command: ceph fs set ceph-fs max_mds 2 --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/045902473
2019-12-23 16:40:11.903783 I | exec: Running command: ceph fs get ceph-fs --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/194079316
2019-12-23 16:40:13.052550 I | exec: Running command: ceph fs set ceph-fs allow_standby_replay true --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/461733795
2019-12-23 16:40:16.004510 I | exec: Running command: ceph fs get ceph-fs --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/615310502
2019-12-23 16:40:17.351313 I | exec: Running command: ceph fs set ceph-fs max_mds 2 --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/351005133
2019-12-23 16:40:20.007706 I | op-file: start running mdses for filesystem ceph-fs
2019-12-23 16:40:20.008321 I | exec: Running command: ceph auth get-or-create-key mds.ceph-fs-a osd allow * mds allow mon allow profile mds --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/381234888
2019-12-23 16:40:21.419837 I | op-mds: deployment for mds rook-ceph-mds-ceph-fs-a already exists. updating if needed
2019-12-23 16:40:21.451551 I | exec: Running command: ceph versions --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/561836935
2019-12-23 16:40:22.742639 I | op-k8sutil: updating deployment rook-ceph-mds-ceph-fs-a
2019-12-23 16:40:24.796688 I | op-k8sutil: finished waiting for updated deployment rook-ceph-mds-ceph-fs-a
2019-12-23 16:40:24.797136 I | exec: Running command: ceph auth get-or-create-key mds.ceph-fs-b osd allow * mds allow mon allow profile mds --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/538170682
2019-12-23 16:40:26.104561 I | op-mds: deployment for mds rook-ceph-mds-ceph-fs-b already exists. updating if needed
2019-12-23 16:40:26.138677 I | exec: Running command: ceph versions --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/073934161
2019-12-23 16:40:27.500131 I | op-k8sutil: updating deployment rook-ceph-mds-ceph-fs-b
2019-12-23 16:40:29.633718 I | op-k8sutil: finished waiting for updated deployment rook-ceph-mds-ceph-fs-b
2019-12-23 16:40:29.634193 I | exec: Running command: ceph auth get-or-create-key mds.ceph-fs-c osd allow * mds allow mon allow profile mds --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/575310460
2019-12-23 16:40:30.846293 I | op-mds: deployment for mds rook-ceph-mds-ceph-fs-c already exists. updating if needed
2019-12-23 16:40:30.885348 I | exec: Running command: ceph versions --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/238151595
2019-12-23 16:40:32.140166 I | op-k8sutil: updating deployment rook-ceph-mds-ceph-fs-c
2019-12-23 16:40:34.192205 I | op-k8sutil: finished waiting for updated deployment rook-ceph-mds-ceph-fs-c
2019-12-23 16:40:34.193157 I | exec: Running command: ceph auth get-or-create-key mds.ceph-fs-d osd allow * mds allow mon allow profile mds --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/996019982
2019-12-23 16:40:35.746539 I | op-mds: deployment for mds rook-ceph-mds-ceph-fs-d already exists. updating if needed
2019-12-23 16:40:35.790639 I | exec: Running command: ceph versions --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/683014421
2019-12-23 16:40:36.957392 I | op-k8sutil: updating deployment rook-ceph-mds-ceph-fs-d
2019-12-23 16:40:39.007762 I | op-k8sutil: finished waiting for updated deployment rook-ceph-mds-ceph-fs-d
W1223 16:43:23.074437       7 reflector.go:289] github.com/rook/rook/pkg/operator/ceph/cluster/controller.go:178: watch of *v1.ConfigMap ended with: too old resource version: 11730 (13229)
W1223 16:48:27.086130       7 reflector.go:289] github.com/rook/rook/pkg/operator/ceph/cluster/controller.go:178: watch of *v1.ConfigMap ended with: too old resource version: 13702 (14845)
I1223 16:49:57.032213       7 leaderelection.go:263] failed to renew lease rook-ceph/ceph.rook.io-block: failed to tryAcquireOrRenew context deadline exceeded
F1223 16:49:57.032339       7 controller.go:847] leaderelection lost

new-operator.log

2019-12-23 16:50:01.150638 I | rookcmd: starting Rook v1.2.0 with arguments '/usr/local/bin/rook ceph operator'
2019-12-23 16:50:01.150803 I | rookcmd: flag values: --add_dir_header=false, --alsologtostderr=false, --csi-attacher-image=quay.io/k8scsi/csi-attacher:v1.2.0, --csi-ceph-image=quay.io/cephcsi/cephcsi:v1.2.2, --csi-cephfs-plugin-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin.yaml, --csi-cephfs-provisioner-dep-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin-provisioner-dep.yaml, --csi-cephfs-provisioner-sts-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin-provisioner-sts.yaml, --csi-driver-name-prefix=, --csi-enable-cephfs=true, --csi-enable-grpc-metrics=false, --csi-enable-rbd=false, --csi-kubelet-dir-path=/var/lib/kubelet, --csi-provisioner-image=quay.io/k8scsi/csi-provisioner:v1.4.0, --csi-rbd-plugin-template-path=/etc/ceph-csi/rbd/csi-rbdplugin.yaml, --csi-rbd-provisioner-dep-template-path=/etc/ceph-csi/rbd/csi-rbdplugin-provisioner-dep.yaml, --csi-rbd-provisioner-sts-template-path=/etc/ceph-csi/rbd/csi-rbdplugin-provisioner-sts.yaml, --csi-registrar-image=quay.io/k8scsi/csi-node-driver-registrar:v1.1.0, --csi-snapshotter-image=quay.io/k8scsi/csi-snapshotter:v1.2.2, --enable-discovery-daemon=true, --enable-flex-driver=false, --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-flush-frequency=5s, --log-level=INFO, --log_backtrace_at=:0, --log_dir=, --log_file=, --log_file_max_size=1800, --logtostderr=true, --master=, --mon-healthcheck-interval=45s, --mon-out-timeout=10m0s, --operator-image=, --service-account=, --skip_headers=false, --skip_log_headers=false, --stderrthreshold=2, --v=0, --vmodule=
2019-12-23 16:50:01.150816 I | cephcmd: starting operator
2019-12-23 16:50:01.270946 I | op-discover: rook-discover daemonset already exists, updating ...
2019-12-23 16:50:01.297171 I | operator: rook-provisioner ceph.rook.io/block started using ceph.rook.io flex vendor dir
2019-12-23 16:50:01.299553 I | operator: rook-provisioner rook.io/block started using rook.io flex vendor dir
2019-12-23 16:50:01.299608 I | operator: Watching all namespaces for cluster CRDs
2019-12-23 16:50:01.299637 I | op-cluster: start watching clusters in all namespaces
2019-12-23 16:50:01.299765 I | op-cluster: Enabling hotplug orchestration: ROOK_DISABLE_DEVICE_HOTPLUG=false
I1223 16:50:01.301328       6 leaderelection.go:217] attempting to acquire leader lease  rook-ceph/ceph.rook.io-block...
2019-12-23 16:50:01.302963 I | operator: setting up the controller-runtime manager
I1223 16:50:01.304553       6 leaderelection.go:217] attempting to acquire leader lease  rook-ceph/rook.io-block...
2019-12-23 16:50:01.326576 I | op-cluster: starting cluster in namespace rook-ceph
2019-12-23 16:50:01.764376 I | ceph-csi: CSIDriver CRD already had been registered for "rook-ceph.rbd.csi.ceph.com"
2019-12-23 16:50:01.802492 I | ceph-csi: CSIDriver CRD already had been registered for "rook-ceph.cephfs.csi.ceph.com"
2019-12-23 16:50:01.802714 I | operator: successfully started Ceph CSI driver(s)
2019-12-23 16:50:01.912521 I | operator: starting the controller-runtime manager
2019-12-23 16:50:07.803419 I | op-cluster: detecting the ceph image version for image ceph/ceph:v14.2.5-20191210...
I1223 16:50:19.939726       6 leaderelection.go:227] successfully acquired lease rook-ceph/rook.io-block
I1223 16:50:19.940293       6 controller.go:769] Starting provisioner controller rook.io/block_rook-ceph-operator-575b89745f-wzjjc_42c01c50-25a4-11ea-bcc9-cafea86acf8a!
I1223 16:50:19.941320       6 event.go:209] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"rook-ceph", Name:"rook.io-block", UID:"a2fbe584-eddf-4161-a215-08fd85b60825", APIVersion:"v1", ResourceVersion:"15924", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' rook-ceph-operator-575b89745f-wzjjc_42c01c50-25a4-11ea-bcc9-cafea86acf8a became leader
I1223 16:50:20.040991       6 controller.go:818] Started provisioner controller rook.io/block_rook-ceph-operator-575b89745f-wzjjc_42c01c50-25a4-11ea-bcc9-cafea86acf8a!
I1223 16:50:23.977414       6 leaderelection.go:227] successfully acquired lease rook-ceph/ceph.rook.io-block
I1223 16:50:23.978170       6 event.go:209] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"rook-ceph", Name:"ceph.rook.io-block", UID:"39e9eac3-7e86-43f7-91b7-6235ca9dcba8", APIVersion:"v1", ResourceVersion:"15953", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' rook-ceph-operator-575b89745f-wzjjc_42bfe755-25a4-11ea-bcc9-cafea86acf8a became leader
I1223 16:50:23.978269       6 controller.go:769] Starting provisioner controller ceph.rook.io/block_rook-ceph-operator-575b89745f-wzjjc_42bfe755-25a4-11ea-bcc9-cafea86acf8a!
I1223 16:50:24.078594       6 controller.go:818] Started provisioner controller ceph.rook.io/block_rook-ceph-operator-575b89745f-wzjjc_42bfe755-25a4-11ea-bcc9-cafea86acf8a!
2019-12-23 16:50:25.741626 I | op-cluster: Detected ceph image version: "14.2.5-0 nautilus"
2019-12-23 16:50:25.791234 I | op-mon: parsing mon endpoints: b=10.96.116.199:6789,c=10.96.253.5:6789,a=10.96.122.158:6789
2019-12-23 16:50:25.791428 I | op-mon: loaded: maxMonID=2, mons=map[b:0xc000f50a20 c:0xc000f50a60 a:0xc000f50aa0], mapping=&{Node:map[c:0xc000c4ced0 a:0xc000c4ce10 b:0xc000c4cea0]}
2019-12-23 16:50:25.792227 I | cephconfig: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2019-12-23 16:50:25.792360 I | cephconfig: generated admin config in /var/lib/rook/rook-ceph
2019-12-23 16:50:25.792550 I | exec: Running command: ceph versions --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/080090546
2019-12-23 16:50:27.141617 I | op-cluster: CephCluster "rook-ceph" status: "Creating". 
2019-12-23 16:50:27.293728 I | op-mon: start running mons
2019-12-23 16:50:27.342961 I | op-mon: parsing mon endpoints: b=10.96.116.199:6789,c=10.96.253.5:6789,a=10.96.122.158:6789
2019-12-23 16:50:27.343313 I | op-mon: loaded: maxMonID=2, mons=map[b:0xc00128e280 c:0xc00128e2c0 a:0xc00128e300], mapping=&{Node:map[a:0xc0012a0210 b:0xc0012a0240 c:0xc0012a0270]}
2019-12-23 16:50:27.394154 I | op-mon: saved mon endpoints to config map map[mapping:{"node":{"a":{"Name":"kubeworker2","Hostname":"kubeworker2","Address":"88.99.121.47"},"b":{"Name":"kubeworker1","Hostname":"kubeworker1","Address":"116.202.30.255"},"c":{"Name":"kubeworker3","Hostname":"kubeworker3","Address":"116.202.30.249"}}} csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.96.122.158:6789","10.96.116.199:6789","10.96.253.5:6789"]}] data:c=10.96.253.5:6789,a=10.96.122.158:6789,b=10.96.116.199:6789 maxMonId:2]
2019-12-23 16:50:27.459500 I | cephconfig: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2019-12-23 16:50:27.518139 I | cephconfig: generated admin config in /var/lib/rook/rook-ceph
2019-12-23 16:50:28.478676 I | op-mon: targeting the mon count 3
2019-12-23 16:50:28.479321 I | exec: Running command: ceph config set global mon_allow_pool_delete true --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/614118249
2019-12-23 16:50:29.697634 I | exec: Running command: ceph config set global rbd_default_features 3 --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/925471668
2019-12-23 16:50:30.926332 I | op-mon: checking for basic quorum with existing mons
2019-12-23 16:50:31.024291 I | op-mon: mon "b" endpoint are [v2:10.96.116.199:3300,v1:10.96.116.199:6789]
2019-12-23 16:50:31.326483 I | op-mon: mon "c" endpoint are [v2:10.96.253.5:3300,v1:10.96.253.5:6789]
2019-12-23 16:50:31.461254 I | op-mon: mon "a" endpoint are [v2:10.96.122.158:3300,v1:10.96.122.158:6789]
2019-12-23 16:50:31.706439 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.96.116.199:6789","10.96.253.5:6789","10.96.122.158:6789"]}] data:b=10.96.116.199:6789,c=10.96.253.5:6789,a=10.96.122.158:6789 maxMonId:2 mapping:{"node":{"a":{"Name":"kubeworker2","Hostname":"kubeworker2","Address":"88.99.121.47"},"b":{"Name":"kubeworker1","Hostname":"kubeworker1","Address":"116.202.30.255"},"c":{"Name":"kubeworker3","Hostname":"kubeworker3","Address":"116.202.30.249"}}}]
2019-12-23 16:50:32.497814 I | cephconfig: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2019-12-23 16:50:32.498173 I | cephconfig: generated admin config in /var/lib/rook/rook-ceph
2019-12-23 16:50:32.879877 I | cephconfig: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2019-12-23 16:50:32.880294 I | cephconfig: generated admin config in /var/lib/rook/rook-ceph
2019-12-23 16:50:32.909360 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed
2019-12-23 16:50:32.918089 I | op-k8sutil: updating deployment rook-ceph-mon-b
2019-12-23 16:50:34.983023 I | op-k8sutil: finished waiting for updated deployment rook-ceph-mon-b
2019-12-23 16:50:34.983203 I | op-mon: waiting for mon quorum with [b c a]
2019-12-23 16:50:35.041481 I | op-mon: mons running: [b c a]
2019-12-23 16:50:35.041985 I | exec: Running command: ceph quorum_status --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/785994115
2019-12-23 16:50:36.573796 I | op-mon: Monitors in quorum: [a b c]
2019-12-23 16:50:36.630928 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed
2019-12-23 16:50:36.643157 I | op-k8sutil: updating deployment rook-ceph-mon-c
2019-12-23 16:50:38.704073 I | op-k8sutil: finished waiting for updated deployment rook-ceph-mon-c
2019-12-23 16:50:38.704142 I | op-mon: waiting for mon quorum with [b c a]
2019-12-23 16:50:38.777777 I | op-mon: mons running: [b c a]
2019-12-23 16:50:38.778192 I | exec: Running command: ceph quorum_status --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/908805382
2019-12-23 16:50:40.118595 I | op-mon: Monitors in quorum: [a b c]
2019-12-23 16:50:40.130201 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed
2019-12-23 16:50:40.149000 I | op-k8sutil: updating deployment rook-ceph-mon-a
2019-12-23 16:50:42.230876 I | op-k8sutil: finished waiting for updated deployment rook-ceph-mon-a
2019-12-23 16:50:42.230996 I | op-mon: waiting for mon quorum with [b c a]
2019-12-23 16:50:42.291275 I | op-mon: mons running: [b c a]
2019-12-23 16:50:42.291856 I | exec: Running command: ceph quorum_status --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/695554221
2019-12-23 16:50:43.769156 I | op-mon: Monitors in quorum: [a b c]
2019-12-23 16:50:43.769203 I | op-mon: mons created: 3
2019-12-23 16:50:43.769313 I | exec: Running command: ceph versions --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/673935912
2019-12-23 16:50:45.027953 I | op-mon: waiting for mon quorum with [b c a]
2019-12-23 16:50:45.070896 I | op-mon: mons running: [b c a]
2019-12-23 16:50:45.071058 I | exec: Running command: ceph quorum_status --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/694153575
2019-12-23 16:50:46.635391 I | op-mon: Monitors in quorum: [a b c]
2019-12-23 16:50:46.635731 I | exec: Running command: ceph auth get-or-create-key client.csi-rbd-provisioner mon profile rbd mgr allow rw osd profile rbd --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/596393882
2019-12-23 16:50:48.034415 I | exec: Running command: ceph auth get-or-create-key client.csi-rbd-node mon profile rbd osd profile rbd --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/569346609
2019-12-23 16:50:49.336490 I | exec: Running command: ceph auth get-or-create-key client.csi-cephfs-provisioner mon allow r mgr allow rw osd allow rw tag cephfs metadata=* --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/849420252
2019-12-23 16:50:50.648652 I | exec: Running command: ceph auth get-or-create-key client.csi-cephfs-node mon allow r mgr allow rw osd allow rw tag cephfs *=* mds allow rw --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/708392331
2019-12-23 16:50:52.017705 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph"
2019-12-23 16:50:52.018286 I | exec: Running command: ceph auth get-or-create-key client.crash mon allow profile crash mgr allow profile crash --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/566686062
2019-12-23 16:50:53.331079 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph"
2019-12-23 16:50:53.331626 I | exec: Running command: ceph version --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/327274485
2019-12-23 16:50:54.566348 I | exec: Running command: ceph versions --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/104093392
2019-12-23 16:50:55.923949 I | exec: Running command: ceph mon enable-msgr2 --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/331068399
2019-12-23 16:50:57.250183 I | cephclient: successfully enabled msgr2 protocol
2019-12-23 16:50:57.250311 I | op-mgr: start running mgr
2019-12-23 16:50:57.250479 I | exec: Running command: ceph auth get-or-create-key mgr.a mon allow * mds allow * osd allow * --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/106402434
2019-12-23 16:50:58.591031 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed
2019-12-23 16:50:58.599922 I | op-k8sutil: updating deployment rook-ceph-mgr-a
2019-12-23 16:51:00.660335 I | op-k8sutil: finished waiting for updated deployment rook-ceph-mgr-a
2019-12-23 16:51:00.772527 I | op-mgr: dashboard service already exists
2019-12-23 16:51:00.783179 I | exec: Running command: ceph mgr module enable dashboard --force --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/253940217
2019-12-23 16:51:00.783312 I | exec: Running command: ceph mgr module enable prometheus --force --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/855704836
2019-12-23 16:51:00.783573 I | exec: Running command: ceph config get mgr.a mgr/dashboard/server_addr --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/475708563
2019-12-23 16:51:00.783811 I | exec: Running command: ceph mgr module enable rook --force --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/423140566
2019-12-23 16:51:00.784564 I | exec: Running command: ceph mgr module enable crash --force --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/672621629
2019-12-23 16:51:00.785258 I | exec: Running command: ceph mgr module enable pg_autoscaler --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/470563960
2019-12-23 16:51:07.018757 I | exec: module 'pg_autoscaler' is already enabled
2019-12-23 16:51:07.019241 I | exec: Running command: ceph config set global osd_pool_default_pg_autoscale_mode on --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/102120823
2019-12-23 16:51:09.132021 I | exec: module 'crash' is already enabled (always-on)
2019-12-23 16:51:09.132313 I | op-mgr: successful modules: crash
2019-12-23 16:51:10.227102 I | exec: Running command: ceph config rm mgr.a mgr/dashboard/server_addr --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/964669034
2019-12-23 16:51:11.329533 I | exec: module 'prometheus' is already enabled
2019-12-23 16:51:11.330485 I | op-mgr: successful modules: prometheus
2019-12-23 16:51:12.040462 I | exec: Running command: ceph config set global mon_pg_warn_min_per_osd 0 --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/933441729
2019-12-23 16:51:12.326413 I | exec: module 'rook' is already enabled
2019-12-23 16:51:12.327219 I | exec: Running command: ceph mgr module enable orchestrator_cli --force --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/537336108
2019-12-23 16:51:12.422891 I | exec: module 'dashboard' is already enabled
2019-12-23 16:51:14.440373 I | exec: Running command: ceph config get mgr.a mgr/dashboard/a/server_addr --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/639550107
2019-12-23 16:51:16.758829 I | exec: module 'orchestrator_cli' is already enabled (always-on)
2019-12-23 16:51:16.759204 I | exec: Running command: ceph orchestrator set backend rook --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/856739134
2019-12-23 16:51:16.761396 I | op-mgr: successful modules: mgr module(s) from the spec
2019-12-23 16:51:17.435145 I | op-mgr: the dashboard secret was already generated
2019-12-23 16:51:17.435305 I | op-mgr: Running command: ceph dashboard set-login-credentials admin *******
2019-12-23 16:51:19.231450 I | exec: Running command: ceph config rm mgr.a mgr/dashboard/a/server_addr --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/827045152
2019-12-23 16:51:21.130636 I | op-mgr: successful modules: orchestrator modules
2019-12-23 16:51:22.419542 I | exec: Running command: ceph config get mgr.a mgr/dashboard/url_prefix --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/916955135
2019-12-23 16:51:22.427170 I | exec: Running command: ceph config get mgr.a mgr/prometheus/server_addr --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/656445778
2019-12-23 16:51:25.459266 I | exec: Running command: ceph config rm mgr.a mgr/prometheus/server_addr --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/886190729
2019-12-23 16:51:26.027552 I | exec: Running command: ceph config rm mgr.a mgr/dashboard/url_prefix --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/968152148
2019-12-23 16:51:28.037209 I | exec: Running command: ceph config get mgr.a mgr/prometheus/a/server_addr --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/623422371
2019-12-23 16:51:29.159169 I | exec: Running command: ceph config get mgr.a mgr/dashboard/ssl --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/217074342
2019-12-23 16:51:30.538728 I | exec: Running command: ceph config rm mgr.a mgr/prometheus/a/server_addr --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/343345101
2019-12-23 16:51:32.223378 I | exec: Running command: ceph config set mgr.a mgr/dashboard/ssl false --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/198405832
2019-12-23 16:51:33.068405 I | op-mgr: successful modules: http bind settings
2019-12-23 16:51:33.961277 I | exec: Running command: ceph config get mgr.a mgr/dashboard/server_port --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/769657223
2019-12-23 16:51:36.021603 I | exec: Running command: ceph config set mgr.a mgr/dashboard/server_port 7000 --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/268032826
2019-12-23 16:51:37.550856 I | op-mgr: dashboard config has changed. restarting the dashboard module.
2019-12-23 16:51:37.550901 I | op-mgr: restarting the mgr module
2019-12-23 16:51:37.551014 I | exec: Running command: ceph mgr module disable dashboard --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/244329809
2019-12-23 16:51:39.205430 I | exec: Running command: ceph mgr module enable dashboard --force --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/175505020
2019-12-23 16:51:41.414118 I | op-mgr: successful modules: dashboard
2019-12-23 16:51:42.383866 I | op-mgr: mgr metrics service already exists
2019-12-23 16:51:42.384204 I | op-mgr: starting monitoring deployment
W1223 16:51:42.386298       6 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2019-12-23 16:51:42.392927 E | op-mgr: failed to enable service monitor. service monitor could not be enabled: failed to create servicemonitor. servicemonitors.monitoring.coreos.com is forbidden: User "system:serviceaccount:rook-ceph:rook-ceph-system" cannot create resource "servicemonitors" in API group "monitoring.coreos.com" in the namespace "rook-ceph"
W1223 16:51:42.399173       6 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2019-12-23 16:51:42.402969 E | op-mgr: failed to deploy prometheus rule. prometheus rule could not be deployed: failed to create prometheusRules. prometheusrules.monitoring.coreos.com is forbidden: User "system:serviceaccount:rook-ceph:rook-ceph-system" cannot create resource "prometheusrules" in API group "monitoring.coreos.com" in the namespace "rook-ceph"
2019-12-23 16:51:42.403259 I | op-osd: start running osds in namespace rook-ceph
2019-12-23 16:51:42.403385 I | op-osd: start provisioning the osds on pvcs, if needed
2019-12-23 16:51:42.403431 I | op-osd: no volume sources defined to configure OSDs on PVCs.
2019-12-23 16:51:42.403489 I | op-osd: start provisioning the osds on nodes, if needed
2019-12-23 16:51:42.460006 I | op-osd: 3 of the 3 storage nodes are valid
2019-12-23 16:51:42.500100 I | op-k8sutil: Removing previous job rook-ceph-osd-prepare-kubeworker1 to start a new one
2019-12-23 16:51:42.572533 I | op-k8sutil: batch job rook-ceph-osd-prepare-kubeworker1 still exists
2019-12-23 16:51:44.583943 I | op-k8sutil: batch job rook-ceph-osd-prepare-kubeworker1 deleted
2019-12-23 16:51:44.616213 I | op-osd: osd provision job started for node kubeworker1
2019-12-23 16:51:44.641392 I | op-k8sutil: Removing previous job rook-ceph-osd-prepare-kubeworker2 to start a new one
2019-12-23 16:51:44.752985 I | op-k8sutil: batch job rook-ceph-osd-prepare-kubeworker2 still exists
2019-12-23 16:51:47.218426 I | op-k8sutil: batch job rook-ceph-osd-prepare-kubeworker2 deleted
2019-12-23 16:51:47.326731 I | op-osd: osd provision job started for node kubeworker2
2019-12-23 16:51:48.044175 I | op-k8sutil: Removing previous job rook-ceph-osd-prepare-kubeworker3 to start a new one
2019-12-23 16:51:48.830793 I | op-k8sutil: batch job rook-ceph-osd-prepare-kubeworker3 still exists
2019-12-23 16:51:50.841152 I | op-k8sutil: batch job rook-ceph-osd-prepare-kubeworker3 deleted
2019-12-23 16:51:51.220418 I | op-osd: osd provision job started for node kubeworker3
2019-12-23 16:51:51.220539 I | op-osd: start osds after provisioning is completed, if needed
2019-12-23 16:51:51.240136 I | op-osd: osd orchestration status for node kubeworker1 is starting
2019-12-23 16:51:51.240199 I | op-osd: osd orchestration status for node kubeworker2 is starting
2019-12-23 16:51:51.240220 I | op-osd: osd orchestration status for node kubeworker3 is starting
2019-12-23 16:51:51.240250 I | op-osd: 0/3 node(s) completed osd provisioning, resource version 16492
2019-12-23 16:51:55.506834 I | op-osd: osd orchestration status for node kubeworker2 is computingDiff
2019-12-23 16:51:55.648806 I | op-osd: osd orchestration status for node kubeworker2 is orchestrating
2019-12-23 16:51:57.228795 I | op-osd: osd orchestration status for node kubeworker2 is completed
2019-12-23 16:51:57.229334 I | op-osd: starting 1 osd daemons on node kubeworker2
2019-12-23 16:51:57.230015 I | exec: Running command: ceph auth get-or-create-key osd.2 osd allow * mon allow profile osd --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/256281003
2019-12-23 16:51:58.427017 I | op-osd: deployment for osd 2 already exists. updating if needed
2019-12-23 16:51:58.499088 I | op-k8sutil: updating deployment rook-ceph-osd-2
2019-12-23 16:52:00.610197 I | op-k8sutil: finished waiting for updated deployment rook-ceph-osd-2
2019-12-23 16:52:00.610259 I | op-osd: started deployment for osd 2 (dir=true, type=)
2019-12-23 16:52:00.686549 I | op-osd: osd orchestration status for node kubeworker3 is computingDiff
2019-12-23 16:52:00.687292 I | op-osd: osd orchestration status for node kubeworker3 is orchestrating
2019-12-23 16:52:00.687883 I | op-osd: osd orchestration status for node kubeworker3 is completed
2019-12-23 16:52:00.688087 I | op-osd: starting 1 osd daemons on node kubeworker3
2019-12-23 16:52:00.688322 I | exec: Running command: ceph auth get-or-create-key osd.0 osd allow * mon allow profile osd --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/408649486
2019-12-23 16:52:02.020495 I | op-osd: deployment for osd 0 already exists. updating if needed
2019-12-23 16:52:02.062946 I | op-k8sutil: updating deployment rook-ceph-osd-0
2019-12-23 16:52:04.106216 I | op-k8sutil: finished waiting for updated deployment rook-ceph-osd-0
2019-12-23 16:52:04.106278 I | op-osd: started deployment for osd 0 (dir=true, type=)
2019-12-23 16:52:04.134961 I | op-osd: osd orchestration status for node kubeworker1 is computingDiff
2019-12-23 16:52:04.135454 I | op-osd: osd orchestration status for node kubeworker1 is orchestrating
2019-12-23 16:52:04.136346 I | op-osd: osd orchestration status for node kubeworker1 is completed
2019-12-23 16:52:04.136391 I | op-osd: starting 1 osd daemons on node kubeworker1
2019-12-23 16:52:04.136688 I | exec: Running command: ceph auth get-or-create-key osd.1 osd allow * mon allow profile osd --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/818052885
2019-12-23 16:52:05.469861 I | op-osd: deployment for osd 1 already exists. updating if needed
2019-12-23 16:52:05.523317 I | op-k8sutil: updating deployment rook-ceph-osd-1
2019-12-23 16:52:07.586756 I | op-k8sutil: finished waiting for updated deployment rook-ceph-osd-1
2019-12-23 16:52:07.586826 I | op-osd: started deployment for osd 1 (dir=true, type=)
2019-12-23 16:52:07.619333 I | op-osd: 3/3 node(s) completed osd provisioning
2019-12-23 16:52:07.620148 I | exec: Running command: ceph versions --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/650719600
2019-12-23 16:52:08.861439 I | exec: Running command: ceph osd require-osd-release nautilus --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/217684495
2019-12-23 16:52:10.134834 I | cephclient: successfully disallowed pre-nautilus osds and enabled all new nautilus-only functionality
2019-12-23 16:52:10.134889 I | op-osd: completed running osds in namespace rook-ceph
2019-12-23 16:52:10.134925 I | rbd-mirror: configure rbd-mirroring with 0 workers
2019-12-23 16:52:10.146855 I | rbd-mirror: no extra daemons to remove
2019-12-23 16:52:10.146903 I | op-cluster: Done creating rook instance in namespace rook-ceph
2019-12-23 16:52:10.146931 I | op-cluster: CephCluster "rook-ceph" status: "Created". 
2019-12-23 16:52:10.187490 I | op-client: start watching client resources in namespace "rook-ceph"
2019-12-23 16:52:10.187534 I | op-pool: start watching pools in namespace "rook-ceph"
2019-12-23 16:52:10.187543 I | op-object: start watching object store resources in namespace rook-ceph
2019-12-23 16:52:10.187552 I | op-object: start watching object store user resources in namespace rook-ceph
2019-12-23 16:52:10.187559 I | op-bucket-prov: Ceph Bucket Provisioner launched
2019-12-23 16:52:10.188634 I | op-file: start watching filesystem resource in namespace rook-ceph
2019-12-23 16:52:10.188660 I | op-nfs: start watching ceph nfs resource in namespace rook-ceph
2019-12-23 16:52:10.188691 I | op-cluster: ceph status check interval is 60s
I1223 16:52:10.195936       6 manager.go:98] objectbucket.io/provisioner-manager "level"=0 "msg"="starting provisioner"  "name"="ceph.rook.io/bucket"
2019-12-23 16:52:10.224724 I | op-cluster: finalizer already set on cluster rook-ceph
2019-12-23 16:52:10.363396 I | exec: Running command: ceph osd crush dump --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/545697049
2019-12-23 16:52:13.172318 I | exec: Running command: ceph osd crush dump --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/314863012
2019-12-23 16:52:14.546343 I | exec: Running command: ceph fs get ceph-fs --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/524192435
2019-12-23 16:52:15.768469 I | op-file: filesystem ceph-fs already exists
2019-12-23 16:52:15.768572 I | exec: Running command: ceph fs get ceph-fs --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/604689526
2019-12-23 16:52:17.034695 I | exec: Running command: ceph fs set ceph-fs max_mds 2 --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/556319069
2019-12-23 16:52:18.587248 I | exec: Running command: ceph fs get ceph-fs --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/178311960
2019-12-23 16:52:19.794376 I | exec: Running command: ceph fs set ceph-fs allow_standby_replay true --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/014133143
2019-12-23 16:52:21.847899 I | exec: Running command: ceph fs get ceph-fs --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/821910026
2019-12-23 16:52:23.127800 I | exec: Running command: ceph fs set ceph-fs max_mds 2 --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/816735713
2019-12-23 16:52:26.274893 I | op-file: start running mdses for filesystem ceph-fs
2019-12-23 16:52:26.275392 I | exec: Running command: ceph auth get-or-create-key mds.ceph-fs-a osd allow * mds allow mon allow profile mds --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/826851788
2019-12-23 16:52:27.539962 I | op-mds: deployment for mds rook-ceph-mds-ceph-fs-a already exists. updating if needed
2019-12-23 16:52:27.594031 I | exec: Running command: ceph versions --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/345789627
2019-12-23 16:52:28.678626 I | op-k8sutil: updating deployment rook-ceph-mds-ceph-fs-a
2019-12-23 16:52:30.748204 I | op-k8sutil: finished waiting for updated deployment rook-ceph-mds-ceph-fs-a
2019-12-23 16:52:30.748654 I | exec: Running command: ceph auth get-or-create-key mds.ceph-fs-b osd allow * mds allow mon allow profile mds --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/630482142
2019-12-23 16:52:32.306791 I | op-mds: deployment for mds rook-ceph-mds-ceph-fs-b already exists. updating if needed
2019-12-23 16:52:32.347022 I | exec: Running command: ceph versions --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/725910181
2019-12-23 16:52:33.678477 I | op-k8sutil: updating deployment rook-ceph-mds-ceph-fs-b
2019-12-23 16:52:35.792914 I | op-k8sutil: finished waiting for updated deployment rook-ceph-mds-ceph-fs-b
2019-12-23 16:52:35.793417 I | exec: Running command: ceph auth get-or-create-key mds.ceph-fs-c osd allow * mds allow mon allow profile mds --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/932984768
2019-12-23 16:52:37.193280 I | op-mds: deployment for mds rook-ceph-mds-ceph-fs-c already exists. updating if needed
2019-12-23 16:52:37.232384 I | exec: Running command: ceph versions --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/123538975
2019-12-23 16:52:38.457178 I | op-k8sutil: updating deployment rook-ceph-mds-ceph-fs-c
2019-12-23 16:52:40.531202 I | op-k8sutil: finished waiting for updated deployment rook-ceph-mds-ceph-fs-c
2019-12-23 16:52:40.531426 I | exec: Running command: ceph auth get-or-create-key mds.ceph-fs-d osd allow * mds allow mon allow profile mds --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/282068210
2019-12-23 16:52:41.897801 I | op-mds: deployment for mds rook-ceph-mds-ceph-fs-d already exists. updating if needed
2019-12-23 16:52:41.947509 I | exec: Running command: ceph versions --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/956224937
2019-12-23 16:52:43.238011 I | op-k8sutil: updating deployment rook-ceph-mds-ceph-fs-d
2019-12-23 16:52:45.291260 I | op-k8sutil: finished waiting for updated deployment rook-ceph-mds-ceph-fs-d
W1223 16:59:38.328588       6 reflector.go:289] github.com/rook/rook/pkg/operator/ceph/cluster/controller.go:178: watch of *v1.ConfigMap ended with: too old resource version: 15758 (18489)

Environment:

• OS (e.g. from /etc/os-release): Ubuntu 18.04.3 LTS
• Kernel (e.g. uname -a): 4.15.0-72-generic (x86_64)
• Cloud provider or hardware configuration: Bare-Metal
• Rook version (use rook version inside of a Rook Pod): 1.2.0
• Storage backend version (e.g. for ceph do ceph -v): v14.2.5-20191210
• Kubernetes version (use kubectl version): 1.17.0
• Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): normal Kubernetes
• Storage backend status (e.g. for Ceph use ceph health): HEALTH_OK