rook: During the update from 1.8 to 1.9 the operator started crashing

During the update, the operator started crashing, rook-ceph-crashcollector were updated to rook-version=v1.9.13, osd mon mds remained in rook-version=v1.8.10. Api server returns issues posted in the log below.

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior:

Expected behavior: Operator starts normally

How to reproduce it (minimal and precise):

Follow steps from https://rook.io/docs/rook/v1.9/ceph-upgrade.html#csi-version upgrade from 1.8 to 1.9 simple install With added env flag to operator ROOK_DISABLE_ADMISSION_CONTROLLER = true File(s) to submit: cluster.txt

Logs to submit:

  • Operator’s pod is crashing
2024-05-16 07:33:30.985433 D | ceph-spec: found 1 ceph clusters in namespace "rook-ceph-external"
2024-05-16 07:33:30.985829 D | ceph-spec: found 1 ceph clusters in namespace "rook-ceph-external"
2024-05-16 07:33:30.986214 D | ceph-spec: found 1 ceph clusters in namespace "rook-ceph-external"
2024-05-16 07:33:30.986589 D | ceph-spec: found 1 ceph clusters in namespace "rook-ceph-external"
2024-05-16 07:33:30.987023 D | ceph-cluster-controller: update event on CephCluster CR
2024-05-16 07:33:30.987448 D | ceph-cluster-controller: update event on CephCluster CR
2024-05-16 07:33:30.987858 D | ceph-cluster-controller: update event on CephCluster CR
2024-05-16 07:33:30.988263 D | ceph-cluster-controller: update event on CephCluster CR
2024-05-16 07:33:30.988643 D | ceph-cluster-controller: update event on CephCluster CR
2024-05-16 07:33:30.989017 D | ceph-cluster-controller: update event on CephCluster CR
2024-05-16 07:33:30.989396 D | ceph-cluster-controller: update event on CephCluster CR
2024-05-16 07:33:30.989775 D | ceph-spec: found existing monitor secrets for cluster rook-ceph-external
2024-05-16 07:33:30.991948 I | ceph-spec: parsing mon endpoints: prceph-mon02=10.11.10.30:6789,prceph-mon03=10.11.10.93:6789,prceph-mon01=10.11.10.190:6789
2024-05-16 07:33:30.992009 D | ceph-spec: loaded: maxMonID=2, mons=map[prceph-mon01:0xc0478f6b80 prceph-mon02:0xc0478f6b00 prceph-mon03:0xc0478f6b40], assignment=&{Schedule:map[]}
2024-05-16 07:33:30.992036 I | ceph-spec: found the cluster info to connect to the external cluster. will use "client.admin" to check health and monitor status. mons=map[prceph-mon01:0xc0478f6b80 prceph-mon02:0xc0478f6b00 prceph-mon03:0xc0478f6b40]
2024-05-16 07:33:30.994455 D | ceph-spec: CephCluster "rook-ceph-external" status: "Progressing". "failed to create csi kubernetes secrets: failed to create kubernetes csi secret: failed to create kubernetes secret \"rook-csi-rbd-provisioner\" for cluster \"rook-ceph-external\": failed to get secret for rook-csi-rbd-provisioner: context canceled"
2024-05-16 07:33:30.995019 I | cephclient: writing config file /var/lib/rook/rook-ceph-external/rook-ceph-external.config
2024-05-16 07:33:30.995400 I | cephclient: generated admin config in /var/lib/rook/rook-ceph-external
2024-05-16 07:33:30.995740 I | ceph-cluster-controller: external cluster identity established
2024-05-16 07:33:30.996049 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner"
2024-05-16 07:33:30.996345 D | exec: Running command: ceph auth get-or-create-key client.csi-rbd-provisioner mon profile rbd mgr allow rw osd profile rbd --connect-timeout=15 --cluster=rook-ceph-external --conf=/var/lib/rook/rook-ceph-external/rook-ceph-external.config --name=client.admin --keyring=/var/lib/rook/rook-ceph-external/client.admin.keyring --format json
2024-05-16 07:33:31.000872 D | ceph-spec: found 1 ceph clusters in namespace "rook-ceph-external"
2024-05-16 07:33:31.000929 D | ceph-cluster-controller: update event on CephCluster CR
2024-05-16 07:33:31.001073 D | ceph-spec: found 1 ceph clusters in namespace "rook-ceph-external"
2024-05-16 07:33:31.001109 D | ceph-cluster-controller: update event on CephCluster CR
2024-05-16 07:33:31.001207 D | ceph-spec: found 1 ceph clusters in namespace "rook-ceph-external"
2024-05-16 07:33:31.001238 D | ceph-cluster-controller: update event on CephCluster CR
2024-05-16 07:33:31.001326 D | ceph-spec: found 1 ceph clusters in namespace "rook-ceph-external"
2024-05-16 07:33:31.001353 D | ceph-cluster-controller: update event on CephCluster CR
2024-05-16 07:33:31.001690 D | ceph-spec: found 1 ceph clusters in namespace "rook-ceph-external"
2024-05-16 07:33:31.001732 D | ceph-cluster-controller: update event on CephCluster CR
2024-05-16 07:33:31.001914 D | ceph-spec: found 1 ceph clusters in namespace "rook-ceph-external"
2024-05-16 07:33:31.002001 D | ceph-cluster-controller: update event on CephCluster CR
2024-05-16 07:33:31.002043 I | ceph-cluster-controller: context cancelled, exiting reconcile
2024-05-16 07:33:31.002099 D | ceph-cluster-controller: successfully configured CephCluster "rook-ceph-external/rook-ceph-external"
2024-05-16 07:33:31.002156 D | ceph-spec: found 1 ceph clusters in namespace "rook-ceph-external"
2024-05-16 07:33:31.002193 D | ceph-cluster-controller: update event on CephCluster CR
2024-05-16 07:33:31.002220 D | ceph-spec: found 1 ceph clusters in namespace "rook-ceph-external"
2024-05-16 07:33:31.002268 D | ceph-cluster-controller: update event on CephCluster CR
2024-05-16 07:33:31.002379 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph"
2024-05-16 07:33:31.002494 D | ceph-spec: found 1 ceph clusters in namespace "rook-ceph-external"
2024-05-16 07:33:31.002519 D | ceph-cluster-controller: update event on CephCluster CR
2024-05-16 07:33:31.004204 D | ceph-spec: found existing monitor secrets for cluster rook-ceph
2024-05-16 07:33:31.005720 I | ceph-spec: parsing mon endpoints: i=10.101.141.73:6789,e=10.102.64.224:6789,h=10.109.166.21:6789
2024-05-16 07:33:31.005789 D | ceph-spec: loaded: maxMonID=8, mons=map[e:0xc04ecc4860 h:0xc04ecc48a0 i:0xc04ecc4820], assignment=&{Schedule:map[e:0xc05849ef40 h:0xc05849ef80 i:0xc05849efc0]}
2024-05-16 07:33:31.010347 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph"
2024-05-16 07:33:31.010396 I | op-osd: ceph osd status in namespace "rook-ceph" check interval "1m0s"
2024-05-16 07:33:31.010419 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph"
2024-05-16 07:33:31.010440 I | ceph-cluster-controller: ceph status check interval is 1m0s
2024-05-16 07:33:31.010457 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph"
2024-05-16 07:33:31.010481 D | op-mon: ceph mon status in namespace "rook-ceph" check interval "45s"
2024-05-16 07:33:31.010559 D | ceph-cluster-controller: checking health of cluster
2024-05-16 07:33:31.010606 D | exec: Running command: ceph status --format json --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --name=client.admin --keyring=/var/lib/rook/rook-ceph/client.admin.keyring
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xf0 pc=0x172ff55]
goroutine 26979 [running]:
github.com/rook/rook/pkg/operator/ceph/cluster/mon.(*HealthChecker).Check(0xc02c9226e0, 0x8cb18a, {0x1f2e303, 0x3})
	/home/runner/work/rook/rook/pkg/operator/ceph/cluster/mon/health.go:137 +0xd5
created by github.com/rook/rook/pkg/operator/ceph/cluster.(*ClusterController).startMonitoringCheck
	/home/runner/work/rook/rook/pkg/operator/ceph/cluster/monitoring.go:85 +0x217

There also logs from api server with timeout to get resources but rbac

kube-apiserver-master02b kube-apiserver E0516 08:24:23.488837       1 wrap.go:54] timeout or abort while handling: method=GET URI="/api/v1/namespaces/rook-ceph/secrets/rook-ceph-mon" audit-ID="af0a9657-ce03-446e-adee-96960b0440b9"
kube-apiserver-master02b kube-apiserver E0516 08:24:23.488885       1 timeout.go:142] post-timeout activity - time-elapsed: 4.48µs, GET "/api/v1/namespaces/rook-ceph/secrets/rook-ceph-mon" result: <nil>
kube-apiserver-master02c kube-apiserver E0516 08:12:40.918918       1 timeout.go:142] post-timeout activity - time-elapsed: 17.360091ms, DELETE "/apis/apps/v1/namespaces/rook-ceph/deployments" result: <nil>
kube-apiserver-master02b kube-apiserver E0516 08:24:23.489337       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
kube-apiserver-master02c kube-apiserver E0516 08:12:40.920969       1 timeout.go:142] post-timeout activity - time-elapsed: 18.995217ms, GET "/api/v1/namespaces/rook-ceph-external/secrets/rook-ceph-mon" result: <nil>
kube-apiserver-master02b kube-apiserver E0516 08:24:23.490618       1 timeout.go:142] post-timeout activity - time-elapsed: 3.061023ms, DELETE "/apis/apps/v1/namespaces/rook-ceph/daemonsets/rook-discover" result: <nil>

Cluster Status to submit:

Inside rook-ceph-tools pod

ceph status
  cluster:
    id:     a72c4707-301f-4acd-8007-41af0a11a860
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum e,h,i (age 6d)
    mgr: a(active, since 6d)
    mds: myfs:1 {0=myfs-a=up:active} 1 up:standby-replay
    osd: 8 osds: 8 up (since 6d), 8 in (since 2w)

  data:
    pools:   4 pools, 177 pgs
    objects: 44.89M objects, 805 GiB
    usage:   2.7 TiB used, 2.0 TiB / 4.7 TiB avail
    pgs:     177 active+clean

  io:
    client:   521 KiB/s rd, 2.6 MiB/s wr, 15 op/s rd, 87 op/s wr

but from cmd line kubectl rook-ceph ceph status Error: . failed to run command. unable to upgrade connection: container not found ("rook-ceph-operator")%!(EXTRA string=failed to get rook version) Environment:

  • OS: Debian 12
  • Kernel: 6.1.0-18-amd64
  • Cloud provider or hardware configuration: Baremetal self hosted
  • Rook version (use rook version inside of a Rook Pod): : rook: v1.9.13 / go: go1.17.13
  • Storage backend version (e.g. for ceph do ceph -v): operator pod says its ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) but in daemons its still 15.2
  • Kubernetes version (use kubectl version): 1.26
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): self hosted kubeadm
  • Storage backend status : HEALTH_OK

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 25 (14 by maintainers)

Most upvoted comments

The crash is happening after the goroutine has a health check. As a workaround, try disabling the health check for mon failover. See this topic. This should do it:

healthCheck:
  daemonHealth:
    mon:
      disabled: true

Then if you can continue upgrading to a newer version of Rook, you can re-enable the health checks again. On a newer version if you’re still seeing an issue then we can look into a fix.

These logs above are directly after delete operator pod I did delete operator-pod and share on info level then changed ConfigMap to debug log level delete pod and again cached output.