rook: External Ceph Cluster stuck in "Connecting"

  • Bug Report

Expected behavior:

CephCluster state is “CONNECTED”

Actual behavior:

CephCluster state is “CONNECTING”

How to reproduce it (minimal and precise):

kubectl create -f common.yaml kubectl create -f operator.yaml kubectl create -f common-external.yaml kubectl create -f cluster-external.yaml bash cluster/examples/kubernetes/ceph/import-external-cluster.sh

File(s) to submit:

2019-09-24 02:09:00.811884 W | op-cluster: waiting for the connection info of the external cluster. retrying in 1m0s.

2019-09-24 02:10:00.816375 W | op-cluster: waiting for the connection info of the external cluster. retrying in 1m0s.

2019-09-24 02:11:00.819592 W | op-cluster: waiting for the connection info of the external cluster. retrying in 1m0s.

2019-09-24 02:12:00.823023 W | op-cluster: waiting for the connection info of the external cluster. retrying in 1m0s.

Environment:

  • OS: RancherOS
  • Kernel (e.g. uname -a): 4.14.85-rancher
  • Cloud provider or hardware configuration: vSphere Virtual Machines
  • Rook version (use rook version inside of a Rook Pod): v1.1.1
  • Storage backend version (e.g. for ceph do ceph -v): ceph version 12.2.12-48.el7cp (26388d73d88602005946d4381cc5796d42904858) luminous (stable)
  • Kubernetes version (use kubectl version): v1.14.6
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): Rancher managed
  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): HEALTH_OK

Other:

I have an external Ceph cluster and would like to use dynamic RBD volumes from Pod PVCs to a StorageClass with Rook, using Ceph CSI.

I think I have missed a step somewhere as I have been unable to get rook to connect to an external Ceph cluster. I’m not using any internal ceph cluster, so I followed documentation from

Ceph External Cluster

Of which I interpreted as;

  • set the environment variables and run the shell script
  • verify a configmap and secret has been created
  • inject common.yaml
  • inject operator.yaml
  • inject common-external.yaml
  • modify cluster-external.yaml changing the namespace to “rook-ceph”

At this stage, the Pods were in a crashbackoff loop, and I had to perform the following.

In the operator yaml I modified;

# RancherOS requires a different directory
- name: FLEXVOLUME_DIR_PATH
  value: "/var/lib/kubelet/volumeplugins"

# RancherOS requires a specific value for the Kubelet
- name: ROOK_CSI_KUBELET_DIR_PATH
  value: "/opt/rke/var/lib/kubelet"

I also had to modify the daemonset and change

- --containerized=false

With this in place, I have the CephCluster state stuck in “CONNECTING”

I can see the following error when I look at the csi-rbdplugin container logs within the operator pod

E0924 02:44:39.175178       1 utils.go:123] ID: 25 GRPC error: rpc error: code = InvalidArgument desc = failed to fetch monitor list using clusterID (rook-ceph): missing configuration for cluster ID (rook-ceph)

So where should the Ceph CSI RBD plugin be reading this information from?

If I look at the following documentation;

Ceph CSI

Seems to indicate a configmap is required with the clusterid and monitors, is the Rook Operator supposed to create this from the information provided in the secret and configmap from the bash script?

The samples given within the Ceph CSI repo indicates there is another configMap

Ceph CSI sample configmap

I’ve definitely missed something here but I can’t figure it out at this point.

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 27 (11 by maintainers)

Most upvoted comments

Getting a similar issue here.

storage01 ~]# ceph -v
ceph version 15.2.5 (2c93eff00150f0cc5f106a559557a58d3d7b6f1f) octopus (stable)

Procedure followed:

kubectl create -f crds.yaml -f common.yaml -f operator.yaml
# on ceph cluster run: 
./create-external-cluster-resources.sh
# export all the keys outputted from create-external-cluster-resources.sh into kubectl cli host. 
export ROOK_EXTERNAL_USER_SECRET=<>
export ROOK_EXTERNAL_USERNAME=client.healthchecker
export CSI_RBD_NODE_SECRET_SECRET=<>
export CSI_RBD_PROVISIONER_SECRET=<>
export CSI_CEPHFS_NODE_SECRET=<>
export CSI_CEPHFS_PROVISIONER_SECRET=<>
export NAMESPACE="rook-ceph"
export ROOK_EXTERNAL_FSID="<>"
export ROOK_EXTERNAL_CEPH_MON_DATA="IP01:6789,IP02:6789,IP03:6789"
./import-external-cluster.sh
sed -i 's/name: rook-ceph-external/name: rook-ceph/' cluster-external.yaml
sed -i 's/namespace: rook-ceph-external/namespace: rook-ceph/' cluster-external.yaml
kubectl create -f cluster-external.yaml

Results:

control01 ~]# kubectl get CephCluster -A
NAMESPACE   NAME        DATADIRHOSTPATH   MONCOUNT   AGE     PHASE        MESSAGE                 HEALTH
rook-ceph   rook-ceph                                2d22h   Connecting   Cluster is connecting   
control01 ~]# kubectl logs -n rook-ceph -l app=rook-ceph-operator
2020-12-07 17:31:43.104445 W | op-mon: waiting for the csi connection info of the external cluster. retrying in 1m0s.
2020-12-07 17:32:43.107853 W | op-mon: waiting for the csi connection info of the external cluster. retrying in 1m0s.
2020-12-07 17:33:43.111118 W | op-mon: waiting for the csi connection info of the external cluster. retrying in 1m0s.
2020-12-07 17:34:43.113949 W | op-mon: waiting for the csi connection info of the external cluster. retrying in 1m0s.
2020-12-07 17:35:43.116510 W | op-mon: waiting for the csi connection info of the external cluster. retrying in 1m0s.
2020-12-07 17:36:43.119298 W | op-mon: waiting for the csi connection info of the external cluster. retrying in 1m0s.
2020-12-07 17:37:43.121864 W | op-mon: waiting for the csi connection info of the external cluster. retrying in 1m0s.
2020-12-07 17:38:43.124678 W | op-mon: waiting for the csi connection info of the external cluster. retrying in 1m0s.
2020-12-07 17:39:43.127947 W | op-mon: waiting for the csi connection info of the external cluster. retrying in 1m0s.
2020-12-07 17:40:43.131406 W | op-mon: waiting for the csi connection info of the external cluster. retrying in 1m0s.
...
control01 ~]# kubectl logs -n rook-ceph csi-rbdplugin-9k4nm csi-rbdplugin
W1204 18:50:59.252835 1370737 driver.go:171] EnableGRPCMetrics is deprecated
[root@control01 ~]# kubectl logs -n rook-ceph csi-rbdplugin-provisioner-65f85f795c-dss2f csi-provisioner
I1204 18:50:59.379229       1 csi-provisioner.go:121] Version: v2.0.0
I1204 18:50:59.379283       1 csi-provisioner.go:135] Building kube configs for running in cluster...
I1204 18:50:59.384295       1 connection.go:153] Connecting to unix:///csi/csi-provisioner.sock
I1204 18:51:00.384920       1 common.go:111] Probing CSI driver for readiness
W1204 18:51:00.389090       1 metrics.go:142] metrics endpoint will not be started because `metrics-address` was not specified.
I1204 18:51:00.395272       1 leaderelection.go:243] attempting to acquire leader lease  rook-ceph/rook-ceph-rbd-csi-ceph-com...
I1204 18:51:00.401884       1 leaderelection.go:253] successfully acquired lease rook-ceph/rook-ceph-rbd-csi-ceph-com
I1204 18:51:00.502320       1 clone_controller.go:66] Starting CloningProtection controller
I1204 18:51:00.502331       1 volume_store.go:97] Starting save volume queue
I1204 18:51:00.502242       1 controller.go:820] Starting provisioner controller rook-ceph.rbd.csi.ceph.com_csi-rbdplugin-provisioner-65f85f795c-dss2f_d5fdc667-5dc6-4973-98a9-eb52386097a7!
I1204 18:51:00.502375       1 clone_controller.go:84] Started CloningProtection controller
I1204 18:51:00.602870       1 controller.go:869] Started provisioner controller rook-ceph.rbd.csi.ceph.com_csi-rbdplugin-provisioner-65f85f795c-dss2f_d5fdc667-5dc6-4973-98a9-eb52386097a7!
control01 ~]# kubectl -n rook-ceph get cm
NAME                        DATA   AGE
rook-ceph-csi-config        1      2d23h
rook-ceph-operator-config   6      2d23h
 control01 ~]# kubectl -n rook-ceph get secret
NAME                                         TYPE                                  DATA   AGE
default-token-5qvz2                          kubernetes.io/service-account-token   3      2d23h
rook-ceph-admission-controller-token-fmfzc   kubernetes.io/service-account-token   3      2d23h
rook-ceph-cmd-reporter-token-dcwft           kubernetes.io/service-account-token   3      2d23h
rook-ceph-mgr-token-cgxcv                    kubernetes.io/service-account-token   3      2d23h
rook-ceph-osd-token-bxphp                    kubernetes.io/service-account-token   3      2d23h
rook-ceph-system-token-z959t                 kubernetes.io/service-account-token   3      2d23h
rook-csi-cephfs-plugin-sa-token-fb5xx        kubernetes.io/service-account-token   3      2d23h
rook-csi-cephfs-provisioner-sa-token-t6bpj   kubernetes.io/service-account-token   3      2d23h
rook-csi-rbd-plugin-sa-token-qrcrg           kubernetes.io/service-account-token   3      2d23h
rook-csi-rbd-provisioner-sa-token-zkn8x      kubernetes.io/service-account-token   3      2d23h
control01 ~]# cat < /dev/tcp/IP01/6789
ceph v027�
���x
��
control01 ~]# cat < /dev/tcp/IP02/6789
ceph v027�
���p
��
control01 ~]# cat < /dev/tcp/IP03/6789
ceph v027�
��˴
��