rook: External cluster connected, but not healthy: Failed to configure external ceph cluster, mon quorum status failed: exit status 1

OVERVIEW

The objective is to have 2 Kubernetes clusters:

  • Storage cluster: a Kubernetes cluster that contains a ceph cluster with RBD pool and storage
  • Consumer cluster: a Kubernetes cluster with rook, that consumes the external ceph cluster storage

The Storage cluster has a ceph cluster created by rook and it should be exposed to the Consumer cluster, from which another instance of rook would be able to connect to the external ceph cluster (in the Consumer cluster).

The problem is that the connection to the external cluster is not healthy:

# Check the connectivity to the external ceph cluster
$ kubectl get CephCluster -n rook-ceph
NAME                 DATADIRHOSTPATH   MONCOUNT   AGE    PHASE       MESSAGE                                     HEALTH       EXTERNAL   FSID
rook-ceph-external                                6m9s   Connected   Failed to configure external ceph cluster   HEALTH_ERR   true

The rook operator pod shows this error in the Consumer cluster:

$ kubectl logs rook-ceph-operator-7979d647b7-6t5jq -n rook-ceph
...
2024-07-10 16:18:44.597830 I | op-bucket-prov: successfully reconciled bucket provisioner
I0710 16:18:44.597928       1 manager.go:135] "msg"="starting provisioner" "logger"="objectbucket.io/provisioner-manager" "name"="rook-ceph.ceph.rook.io/bucket"
2024-07-10 16:18:44.616748 E | ceph-spec: failed to update cluster condition to {Type:Connecting Status:True Reason:ClusterConnecting Message:Attempting to connect to an external Ceph cluster LastHeartbeatTime:2024-07-10 16:18:44.596867806 +0000 UTC m=+763.010000215 LastTransitionTime:2024-07-10 16:18:44.596867748 +0000 UTC m=+763.010000165}. failed to update object "rook-ceph/rook-ceph-external" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph-external": the object has been modified; please apply your changes to the latest version and try again
...
2024-07-10 16:19:00.329264 E | ceph-cluster-controller: failed to get ceph status. failed to get status. . timed out: exit status 1
2024-07-10 16:19:15.796062 E | ceph-cluster-controller: failed to get ceph daemons versions. failed to run 'ceph versions'. . timed out: exit status 1
...
2024-07-10 16:19:20.512357 I | ceph-csi: successfully removed CSI NFS driver
2024-07-10 16:19:45.454808 W | op-mon: failed to check mon health. failed to get external mon quorum status: mon quorum status failed: exit status 1
2024-07-10 16:20:31.097914 E | ceph-cluster-controller: failed to get ceph status. failed to get status. . timed out: exit status 1
2024-07-10 16:20:45.622343 W | op-mon: failed to check mon health. failed to get external mon quorum status: mon quorum status failed: exit status 1
2024-07-10 16:20:46.242919 E | ceph-cluster-controller: failed to get ceph daemons versions. failed to run 'ceph versions'. . timed out: exit status 1

Could you please help on how I can troubleshoot this further? Thank you.

STEPS TO REPRODUCE

1. Create the Storage cluster

Execute on the control plane node of the Storage Kubernetes cluster:

# Install rook and deploy the operator
$ git clone --single-branch --branch v1.14.7 https://github.com/rook/rook.git
$ cd rook/deploy/examples
$ kubectl create -f crds.yaml -f common.yaml -f operator.yaml

# Enable host networking to properly expose the services
$ sed -i "s/#provider: host/provider: host/g" cluster.yaml

# Create the ceph cluster
$ kubectl create -f cluster.yaml
$ kubectl get pods -n rook-ceph

# Check that the ceph cluster is healthy
$ kubectl create -f toolbox.yaml
$ kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph osd status
ID  HOST               USED  AVAIL  WR OPS  WR DATA  RD OPS  RD DATA  STATE      
 0  storage-worker-1  26.7M  19.9G      0        0       0        0   exists,up  
 1  storage-worker-3  26.7M  19.9G      0        0       0        0   exists,up  
 2  storage-worker-2  26.7M  19.9G      0        0       0        0   exists,up

# Create the rbd pool and the block storage class
$ kubectl create -f csi/rbd/storageclass.yaml
$ kubectl get CephBlockPool -n rook-ceph
NAME          PHASE   TYPE         FAILUREDOMAIN   AGE
replicapool   Ready   Replicated   host            60s

2. Export the Ceph cluster configuration (from the Storage cluster)

Execute on the control plane node of the Storage Kubernetes cluster:

# Deploy a privileged toolbox pod to run the python script
$ kubectl delete -f toolbox.yaml
$ vi toolbox.yaml
# Remove the lines with security context
# https://github.com/rook/rook/blob/0837c15b86c4d836c486aca03ca3e586f78c5dfb/deploy/examples/toolbox.yaml#L91C11-L96C28
$ kubectl apply -f toolbox.yaml

# Copy the python script to the toolbox pod
$ POD_NAME=$(kubectl get pod -l app=rook-ceph-tools -n rook-ceph -o jsonpath="{.items[0].metadata.name}")
$ kubectl cp create-external-cluster-resources.py -n rook-ceph $POD_NAME:/

# Run the python script to get the environment variables
$ kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- /bin/bash
> python3 create-external-cluster-resources.py --rbd-data-pool-name replicapool --namespace rook-ceph --format bash
export NAMESPACE=rook-ceph
export ROOK_EXTERNAL_FSID=7acde693-53a7-4169-84c5-19f20d801aa6
export ROOK_EXTERNAL_USERNAME=client.healthchecker
export ROOK_EXTERNAL_CEPH_MON_DATA=a=192.168.35.177:6789
export ROOK_EXTERNAL_USER_SECRET=AQDlq45mD5g0IhAAkNBYLoUe4/BZYrHvz5IJcw==
export ROOK_EXTERNAL_DASHBOARD_LINK=https://192.168.35.176:8443/
export CSI_RBD_NODE_SECRET=AQBxp45msGrgNxAAgqTxhS3Eqs+tOQ04V6itjw==
export CSI_RBD_NODE_SECRET_NAME=csi-rbd-node
export CSI_RBD_PROVISIONER_SECRET=AQBxp45mmSVTChAAp0p4cLFtJIc02WlV0jRaHA==
export CSI_RBD_PROVISIONER_SECRET_NAME=csi-rbd-provisioner
export MONITORING_ENDPOINT=192.168.35.176
export MONITORING_ENDPOINT_PORT=9283
export RBD_POOL_NAME=replicapool
export RGW_POOL_PREFIX=default

All pods in the rook-ceph namespace are healthy. I can access the dashboard, and in the Pools section I see the replicapool RBD pool.

3. Check the connectivity from the Consumer to the Storage cluster

Execute on the control plane node of the Consumer Kubernetes cluster:

$ cat < /dev/tcp/192.168.35.177/6789
ceph v027���#����#�

$ curl -k https://192.168.35.176:8443
<!DOCTYPE html><html lang="en-US" dir="ltr"><head>
  <meta charset="utf-8">
  <title>Ceph</title>
  ...

$ curl http://192.168.35.176:9283
<!DOCTYPE html>
<html>
    <head><title>Ceph Exporter</title></head>
    <body>
        <h1>Ceph Exporter</h1>
        <p><a href='/metrics'>Metrics</a></p>
    </body>
</html>

4. Create the Consumer cluster

Execute on the control plane node of the Consumer Kubernetes cluster:

# Install rook and deploy the operator
$ git clone --single-branch --branch v1.14.7 https://github.com/rook/rook.git
$ cd rook/deploy/examples
$ kubectl create -f crds.yaml -f common.yaml -f operator.yaml
$ kubectl get pods -n rook-ceph
NAME                                  READY   STATUS    RESTARTS   AGE
rook-ceph-operator-7979d647b7-6t5jq   1/1     Running   0          41s

# Import the ceph cluster secrets and configuration
$ export NAMESPACE=rook-ceph
$ export ROOK_EXTERNAL_FSID=7acde693-53a7-4169-84c5-19f20d801aa6
$ export ROOK_EXTERNAL_USERNAME=client.healthchecker
$ export ROOK_EXTERNAL_CEPH_MON_DATA=a=192.168.35.177:6789
$ export ROOK_EXTERNAL_USER_SECRET=AQDlq45mD5g0IhAAkNBYLoUe4/BZYrHvz5IJcw==
$ export ROOK_EXTERNAL_DASHBOARD_LINK=https://192.168.35.176:8443/
$ export CSI_RBD_NODE_SECRET=AQBxp45msGrgNxAAgqTxhS3Eqs+tOQ04V6itjw==
$ export CSI_RBD_NODE_SECRET_NAME=csi-rbd-node
$ export CSI_RBD_PROVISIONER_SECRET=AQBxp45mmSVTChAAp0p4cLFtJIc02WlV0jRaHA==
$ export CSI_RBD_PROVISIONER_SECRET_NAME=csi-rbd-provisioner
$ export MONITORING_ENDPOINT=192.168.35.176
$ export MONITORING_ENDPOINT_PORT=9283
$ export RBD_POOL_NAME=replicapool
$ export RGW_POOL_PREFIX=default

# Run the import script
$ bash external/import-external-cluster.sh
cluster namespace rook-ceph already exists
secret/rook-ceph-mon created
configmap/rook-ceph-mon-endpoints created
secret/rook-csi-rbd-node created
secret/rook-csi-rbd-provisioner created
storageclass.storage.k8s.io/ceph-rbd create

# Create the external cluster
$ kubectl create -f external/cluster-external.yaml
$ kubectl get pods -n rook-ceph
NAME                                            READY   STATUS    RESTARTS        AGE
csi-cephfsplugin-lvz9s                          2/2     Running   0               3m42s
csi-cephfsplugin-provisioner-868bf46b56-bj9b9   5/5     Running   1 (2m54s ago)   3m42s
csi-cephfsplugin-provisioner-868bf46b56-fw5nb   5/5     Running   0               3m42s
csi-cephfsplugin-zvhzb                          2/2     Running   1 (3m3s ago)    3m42s
csi-rbdplugin-dqhcv                             2/2     Running   1 (3m4s ago)    3m42s
csi-rbdplugin-lv8j9                             2/2     Running   0               3m42s
csi-rbdplugin-provisioner-d9b9d694c-6kgpk       5/5     Running   1 (2m44s ago)   3m42s
csi-rbdplugin-provisioner-d9b9d694c-dhbnt       5/5     Running   1 (2m52s ago)   3m42s
rook-ceph-operator-7979d647b7-6t5jq             1/1     Running   0               17m

# Check the connectivity to the external ceph cluster
$ kubectl get CephCluster -n rook-ceph
NAME                 DATADIRHOSTPATH   MONCOUNT   AGE    PHASE       MESSAGE                                     HEALTH       EXTERNAL   FSID
rook-ceph-external                                6m9s   Connected   Failed to configure external ceph cluster   HEALTH_ERR   true

LOGS & ENVIRONMENT

Rook operator pod logs in the Consumer cluster

$ kubectl logs rook-ceph-operator-7979d647b7-6t5jq -n rook-ceph
2024/07/10 16:06:01 maxprocs: Leaving GOMAXPROCS=3: CPU quota undefined
2024-07-10 16:06:01.655425 I | rookcmd: starting Rook v1.14.7 with arguments '/usr/local/bin/rook ceph operator'
2024-07-10 16:06:01.655469 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO
2024-07-10 16:06:01.655474 I | cephcmd: starting Rook-Ceph operator
2024-07-10 16:06:02.276908 I | cephcmd: base ceph version inside the rook operator image is "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)"
2024-07-10 16:06:02.286318 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var)
2024-07-10 16:06:02.286340 I | operator: watching all namespaces for Ceph CRs
2024-07-10 16:06:02.286414 I | operator: setting up schemes
2024-07-10 16:06:02.288714 I | operator: setting up the controller-runtime manager
2024-07-10 16:06:02.289863 I | ceph-cluster-controller: successfully started
2024-07-10 16:06:02.294289 I | op-k8sutil: ROOK_DISABLE_DEVICE_HOTPLUG="false" (configmap)
2024-07-10 16:06:02.294428 I | ceph-cluster-controller: enabling hotplug orchestration
2024-07-10 16:06:02.294663 I | ceph-nodedaemon-controller: successfully started
2024-07-10 16:06:02.294714 I | ceph-block-pool-controller: successfully started
2024-07-10 16:06:02.294897 I | ceph-object-store-user-controller: successfully started
2024-07-10 16:06:02.295043 I | ceph-object-realm-controller: successfully started
2024-07-10 16:06:02.295114 I | ceph-object-zonegroup-controller: successfully started
2024-07-10 16:06:02.295177 I | ceph-object-zone-controller: successfully started
2024-07-10 16:06:02.295759 I | ceph-object-controller: successfully started
2024-07-10 16:06:02.296054 I | ceph-file-controller: successfully started
2024-07-10 16:06:02.296156 I | ceph-nfs-controller: successfully started
2024-07-10 16:06:02.296248 I | ceph-rbd-mirror-controller: successfully started
2024-07-10 16:06:02.296325 I | ceph-client-controller: successfully started
2024-07-10 16:06:02.296474 I | ceph-filesystem-mirror-controller: successfully started
2024-07-10 16:06:02.296527 I | operator: rook-ceph-operator-config-controller successfully started
2024-07-10 16:06:02.296554 I | ceph-csi: rook-ceph-operator-csi-controller successfully started
2024-07-10 16:06:02.296648 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started
2024-07-10 16:06:02.296681 I | ceph-bucket-topic: successfully started
2024-07-10 16:06:02.296707 I | ceph-bucket-notification: successfully started
2024-07-10 16:06:02.296730 I | ceph-bucket-notification: successfully started
2024-07-10 16:06:02.296758 I | ceph-fs-subvolumegroup-controller: successfully started
2024-07-10 16:06:02.296783 I | blockpool-rados-namespace-controller: successfully started
2024-07-10 16:06:02.296808 I | ceph-cosi-controller: successfully started
2024-07-10 16:06:02.296840 I | operator: starting the controller-runtime manager
2024-07-10 16:06:02.402951 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap)
2024-07-10 16:06:02.402977 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap)
2024-07-10 16:06:02.402991 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap)
2024-07-10 16:06:02.408097 I | op-k8sutil: ROOK_CEPH_ALLOW_LOOP_DEVICES="false" (configmap)
2024-07-10 16:06:02.408311 I | operator: rook-ceph-operator-config-controller done reconciling
2024-07-10 16:06:02.443537 I | op-k8sutil: ROOK_WATCH_FOR_NODE_FAILURE="true" (configmap)
2024-07-10 16:06:02.467126 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config"
2024-07-10 16:06:02.467190 I | op-k8sutil: ROOK_CSI_DISABLE_DRIVER="false" (configmap)
2024-07-10 16:06:02.468775 I | ceph-csi: CSI Ceph RBD driver disabled
2024-07-10 16:06:02.468920 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists
2024-07-10 16:06:02.472009 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists
2024-07-10 16:06:02.482219 I | ceph-csi: successfully removed CSI Ceph RBD driver
2024-07-10 16:06:02.482264 I | ceph-csi: CSI CephFS driver disabled
2024-07-10 16:06:02.482271 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists
2024-07-10 16:06:02.485703 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists
2024-07-10 16:06:02.497031 I | ceph-csi: successfully removed CSI CephFS driver
2024-07-10 16:06:02.497063 I | ceph-csi: CSI NFS driver disabled
2024-07-10 16:06:02.497070 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists
2024-07-10 16:06:02.501433 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists
2024-07-10 16:06:02.511547 I | ceph-csi: successfully removed CSI NFS driver
2024-07-10 16:18:44.584296 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph-external"
2024-07-10 16:18:44.596836 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph"
2024-07-10 16:18:44.597227 I | ceph-spec: parsing mon endpoints: a=192.168.35.177:6789
2024-07-10 16:18:44.597381 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap)
2024-07-10 16:18:44.597388 I | op-k8sutil: ROOK_OBC_PROVISIONER_NAME_PREFIX="" (default)
2024-07-10 16:18:44.597392 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket"
2024-07-10 16:18:44.597830 I | op-bucket-prov: successfully reconciled bucket provisioner
I0710 16:18:44.597928       1 manager.go:135] "msg"="starting provisioner" "logger"="objectbucket.io/provisioner-manager" "name"="rook-ceph.ceph.rook.io/bucket"
2024-07-10 16:18:44.616748 E | ceph-spec: failed to update cluster condition to {Type:Connecting Status:True Reason:ClusterConnecting Message:Attempting to connect to an external Ceph cluster LastHeartbeatTime:2024-07-10 16:18:44.596867806 +0000 UTC m=+763.010000215 LastTransitionTime:2024-07-10 16:18:44.596867748 +0000 UTC m=+763.010000165}. failed to update object "rook-ceph/rook-ceph-external" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph-external": the object has been modified; please apply your changes to the latest version and try again
2024-07-10 16:18:44.636194 I | ceph-spec: parsing mon endpoints: a=192.168.35.177:6789
2024-07-10 16:18:44.636276 I | ceph-spec: found the cluster info to connect to the external cluster. will use "client.healthchecker" to check health and monitor status. mons=map[a:0xc0010889f0]
2024-07-10 16:18:44.645757 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2024-07-10 16:18:44.645994 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2024-07-10 16:18:44.646010 I | ceph-cluster-controller: external cluster identity established
2024-07-10 16:18:44.795048 I | ceph-cluster-controller: successfully updated csi config map
2024-07-10 16:18:44.991344 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default)
2024-07-10 16:18:44.991477 I | op-k8sutil: CSI_DISABLE_HOLDER_PODS="true" (configmap)
2024-07-10 16:18:45.189437 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph"
2024-07-10 16:18:45.189642 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph"
2024-07-10 16:18:45.588997 I | ceph-spec: parsing mon endpoints: a=192.168.35.177:6789
2024-07-10 16:18:45.788996 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap)
2024-07-10 16:18:45.789086 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap)
2024-07-10 16:18:45.789120 I | op-k8sutil: ROOK_CSI_ENABLE_NFS="false" (configmap)
2024-07-10 16:18:45.789137 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap)
2024-07-10 16:18:45.789232 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap)
2024-07-10 16:18:45.789249 I | op-k8sutil: CSI_GRPC_TIMEOUT_SECONDS="150" (configmap)
2024-07-10 16:18:45.789267 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default)
2024-07-10 16:18:45.789285 I | op-k8sutil: CSIADDONS_PORT="9070" (default)
2024-07-10 16:18:45.789302 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default)
2024-07-10 16:18:45.789318 I | op-k8sutil: CSI_ENABLE_LIVENESS="false" (configmap)
2024-07-10 16:18:45.789341 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="system-node-critical" (configmap)
2024-07-10 16:18:45.789357 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="system-cluster-critical" (configmap)
2024-07-10 16:18:45.789377 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default)
2024-07-10 16:18:45.789395 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap)
2024-07-10 16:18:45.789410 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap)
2024-07-10 16:18:45.789449 I | op-k8sutil: CSI_ENABLE_NFS_SNAPSHOTTER="true" (configmap)
2024-07-10 16:18:45.792140 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap)
2024-07-10 16:18:45.792320 I | op-k8sutil: CSI_ENABLE_TOPOLOGY="false" (configmap)
2024-07-10 16:18:45.792342 I | op-k8sutil: CSI_ENABLE_ENCRYPTION="false" (configmap)
2024-07-10 16:18:45.792363 I | op-k8sutil: CSI_ENABLE_METADATA="false" (default)
2024-07-10 16:18:45.792379 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2024-07-10 16:18:45.792396 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE="1" (default)
2024-07-10 16:18:45.792412 I | op-k8sutil: CSI_NFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2024-07-10 16:18:45.792473 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2024-07-10 16:18:45.792500 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE="1" (default)
2024-07-10 16:18:45.792518 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap)
2024-07-10 16:18:45.792534 I | ceph-csi: Kubernetes version is 1.30
2024-07-10 16:18:45.792555 I | op-k8sutil: CSI_LOG_LEVEL="" (default)
2024-07-10 16:18:45.792580 I | op-k8sutil: CSI_SIDECAR_LOG_LEVEL="" (default)
2024-07-10 16:18:45.792596 I | op-k8sutil: CSI_LEADER_ELECTION_LEASE_DURATION="" (default)
2024-07-10 16:18:45.792612 I | op-k8sutil: CSI_LEADER_ELECTION_RENEW_DEADLINE="" (default)
2024-07-10 16:18:45.792628 I | op-k8sutil: CSI_LEADER_ELECTION_RETRY_PERIOD="" (default)
2024-07-10 16:18:45.990249 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap)
2024-07-10 16:18:45.990301 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.11.0" (default)
2024-07-10 16:18:45.990315 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.1" (default)
2024-07-10 16:18:45.990333 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="registry.k8s.io/sig-storage/csi-provisioner:v4.0.1" (default)
2024-07-10 16:18:45.990345 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="registry.k8s.io/sig-storage/csi-attacher:v4.5.1" (default)
2024-07-10 16:18:45.990440 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="registry.k8s.io/sig-storage/csi-snapshotter:v7.0.2" (default)
2024-07-10 16:18:45.990456 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="registry.k8s.io/sig-storage/csi-resizer:v1.10.1" (default)
2024-07-10 16:18:45.990467 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default)
2024-07-10 16:18:45.990477 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.8.0" (default)
2024-07-10 16:18:45.990560 I | op-k8sutil: CSI_TOPOLOGY_DOMAIN_LABELS="" (default)
2024-07-10 16:18:45.990570 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default)
2024-07-10 16:18:45.990578 I | op-k8sutil: ROOK_CSI_NFS_POD_LABELS="" (default)
2024-07-10 16:18:45.990585 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default)
2024-07-10 16:18:45.990593 I | op-k8sutil: CSI_CLUSTER_NAME="" (default)
2024-07-10 16:18:45.990600 I | op-k8sutil: ROOK_CSI_IMAGE_PULL_POLICY="IfNotPresent" (default)
2024-07-10 16:18:45.990607 I | op-k8sutil: CSI_CEPHFS_KERNEL_MOUNT_OPTIONS="" (default)
2024-07-10 16:18:45.990615 I | op-k8sutil: CSI_CEPHFS_ATTACH_REQUIRED="true" (configmap)
2024-07-10 16:18:45.990622 I | op-k8sutil: CSI_RBD_ATTACH_REQUIRED="true" (configmap)
2024-07-10 16:18:45.990630 I | op-k8sutil: CSI_NFS_ATTACH_REQUIRED="true" (configmap)
2024-07-10 16:18:45.990637 I | op-k8sutil: CSI_DRIVER_NAME_PREFIX="rook-ceph" (default)
2024-07-10 16:18:45.995997 I | op-k8sutil: CSI_ENABLE_VOLUME_GROUP_SNAPSHOT="true" (configmap)
2024-07-10 16:18:45.996055 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.11.0"
2024-07-10 16:18:45.996193 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default)
2024-07-10 16:18:45.996213 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default)
2024-07-10 16:19:00.329264 E | ceph-cluster-controller: failed to get ceph status. failed to get status. . timed out: exit status 1
2024-07-10 16:19:15.796062 E | ceph-cluster-controller: failed to get ceph daemons versions. failed to run 'ceph versions'. . timed out: exit status 1
2024-07-10 16:19:19.891503 I | ceph-csi: Detected ceph CSI image version: "v3.11.0"
2024-07-10 16:19:19.943175 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default)
2024-07-10 16:19:19.943201 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default)
2024-07-10 16:19:19.943213 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default)
2024-07-10 16:19:19.943230 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default)
2024-07-10 16:19:19.943236 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default)
2024-07-10 16:19:19.943241 I | op-k8sutil: CSI_RBD_PLUGIN_VOLUME="" (default)
2024-07-10 16:19:19.943246 I | op-k8sutil: CSI_RBD_PLUGIN_VOLUME_MOUNT="" (default)
2024-07-10 16:19:19.968438 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default)
2024-07-10 16:19:19.968494 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default)
2024-07-10 16:19:19.968507 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default)
2024-07-10 16:19:20.034862 I | ceph-csi: successfully started CSI Ceph RBD driver
2024-07-10 16:19:20.034908 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default)
2024-07-10 16:19:20.034923 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default)
2024-07-10 16:19:20.034962 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default)
2024-07-10 16:19:20.034974 I | op-k8sutil: CSI_CEPHFS_PLUGIN_VOLUME="" (default)
2024-07-10 16:19:20.034985 I | op-k8sutil: CSI_CEPHFS_PLUGIN_VOLUME_MOUNT="" (default)
2024-07-10 16:19:20.098928 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default)
2024-07-10 16:19:20.098953 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default)
2024-07-10 16:19:20.098962 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default)
2024-07-10 16:19:20.147555 I | ceph-csi: successfully started CSI CephFS driver
2024-07-10 16:19:20.147634 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="File" (configmap)
2024-07-10 16:19:20.201062 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com"
2024-07-10 16:19:20.201103 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="File" (configmap)
2024-07-10 16:19:20.299109 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com"
2024-07-10 16:19:20.299204 I | ceph-csi: CSI NFS driver disabled
2024-07-10 16:19:20.299226 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists
2024-07-10 16:19:20.345977 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists
2024-07-10 16:19:20.512357 I | ceph-csi: successfully removed CSI NFS driver
2024-07-10 16:19:45.454808 W | op-mon: failed to check mon health. failed to get external mon quorum status: mon quorum status failed: exit status 1
2024-07-10 16:20:31.097914 E | ceph-cluster-controller: failed to get ceph status. failed to get status. . timed out: exit status 1
2024-07-10 16:20:45.622343 W | op-mon: failed to check mon health. failed to get external mon quorum status: mon quorum status failed: exit status 1
2024-07-10 16:20:46.242919 E | ceph-cluster-controller: failed to get ceph daemons versions. failed to run 'ceph versions'. . timed out: exit status 1
2024-07-10 16:21:45.759154 W | op-mon: failed to check mon health. failed to get external mon quorum status: mon quorum status failed: exit status 1
2024-07-10 16:22:01.381425 E | ceph-cluster-controller: failed to get ceph status. failed to get status. . timed out: exit status 1
2024-07-10 16:22:16.499964 E | ceph-cluster-controller: failed to get ceph daemons versions. failed to run 'ceph versions'. . timed out: exit status 1
2024-07-10 16:22:45.886656 W | op-mon: failed to check mon health. failed to get external mon quorum status: mon quorum status failed: exit status 1
2024-07-10 16:23:31.673924 E | ceph-cluster-controller: failed to get ceph status. failed to get status. . timed out: exit status 1
2024-07-10 16:23:46.039062 W | op-mon: failed to check mon health. failed to get external mon quorum status: mon quorum status failed: exit status 1
2024-07-10 16:23:46.809927 E | ceph-cluster-controller: failed to get ceph daemons versions. failed to run 'ceph versions'. . timed out: exit status 1
2024-07-10 16:24:46.154165 W | op-mon: failed to check mon health. failed to get external mon quorum status: mon quorum status failed: exit status 1
2024-07-10 16:25:01.932469 E | ceph-cluster-controller: failed to get ceph status. failed to get status. . timed out: exit status 1
2024-07-10 16:25:17.077955 E | ceph-cluster-controller: failed to get ceph daemons versions. failed to run 'ceph versions'. . timed out: exit status 1
2024-07-10 16:25:46.279390 W | op-mon: failed to check mon health. failed to get external mon quorum status: mon quorum status failed: exit status 1
2024-07-10 16:26:32.186998 E | ceph-cluster-controller: failed to get ceph status. failed to get status. . timed out: exit status 1

Environment

  • The Kubernetes clusters are created on Linux nodes with kubeadm
  • The nodes are Linux VMs created in VMware Workstation
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)" 
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

$ uname -a
Linux storage-master-1 5.15.0-113-generic #123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

$ kubectl version
Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.2

# Consumer cluster
$ kubectl get nodes
NAME                  STATUS   ROLES           AGE    VERSION
kubernetes-master-1   Ready    control-plane   6d1h   v1.30.2
kubernetes-worker-1   Ready    <none>          6d1h   v1.30.2
kubernetes-worker-2   Ready    <none>          6d1h   v1.30.2

# Storage cluster
$ kubectl get nodes
NAME               STATUS   ROLES           AGE    VERSION
storage-master-1   Ready    control-plane   6d1h   v1.30.2
storage-worker-1   Ready    <none>          6d1h   v1.30.2
storage-worker-2   Ready    <none>          6d1h   v1.30.2
storage-worker-3   Ready    <none>          6d1h   v1.30.2

Rook information on both clusters

$ kubectl exec -it rook-ceph-operator-7979d647b7-6t5jq -n rook-ceph -- rook version
2024/07/10 16:39:12 maxprocs: Leaving GOMAXPROCS=3: CPU quota undefined
rook: v1.14.7
go: go1.21.11

$ kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph -v
ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)

$ kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph status
  cluster:
    id:     7acde693-53a7-4169-84c5-19f20d801aa6
    health: HEALTH_WARN
            Slow OSD heartbeats on back (longest 3460.954ms)
            Slow OSD heartbeats on front (longest 3468.227ms)
 
  services:
    mon: 3 daemons, quorum a,b,c (age 4m)
    mgr: b(active, since 52s), standbys: a
    osd: 3 osds: 3 up (since 4m), 3 in (since 77m)
 
  data:
    pools:   2 pools, 33 pgs
    objects: 3 objects, 449 KiB
    usage:   85 MiB used, 60 GiB / 60 GiB avail
    pgs:     33 active+clean

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 21 (12 by maintainers)

Commits related to this issue

Most upvoted comments

@travisn @BlaineEXE

Root cause of the problem:

The pod CIDR in the clusters (192.168.0.0/16) was overlapping with the host network nodes IP addresses (192.168.35.x). Pod CIDR was configured when creating the cluster, and when deploying Calico.

When Calico sees a 192.x.x.x IP, it considers it as an internal IP address to the cluster, so it will not do SNAT for these IPs. The problem was that my external server IP was considered as an internal IP because of the overlap. This was clearly a misconfiguration.

How to fix this:

If someone is in the same situation, he can check the current pod CIDR with kubectl get ippool -o yaml and compare it with the local network IPs.

  1. In order to solve this, I recreated the clusters with kubeadm and using the parameter --pod-network-cidr=10.200.0.0/16. Also, when deploying Calico, I replaced the default CIDR 192.168.0.0/16 with 10.200.0.0/16 (Calico custom resources).

  2. And if someone follows the “steps to reproduce” in the first post, you need to enable host networking with sed -i "s/#provider: host/provider: host/g" cluster.yaml in the Storage cluster, before applying the cluster.yaml file. This is required, so that the Ceph services can be properly accessed from outside the Kubernetes Storage cluster.

Now I see that the pvc and pv are in the Bound status, so it works as intended. Feel free to close the issue.

Let us know what your digging results are.

I would propose that we could fix this issue and the other at the same time with a ROOK_ALL_HOST_NETWORK config that would set host network on all pods, which would include the provisioners also. We could try to move that higher on the backlog if we become aware of a K8s node mechanism that doesn’t allow egress.

There is another issue that relates to this tangentially: https://github.com/rook/rook/issues/13571

The user who reported the issue linked above is in an environment without a CNI, which is why they want all pods to use host networking.

@ioannisgk 's scenario may be another case where global host network enablement might also be an appropriate user-level config. I can only theorize about the user environment here, but perhaps the K8s cluster disallows egress for some configured security reason.

@obnoxxx had done some work on the global host net work, but I think we may have gotten into PR review fatigue there. @parth-gr (or anyone else) could take that over if desired, but we may need to just add this to Rook’s backlog.

But I don’t think we want to force all Rook users to have host-net-enabled pods that aren’t necessary just to unblock this user’s case.

I find this curious. When I have done external testing, pods are generally able to reach outside the K8s cluster as long as the hosts themselves have their networking configured. I wonder if your k8s cluster has some kind of firewall configured.