rook: Configure external ceph cluster failed as "failed to run 'ceph version'".

  • Bug Report

I’m installing Rook v1.4 on an Kubernetes v1.18 cluster, which running on 4 bare metal machines with CentOS 8. First I want to use an internal ceph cluster which created by Rook, but I failed, so I create an ceph cluster by myself.

At first I got the following error “cephclient: ceph username is empty”:

image

So I set ceph-username and ceph-secret in rook-ceph-mon secret:

image

And I got another new error:

image

I also entered into rook-ceph-operator pod to execute ceph version command:

image

cluster-external-management.yaml

#################################################################################################################
# Define the settings for the rook-ceph-external cluster with common settings for a production cluster.

# For example, if Rook is not managing any existing cluster in the 'rook-ceph' namespace do:
#   kubectl create -f common.yaml
#   kubectl create -f operator.yaml
#   kubectl create -f cluster-external.yaml

# If there is already a cluster managed by Rook in 'rook-ceph' then do:
#   kubectl create -f common-external.yaml
#   kubectl create -f cluster-external-management.yaml
#################################################################################################################
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  external:
    enable: true
  dataDirHostPath: /var/lib/rook
  # providing an image is required, if you want to create other CRs (rgw, mds, nfs)
  cephVersion:
    image: ceph/ceph:v15.2.4 # Should match external cluster version
  • Operator’s logs
2020-08-17 02:15:43.549630 I | ceph-cluster-controller: detected ceph image version: "15.2.4-0 octopus"
2020-08-17 02:15:43.549660 I | ceph-cluster-controller: validating ceph version from provided image
2020-08-17 02:15:43.554853 I | op-mon: parsing mon endpoints: test-kubenode-1=10.28.112.11,test-kubenode-2=10.28.112.12,test-kubenode-3=10.28.112.13
2020-08-17 02:15:43.555167 E | op-ceph-util: failed to split ip and port for endpoint "10.28.112.11". address 10.28.112.11: missing port in address
2020-08-17 02:15:43.555193 E | op-ceph-util: failed to split host and port for endpoint "10.28.112.11", assuming default Ceph port "". address 10.28.112.11: missing port in address
2020-08-17 02:15:43.555202 E | op-ceph-util: failed to split ip and port for endpoint "10.28.112.12". address 10.28.112.12: missing port in address
2020-08-17 02:15:43.555210 E | op-ceph-util: failed to split host and port for endpoint "10.28.112.12", assuming default Ceph port "". address 10.28.112.12: missing port in address
2020-08-17 02:15:43.555222 E | op-ceph-util: failed to split ip and port for endpoint "10.28.112.13". address 10.28.112.13: missing port in address
2020-08-17 02:15:43.555230 E | op-ceph-util: failed to split host and port for endpoint "10.28.112.13", assuming default Ceph port "". address 10.28.112.13: missing port in address
2020-08-17 02:15:43.555321 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-08-17 02:15:43.555504 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-08-17 02:15:43.715272 E | ceph-cluster-controller: failed to reconcile. failed to reconcile cluster "rook-ceph": failed to configure external ceph cluster: failed to detect and validate ceph version: failed to validate ceph version between external and local: failed to get ceph mon version: failed to run 'ceph version': exit status 1

Environment:

  • OS (e.g. from /etc/os-release):
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"
  • Kernel (e.g. uname -a):
Linux test-kubenode-1 4.18.0-193.6.3.el8_2.x86_64 #1 SMP Wed Jun 10 11:09:32 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
  • Rook version (use rook version inside of a Rook Pod):
rook: v1.4.0
go: go1.13.8
  • Storage backend version (e.g. for ceph do ceph -v):
ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)
  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): Created by Rancher

  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): HEALTH_OK

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 15 (5 by maintainers)

Most upvoted comments

In case someone else stumbles into this issue via Google… I had a similar issue:

root@kube-cp01-dt:~/rook/deploy/examples# kubectl -n rook-ceph logs pod/rook-ceph-operator-76595f69d-wnvl8
2024/11/05 21:44:18 maxprocs: Leaving GOMAXPROCS=16: CPU quota undefined
2024-11-05 21:44:18.628761 I | rookcmd: starting Rook v1.15.4 with arguments '/usr/local/bin/rook ceph operator'
2024-11-05 21:44:18.628817 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO
2024-11-05 21:44:18.628826 I | cephcmd: starting Rook-Ceph operator
2024-11-05 21:44:18.656001 E | cephcmd: failed to get operator base image ceph version. failed to execute command to detect ceph version: exit status 127
2024-11-05 21:44:18.656044 I | cephcmd: base ceph version inside the rook operator image is ""

“exit status 127” when the operator would try and detect the ceph version… seen again like this:

root@kube-cp01-dt:~/rook/deploy/examples# kubectl exec --stdin --tty -n rook-ceph pod/rook-ceph-operator-76595f69d-wnvl8 -- bash
Fatal glibc error: CPU does not support x86-64-v2
command terminated with exit code 127
root@kube-cp01-dt:~/rook/deploy/examples#

The error was actually Fatal glibc error: CPU does not support x86-64-v2

I have the node living on a ProxMox hypervisor. I was using the qemu processor type (the default). Switching the processor type to host on my worker nodes and control plane (that in my lab is also a worker node), corrected the issue for me. Found via https://support.regulaforensics.com/hc/en-us/articles/20319657014929-CPU-doesn-t-support-image-x86-64-v2-error-using-Proxmox

I solved this issue by getting the parameters it wants with create-external-cluster-resources.sh and setting ROOK_EXTERNAL_ADMIN_SECRET to admin-secret

I set my namespaces to rook-ceph because I don’t have an existing internal cluster. This meant I didn’t have to run common-external.yaml and didn’t have an extra namespace but I had to set the namespace to rook-ceph in a few places (cephcluster, storageclass…). Either way works.

I solved it too, thanks @user-name-is-taken

Here is the environment:

  • Ceph v14.2.11 (External only)
  • OKD 4.5.0-0.okd-2020-09-04-180756 (FCOS 32.20200629.3.0, Kubernetes Version: v1.18.3)
  • Rook v1.4.3

Here are the steps I performed:

oc create -f common.yaml
oc create -f operator-openshift.yaml
oc create -f common-external.yaml
export NAMESPACE=rook-ceph-external
export ROOK_EXTERNAL_FSID=ae910299-954d-4087-823d-56cca50bdf3c
export ROOK_EXTERNAL_CEPH_MON_DATA=ceph-4=30.30.103.146:6789
export ROOK_EXTERNAL_ADMIN_SECRET=AQAOQ.....==
# Following six environments are output of create-external-cluster-resources.sh
export ROOK_EXTERNAL_USER_SECRET=AQAbM.....==
export ROOK_EXTERNAL_USERNAME=client.healthchecker
export CSI_RBD_NODE_SECRET_SECRET=AQAbM.....==
export CSI_RBD_PROVISIONER_SECRET=AQAcM.....==
export CSI_CEPHFS_NODE_SECRET=AQAcM.....==
export CSI_CEPHFS_PROVISIONER_SECRET=AQAdM.....==
sed "s/kubectl/oc/" import-external-cluster.sh > tmp.sh
sh tmp.sh
oc create -f cluster-external-management.yaml
oc crete -f storageclass-rbd.yml
oc create -f storageclass-cephfs.yml
oc patch storageclass rbd -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'

Modified cluster-external-management.yaml:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph-external
  namespace: rook-ceph-external
spec:
  external:
    enable: true
  # add crashCollector value, it's not present on my environment
  crashCollector:
    disable: true
  dataDirHostPath: /var/lib/rook
  # providing an image is required, if you want to create other CRs (rgw, mds, nfs)
  cephVersion:
    image: ceph/ceph:v15.2.4 # Should match external cluster version

Create a RBD Storage Class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rbd
# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
    # clusterID is the namespace where the rook cluster is running
    clusterID: rook-ceph-external
    # Ceph pool into which the RBD image shall be created
    pool: okd

    # RBD image format. Defaults to "2".
    imageFormat: "2"

    # RBD image features. Available for imageFormat: "2". CSI RBD currently supports only `layering` feature.
    imageFeatures: layering

    # The secrets contain Ceph admin credentials.
    csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
    csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph-external
    csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
    csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph-external
    csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
    csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph-external

    # Specify the filesystem type of the volume. If not specified, csi-provisioner
    # will set default as `ext4`. Note that `xfs` is not recommended due to potential deadlock
    # in hyperconverged settings where the volume is mounted on the same node as the osds.
    csi.storage.k8s.io/fstype: ext4

# Delete the rbd volume when a PVC is deleted
reclaimPolicy: Delete
allowVolumeExpansion: true

Create a CephFS Storage Class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cephfs
# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
  # clusterID is the namespace where operator is deployed.
  clusterID: rook-ceph-external

  # CephFS filesystem name into which the volume shall be created
  fsName: cephfs

  # Ceph pool into which the volume shall be created
  # Required for provisionVolume: "true"
  #pool: myfs-data0

  # Root path of an existing CephFS volume
  # Required for provisionVolume: "false"
  # rootPath: /absolute/path

  # The secrets contain Ceph admin credentials. These are generated automatically by the operator
  # in the same namespace as the cluster.
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph-external
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph-external
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph-external

reclaimPolicy: Delete