rook: Configure external ceph cluster failed as "failed to run 'ceph version'".
- Bug Report
I’m installing Rook v1.4 on an Kubernetes v1.18 cluster, which running on 4 bare metal machines with CentOS 8. First I want to use an internal ceph cluster which created by Rook, but I failed, so I create an ceph cluster by myself.
At first I got the following error “cephclient: ceph username is empty”:

So I set ceph-username and ceph-secret in rook-ceph-mon secret:

And I got another new error:

I also entered into rook-ceph-operator pod to execute ceph version command:

cluster-external-management.yaml
#################################################################################################################
# Define the settings for the rook-ceph-external cluster with common settings for a production cluster.
# For example, if Rook is not managing any existing cluster in the 'rook-ceph' namespace do:
# kubectl create -f common.yaml
# kubectl create -f operator.yaml
# kubectl create -f cluster-external.yaml
# If there is already a cluster managed by Rook in 'rook-ceph' then do:
# kubectl create -f common-external.yaml
# kubectl create -f cluster-external-management.yaml
#################################################################################################################
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
name: rook-ceph
namespace: rook-ceph
spec:
external:
enable: true
dataDirHostPath: /var/lib/rook
# providing an image is required, if you want to create other CRs (rgw, mds, nfs)
cephVersion:
image: ceph/ceph:v15.2.4 # Should match external cluster version
- Operator’s logs
2020-08-17 02:15:43.549630 I | ceph-cluster-controller: detected ceph image version: "15.2.4-0 octopus"
2020-08-17 02:15:43.549660 I | ceph-cluster-controller: validating ceph version from provided image
2020-08-17 02:15:43.554853 I | op-mon: parsing mon endpoints: test-kubenode-1=10.28.112.11,test-kubenode-2=10.28.112.12,test-kubenode-3=10.28.112.13
2020-08-17 02:15:43.555167 E | op-ceph-util: failed to split ip and port for endpoint "10.28.112.11". address 10.28.112.11: missing port in address
2020-08-17 02:15:43.555193 E | op-ceph-util: failed to split host and port for endpoint "10.28.112.11", assuming default Ceph port "". address 10.28.112.11: missing port in address
2020-08-17 02:15:43.555202 E | op-ceph-util: failed to split ip and port for endpoint "10.28.112.12". address 10.28.112.12: missing port in address
2020-08-17 02:15:43.555210 E | op-ceph-util: failed to split host and port for endpoint "10.28.112.12", assuming default Ceph port "". address 10.28.112.12: missing port in address
2020-08-17 02:15:43.555222 E | op-ceph-util: failed to split ip and port for endpoint "10.28.112.13". address 10.28.112.13: missing port in address
2020-08-17 02:15:43.555230 E | op-ceph-util: failed to split host and port for endpoint "10.28.112.13", assuming default Ceph port "". address 10.28.112.13: missing port in address
2020-08-17 02:15:43.555321 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-08-17 02:15:43.555504 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-08-17 02:15:43.715272 E | ceph-cluster-controller: failed to reconcile. failed to reconcile cluster "rook-ceph": failed to configure external ceph cluster: failed to detect and validate ceph version: failed to validate ceph version between external and local: failed to get ceph mon version: failed to run 'ceph version': exit status 1
Environment:
- OS (e.g. from /etc/os-release):
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"
- Kernel (e.g.
uname -a):
Linux test-kubenode-1 4.18.0-193.6.3.el8_2.x86_64 #1 SMP Wed Jun 10 11:09:32 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
- Rook version (use
rook versioninside of a Rook Pod):
rook: v1.4.0
go: go1.13.8
- Storage backend version (e.g. for ceph do
ceph -v):
ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)
- Kubernetes version (use
kubectl version):
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
-
Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): Created by Rancher
-
Storage backend status (e.g. for Ceph use
ceph healthin the Rook Ceph toolbox): HEALTH_OK
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 15 (5 by maintainers)
In case someone else stumbles into this issue via Google… I had a similar issue:
“exit status 127” when the operator would try and detect the ceph version… seen again like this:
The error was actually
Fatal glibc error: CPU does not support x86-64-v2I have the node living on a ProxMox hypervisor. I was using the
qemuprocessor type (the default). Switching the processor type tohoston my worker nodes and control plane (that in my lab is also a worker node), corrected the issue for me. Found via https://support.regulaforensics.com/hc/en-us/articles/20319657014929-CPU-doesn-t-support-image-x86-64-v2-error-using-ProxmoxI solved this issue by getting the parameters it wants with create-external-cluster-resources.sh and setting
ROOK_EXTERNAL_ADMIN_SECRETtoadmin-secretI set my namespaces to rook-ceph because I don’t have an existing internal cluster. This meant I didn’t have to run
common-external.yamland didn’t have an extra namespace but I had to set the namespace torook-cephin a few places (cephcluster, storageclass…). Either way works.I solved it too, thanks @user-name-is-taken
Here is the environment:
Here are the steps I performed:
Modified cluster-external-management.yaml:
Create a RBD Storage Class:
Create a CephFS Storage Class: