rook: unable to get monitor info from DNS SRV with service name: ceph-mon Error in pod when mounting PVC
Is this a bug report or feature request?
- Bug Report
Deviation from expected behavior: pods are unable to mount provisioned storage.
Expected behavior: Pods should mount the PVC
How to reproduce it (minimal and precise):
Installed rook-ceph cluster on a K8S cluster with 4 nodes (1 controller + 3 Worker) successfully using the quickstart guide. I am able to see all pods in rook-ceph NS running and no errors found in logs.
[admin@controller195442: ~] $ k get pods -n rook-ceph
NAME READY STATUS RESTARTS AGE
csi-cephfsplugin-9whvp 3/3 Running 0 3d
csi-cephfsplugin-fc5s8 3/3 Running 1 (3d ago) 3d
csi-cephfsplugin-provisioner-6fdd9b7dd7-27wd5 6/6 Running 1 (3d ago) 3d
csi-cephfsplugin-provisioner-6fdd9b7dd7-l6r65 6/6 Running 2 (3d ago) 3d
csi-cephfsplugin-tlq9j 3/3 Running 1 (3d ago) 3d
csi-cephfsplugin-vp4xv 3/3 Running 1 (3d ago) 3d
csi-rbdplugin-5n6ll 3/3 Running 0 3d
csi-rbdplugin-95dpg 3/3 Running 1 (3d ago) 3d
csi-rbdplugin-mx75w 3/3 Running 1 (3d ago) 3d
csi-rbdplugin-provisioner-5d9fb8699c-bngc6 6/6 Running 3 (3d ago) 3d
csi-rbdplugin-provisioner-5d9fb8699c-zqrw6 6/6 Running 1 (3d ago) 3d
csi-rbdplugin-s97vx 3/3 Running 1 (3d ago) 3d
rook-ceph-crashcollector-controller195442-7b985c998c-qmhgk 1/1 Running 0 3d
rook-ceph-crashcollector-worker195443-5cd88776-hml78 1/1 Running 0 3d
rook-ceph-crashcollector-worker1954466-dff65bc9c-l6lsn 1/1 Running 0 3d
rook-ceph-crashcollector-worker1954467-6bcdb4796d-7bdsm 1/1 Running 0 3d
rook-ceph-exporter-controller195442-7dbdbbc995-6sn88 1/1 Running 0 3d
rook-ceph-exporter-worker195443-8f69f6b48-rn78g 1/1 Running 0 3d
rook-ceph-exporter-worker1954466-b9947b897-r5wxs 1/1 Running 0 3d
rook-ceph-exporter-worker1954467-6d88bdc988-xmr5n 1/1 Running 0 3d
rook-ceph-mds-myfs-a-6f78fb8555-995lm 2/2 Running 0 3d
rook-ceph-mds-myfs-b-6c59d89685-4pgl7 2/2 Running 0 3d
rook-ceph-mgr-a-7574779c65-7rbc2 3/3 Running 0 3d
rook-ceph-mgr-b-599b446ffb-29k74 3/3 Running 0 3d
rook-ceph-mon-a-7c6cb68c66-4m898 2/2 Running 0 3d
rook-ceph-mon-b-5cc4bf7bf6-hb8b5 2/2 Running 0 3d
rook-ceph-mon-c-f7dc9b94-mqcjt 2/2 Running 0 3d
rook-ceph-operator-7b878c7c4b-5hgjv 1/1 Running 0 3d1h
rook-ceph-osd-0-8699d8cf8d-g8kdg 2/2 Running 0 3d
rook-ceph-osd-1-998bcc57-8pn5c 2/2 Running 0 3d
rook-ceph-osd-2-68577d8479-zpwk4 2/2 Running 0 3d
rook-ceph-osd-prepare-controller195442-wff5f 0/1 Completed 0 3d
rook-ceph-osd-prepare-worker195443-mgtbz 0/1 Completed 0 3d
rook-ceph-osd-prepare-worker1954466-2fmw9 0/1 Completed 0 3d
rook-ceph-osd-prepare-worker1954467-6rzwd 0/1 Completed 0 3d
rook-ceph-rgw-my-store-a-65d4855697-t4d7j 2/2 Running 0 3h51m
rook-ceph-tools-7b67b65bd-wxmkv 1/1 Running 0 3d
rook-ceph-tools-operator-image-86bdccc87f-ng5fs 1/1 Running 0 145m
Created cephfs using cephfilesystem.txt and cephfs stroageclass using cephsc.txt and pvc using cephfs-pvc.txt
Now I am deploying a demo pod (from deploy/example/csi/pod.yaml) to test cephfs. I expect the pod to be created with the filesystem. but pod is staying in containercreating stage and pod desscription shows “unable to get monitor info from DNS SRV with service name: ceph-mon”
Result: PVCs are successfully bound, but pods get the following error message:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 12s default-scheduler Successfully assigned default/csicephfs-demo-pod to worker1954466
Normal SuccessfulAttachVolume 12s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-9a788053-f454-4270-9e1f-783d3756dd6d"
Warning FailedMount 10s kubelet MountVolume.MountDevice failed for volume "pvc-9a788053-f454-4270-9e1f-783d3756dd6d" : rpc error: code = Internal desc = an error (exit status 32 occurred while running mount args: [-t ceph 10.100.245.171:6789,10.104.205.146:6789,10.96.21.73:6789:/volumes/csi/csi-vol-82206687-4354-441e-b7f2-8c2a533e9d84/b1a22cf7-4484-453f-b72c-cbd1ddb5e047 /var/lib/kubelt/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/240018be939e31037b18780ad9736432a9e70cc48032c4e1e71d18bc3e0c483a/globalmount -o name=csi-cephfs-node,secretfile=/tmp/csi/keys/keyfile-3097879732,mds_namepace=myfs,_netdev] stderr: unable to get monitor info from DNS SRV with service name: ceph-mon
2024-11-11T10:06:52.478+0000 7f2266ce0000 -1 failed for service _ceph-mon._tcp
mount error 95 = Operation not supported
File(s) to submit:
- Cluster CR (custom resource), typically called
cluster.yaml, if necessary
Logs to submit:
-
Operator’s logs, if necessary
-
Crashing pod(s) logs, if necessary
To get logs, use
kubectl -n <namespace> logs <pod name>When pasting logs, always surround them with backticks or use theinsert codebutton from the Github UI. Read GitHub documentation if you need help.
Cluster Status to submit:
`[admin@controller195442: cephfs] $ k exec -it rook-ceph-tools-operator-image-86bdccc87f-ng5fs -n rook-ceph -- bash
[rook@rook-ceph-tools-operator-image-86bdccc87f-ng5fs /]$ ceph status
cluster:
id: 66f79a8e-4f09-4ae2-9bee-b2cf5357484c
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 3d)
mgr: b(active, since 3d), standbys: a
mds: 1/1 daemons up, 1 hot standby
osd: 3 osds: 3 up (since 3d), 3 in (since 3d)
rgw: 1 daemon active (1 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 11 pools, 137 pgs
objects: 387 objects, 1.8 MiB
usage: 407 MiB used, 15 GiB / 15 GiB avail
pgs: 137 active+clean
io:
client: 1.2 KiB/s rd, 2 op/s rd, 0 op/s wr`
-
Output of kubectl commands, if necessary
To get the health of the cluster, use
kubectl rook-ceph healthTo get the status of the cluster, usekubectl rook-ceph ceph statusFor more details, see the Rook kubectl Plugin
Environment:
- OS (e.g. from /etc/os-release):
1 NAME="Ubuntu"
2 VERSION="18.04.6 LTS (Bionic Beaver)"
3 ID=ubuntu
4 ID_LIKE=debian
5 PRETTY_NAME="Ubuntu 18.04.6 LTS"
6 VERSION_ID="18.04"
7 HOME_URL="https://www.ubuntu.com/"
8 SUPPORT_URL="https://help.ubuntu.com/"
9 BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
10 PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
11 VERSION_CODENAME=bionic
12 UBUNTU_CODENAME=bionic
-
Kernel (e.g.
uname -a):Linux controller195442 4.15.0-1011-fips #12-Ubuntu SMP Tue Jul 2 21:04:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux -
Cloud provider or hardware configuration: 64 GM RAM, 14 vCPUs on each node of cluster
-
Rook version (use
rook versioninside of a Rook Pod): rook: v1.15.4 go: go1.22.8 -
Storage backend version (e.g. for ceph do
ceph -v): ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) -
Kubernetes version (use
kubectl version): `Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.2 -
Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): kubeadm
-
Storage backend status (e.g. for Ceph use
ceph healthin the Rook Ceph toolbox):
rook@rook-ceph-tools-operator-image-86bdccc87f-ng5fs /]$ ceph health
HEALTH_OK
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 26 (6 by maintainers)
according to what @Madhu-1 said
after setting this option, the issue resolved.