rook: unable to get monitor info from DNS SRV with service name: ceph-mon Error in pod when mounting PVC

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior: pods are unable to mount provisioned storage.

Expected behavior: Pods should mount the PVC

How to reproduce it (minimal and precise):

Installed rook-ceph cluster on a K8S cluster with 4 nodes (1 controller + 3 Worker) successfully using the quickstart guide. I am able to see all pods in rook-ceph NS running and no errors found in logs.

[admin@controller195442: ~] $ k get pods -n rook-ceph
NAME                                                         READY   STATUS      RESTARTS     AGE
csi-cephfsplugin-9whvp                                       3/3     Running     0            3d
csi-cephfsplugin-fc5s8                                       3/3     Running     1 (3d ago)   3d
csi-cephfsplugin-provisioner-6fdd9b7dd7-27wd5                6/6     Running     1 (3d ago)   3d
csi-cephfsplugin-provisioner-6fdd9b7dd7-l6r65                6/6     Running     2 (3d ago)   3d
csi-cephfsplugin-tlq9j                                       3/3     Running     1 (3d ago)   3d
csi-cephfsplugin-vp4xv                                       3/3     Running     1 (3d ago)   3d
csi-rbdplugin-5n6ll                                          3/3     Running     0            3d
csi-rbdplugin-95dpg                                          3/3     Running     1 (3d ago)   3d
csi-rbdplugin-mx75w                                          3/3     Running     1 (3d ago)   3d
csi-rbdplugin-provisioner-5d9fb8699c-bngc6                   6/6     Running     3 (3d ago)   3d
csi-rbdplugin-provisioner-5d9fb8699c-zqrw6                   6/6     Running     1 (3d ago)   3d
csi-rbdplugin-s97vx                                          3/3     Running     1 (3d ago)   3d
rook-ceph-crashcollector-controller195442-7b985c998c-qmhgk   1/1     Running     0            3d
rook-ceph-crashcollector-worker195443-5cd88776-hml78         1/1     Running     0            3d
rook-ceph-crashcollector-worker1954466-dff65bc9c-l6lsn       1/1     Running     0            3d
rook-ceph-crashcollector-worker1954467-6bcdb4796d-7bdsm      1/1     Running     0            3d
rook-ceph-exporter-controller195442-7dbdbbc995-6sn88         1/1     Running     0            3d
rook-ceph-exporter-worker195443-8f69f6b48-rn78g              1/1     Running     0            3d
rook-ceph-exporter-worker1954466-b9947b897-r5wxs             1/1     Running     0            3d
rook-ceph-exporter-worker1954467-6d88bdc988-xmr5n            1/1     Running     0            3d
rook-ceph-mds-myfs-a-6f78fb8555-995lm                        2/2     Running     0            3d
rook-ceph-mds-myfs-b-6c59d89685-4pgl7                        2/2     Running     0            3d
rook-ceph-mgr-a-7574779c65-7rbc2                             3/3     Running     0            3d
rook-ceph-mgr-b-599b446ffb-29k74                             3/3     Running     0            3d
rook-ceph-mon-a-7c6cb68c66-4m898                             2/2     Running     0            3d
rook-ceph-mon-b-5cc4bf7bf6-hb8b5                             2/2     Running     0            3d
rook-ceph-mon-c-f7dc9b94-mqcjt                               2/2     Running     0            3d
rook-ceph-operator-7b878c7c4b-5hgjv                          1/1     Running     0            3d1h
rook-ceph-osd-0-8699d8cf8d-g8kdg                             2/2     Running     0            3d
rook-ceph-osd-1-998bcc57-8pn5c                               2/2     Running     0            3d
rook-ceph-osd-2-68577d8479-zpwk4                             2/2     Running     0            3d
rook-ceph-osd-prepare-controller195442-wff5f                 0/1     Completed   0            3d
rook-ceph-osd-prepare-worker195443-mgtbz                     0/1     Completed   0            3d
rook-ceph-osd-prepare-worker1954466-2fmw9                    0/1     Completed   0            3d
rook-ceph-osd-prepare-worker1954467-6rzwd                    0/1     Completed   0            3d
rook-ceph-rgw-my-store-a-65d4855697-t4d7j                    2/2     Running     0            3h51m
rook-ceph-tools-7b67b65bd-wxmkv                              1/1     Running     0            3d
rook-ceph-tools-operator-image-86bdccc87f-ng5fs              1/1     Running     0            145m

Created cephfs using cephfilesystem.txt and cephfs stroageclass using cephsc.txt and pvc using cephfs-pvc.txt

Now I am deploying a demo pod (from deploy/example/csi/pod.yaml) to test cephfs. I expect the pod to be created with the filesystem. but pod is staying in containercreating stage and pod desscription shows “unable to get monitor info from DNS SRV with service name: ceph-mon”

Result: PVCs are successfully bound, but pods get the following error message:

Events:
  Type     Reason                  Age   From                     Message
  ----     ------                  ----  ----                     -------
  Normal   Scheduled               12s   default-scheduler        Successfully assigned default/csicephfs-demo-pod to worker1954466
  Normal   SuccessfulAttachVolume  12s   attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-9a788053-f454-4270-9e1f-783d3756dd6d"
  Warning  FailedMount             10s   kubelet                  MountVolume.MountDevice failed for volume "pvc-9a788053-f454-4270-9e1f-783d3756dd6d" : rpc error: code = Internal desc = an error (exit status 32 occurred while running mount args: [-t ceph 10.100.245.171:6789,10.104.205.146:6789,10.96.21.73:6789:/volumes/csi/csi-vol-82206687-4354-441e-b7f2-8c2a533e9d84/b1a22cf7-4484-453f-b72c-cbd1ddb5e047 /var/lib/kubelt/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/240018be939e31037b18780ad9736432a9e70cc48032c4e1e71d18bc3e0c483a/globalmount -o name=csi-cephfs-node,secretfile=/tmp/csi/keys/keyfile-3097879732,mds_namepace=myfs,_netdev] stderr: unable to get monitor info from DNS SRV with service name: ceph-mon
2024-11-11T10:06:52.478+0000 7f2266ce0000 -1 failed for service _ceph-mon._tcp
mount error 95 = Operation not supported


File(s) to submit:

  • Cluster CR (custom resource), typically called cluster.yaml, if necessary

Logs to submit:

  • Operator’s logs, if necessary

  • Crashing pod(s) logs, if necessary

    To get logs, use kubectl -n <namespace> logs <pod name> When pasting logs, always surround them with backticks or use the insert code button from the Github UI. Read GitHub documentation if you need help.

Cluster Status to submit:

`[admin@controller195442: cephfs] $ k exec -it rook-ceph-tools-operator-image-86bdccc87f-ng5fs -n rook-ceph -- bash
[rook@rook-ceph-tools-operator-image-86bdccc87f-ng5fs /]$ ceph status
  cluster:
    id:     66f79a8e-4f09-4ae2-9bee-b2cf5357484c
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum a,b,c (age 3d)
    mgr: b(active, since 3d), standbys: a
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 3 up (since 3d), 3 in (since 3d)
    rgw: 1 daemon active (1 hosts, 1 zones)

  data:
    volumes: 1/1 healthy
    pools:   11 pools, 137 pgs
    objects: 387 objects, 1.8 MiB
    usage:   407 MiB used, 15 GiB / 15 GiB avail
    pgs:     137 active+clean

  io:
    client:   1.2 KiB/s rd, 2 op/s rd, 0 op/s wr`
  • Output of kubectl commands, if necessary

    To get the health of the cluster, use kubectl rook-ceph health To get the status of the cluster, use kubectl rook-ceph ceph status For more details, see the Rook kubectl Plugin

Environment:

  • OS (e.g. from /etc/os-release):
1 NAME="Ubuntu"
 2 VERSION="18.04.6 LTS (Bionic Beaver)"
 3 ID=ubuntu
 4 ID_LIKE=debian
 5 PRETTY_NAME="Ubuntu 18.04.6 LTS"
 6 VERSION_ID="18.04"
 7 HOME_URL="https://www.ubuntu.com/"
 8 SUPPORT_URL="https://help.ubuntu.com/"
 9 BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
10 PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
11 VERSION_CODENAME=bionic
12 UBUNTU_CODENAME=bionic
  • Kernel (e.g. uname -a): Linux controller195442 4.15.0-1011-fips #12-Ubuntu SMP Tue Jul 2 21:04:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

  • Cloud provider or hardware configuration: 64 GM RAM, 14 vCPUs on each node of cluster

  • Rook version (use rook version inside of a Rook Pod): rook: v1.15.4 go: go1.22.8

  • Storage backend version (e.g. for ceph do ceph -v): ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)

  • Kubernetes version (use kubectl version): `Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.2

  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): kubeadm

  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):

rook@rook-ceph-tools-operator-image-86bdccc87f-ng5fs /]$ ceph health
HEALTH_OK

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 26 (6 by maintainers)

Most upvoted comments

according to what @Madhu-1 said

if its v2 cephfs kernel mount options need to be set.

after setting this option, the issue resolved.

csi:
  cephFSKernelMountOptions: "ms_mode=prefer-crc"