rook: MDS liveness probe script failing to execute

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior: The livenessprobe.sh script used by the mds container always fails to execute correctly due to incorrect authentication when calling the “ceph fs dump” command.

Expected behavior: The livenessprobe.sh should correctly execute with proper authentication so the liveness can be asserted. The “ceph fs dump” command in the mds liveness probe script should be successful when the cluster is healthy.

How to reproduce it (minimal and precise):

  1. Describe the MDS pod and copy the livenessprobe command/script called with exec.
  2. Enter the mds container and create that same script into a file:
[root@rook-ceph-mds-kube-cephfs-a-65f8485847-wrnsm ceph]# cat livenessprobe-test.sh 
#!/usr/bin/env bash
# do not use 'set -e -u' etc. because it is important to only fail this probe when failure is certain
# spurious failures risk destabilizing ceph or the filesystem

MDS_ID="kube-cephfs-a"
FILESYSTEM_NAME="kube-cephfs"
KEYRING="/etc/ceph/keyring-store/keyring"

outp="$(ceph fs dump --mon-host="$ROOK_CEPH_MON_HOST" --mon-initial-members="$ROOK_CEPH_MON_INITIAL_MEMBERS" --keyring "$KEYRING" --format json)"
rc=$?
if [ $rc -ne 0 ]; then
    echo "ceph MDS dump check failed with the following output:"
    echo "$outp"
    echo "passing probe to avoid restarting MDS. cannot determine if MDS is unhealthy. restarting MDS risks destabilizing ceph/filesystem, which is likely unreachable or in error state"
    exit 0
fi

# get the active and standby MDS in the fs map
standbyMds=$(echo "$outp" | jq ".standbys | map(.name) | any(.[]; . == \"$MDS_ID\")")
activeMds=$(echo "$outp" | jq ".filesystems[] | select(.mdsmap.fs_name == \"$FILESYSTEM_NAME\") | .mdsmap.info | map(.name) | any(.[]; . == \"$MDS_ID\")")

if [[ $standbyMds == true || $activeMds == true ]]; then
    echo "MDS ID present in MDS map, no need to re-start the container"
    exit 0
fi

echo "Error: MDS ID not present in MDS map"
exit 1
  1. Execute the script inside the container:
[root@rook-ceph-mds-kube-cephfs-a-65f8485847-wrnsm ceph]# ./livenessprobe-test.sh 
2025-02-05T14:50:45.548+0000 7f23686ac700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
2025-02-05T14:50:45.548+0000 7f23686ac700 -1 AuthRegistry(0x7f23600647f8) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx
2025-02-05T14:50:45.557+0000 7f2366448700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
[errno 13] RADOS permission denied (error connecting to the cluster)
ceph MDS dump check failed with the following output:

passing probe to avoid restarting MDS. cannot determine if MDS is unhealthy. restarting MDS risks destabilizing ceph/filesystem, which is likely unreachable or in error state

Notice it fails the “ceph fs dump” command while looking fir the keyring and shows no output inside of $outp.

  1. Execute the “ceph fs dump” command outside the container to verify that the mds is actually ok:
$ ceph fs dump
e6
enable_multiple, ever_enabled_multiple: 1,1
default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
 
Filesystem 'kube-cephfs' (1)
fs_name	kube-cephfs
epoch	6
flags	32 joinable allow_snaps allow_multimds_snaps allow_standby_replay
created	2025-02-05T13:53:38.856768+0000
modified	2025-02-05T13:53:39.894174+0000
tableserver	0
root	0
session_timeout	60
session_autoclose	300
max_file_size	1099511627776
required_client_features	{}
last_failure	0
last_failure_osd_epoch	0
compat	compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds	1
in	0
up	{0=4293}
failed	
damaged	
stopped	
data_pools	[4]
metadata_pool	3
inline_data	disabled
balancer	
bal_rank_mask	-1
standby_count_wanted	1
[mds.kube-cephfs-b{0:4293} state up:active seq 5 join_fscid=1 addr [v2:172.16.192.64:6800/4126807878,v1:172.16.192.64:6801/4126807878] compat {c=[1],r=[1],i=[7ff]}]
[mds.kube-cephfs-a{0:4281} state up:standby-replay seq 5 join_fscid=1 addr [v2:172.16.192.69:6800/3599992135,v1:172.16.192.69:6801/3599992135] compat {c=[1],r=[1],i=[7ff]}]
 
 
dumped fsmap epoch 6

How to fix it:

  1. Inside the created livenessprobe-test.sh script, add the mds client name to the “ceph fs dump” command and pass in the “–keyring $KEYRING” argument as part of the exported variable CEPH_ARGS, resulting in the following:
[root@rook-ceph-mds-kube-cephfs-a-65f8485847-wrnsm ceph]# cat livenessprobe-test.sh 
#!/usr/bin/env bash
# do not use 'set -e -u' etc. because it is important to only fail this probe when failure is certain
# spurious failures risk destabilizing ceph or the filesystem

MDS_ID="kube-cephfs-a"
FILESYSTEM_NAME="kube-cephfs"
KEYRING="/etc/ceph/keyring-store/keyring"

export CEPH_ARGS="--keyring $KEYRING"

outp="$(ceph fs dump -n "mds.$MDS_ID" --mon-host="$ROOK_CEPH_MON_HOST" --mon-initial-members="$ROOK_CEPH_MON_INITIAL_MEMBERS" --format json)"
rc=$?
if [ $rc -ne 0 ]; then
    echo "ceph MDS dump check failed with the following output:"
    echo "$outp"
    echo "passing probe to avoid restarting MDS. cannot determine if MDS is unhealthy. restarting MDS risks destabilizing ceph/filesystem, which is likely unreachable or in error state"
    exit 0
fi

# get the active and standby MDS in the fs map
standbyMds=$(echo "$outp" | jq ".standbys | map(.name) | any(.[]; . == \"$MDS_ID\")")
activeMds=$(echo "$outp" | jq ".filesystems[] | select(.mdsmap.fs_name == \"$FILESYSTEM_NAME\") | .mdsmap.info | map(.name) | any(.[]; . == \"$MDS_ID\")")

if [[ $standbyMds == true || $activeMds == true ]]; then
    echo "MDS ID present in MDS map, no need to re-start the container"
    exit 0
fi

echo "Error: MDS ID not present in MDS map"
exit 1
  1. Execute the script and verify that it works correctly now:
[root@rook-ceph-mds-kube-cephfs-a-65f8485847-wrnsm ceph]# ./livenessprobe-test.sh 
dumped fsmap epoch 6
MDS ID present in MDS map, no need to re-start the container

Cluster Status to submit:

$ ceph status
  cluster:
    id:     e410b695-e6b7-4f65-b162-a11e37bf1057
    health: HEALTH_OK
 
  services:
    mon: 1 daemons, quorum a (age 68m)
    mgr: a(active, since 66m)
    mds: 1/1 daemons up, 1 hot standby
    osd: 1 osds: 1 up (since 67m), 1 in (since 67m)
 
  data:
    volumes: 1/1 healthy
    pools:   4 pools, 81 pgs
    objects: 22 objects, 596 KiB
    usage:   27 MiB used, 9.7 GiB / 9.8 GiB avail
    pgs:     81 active+clean
 
  io:
    client:   1.2 KiB/s rd, 2 op/s rd, 0 op/s wr

Environment:

  • OS (e.g. from /etc/os-release): Debian GNU/Linux 11 (bullseye)
  • Kernel (e.g. uname -a): Linux controller-0 6.6.0-1-amd64 #1 SMP PREEMPT_DYNAMIC
  • Rook version (use rook version inside of a Rook Pod): 1.13.7
  • Storage backend version (e.g. for ceph do ceph -v): ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)
  • Kubernetes version (use kubectl version): v1.29.2
  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): HEALTH_OK

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 30 (25 by maintainers)

Commits related to this issue

Most upvoted comments

I think it might be an alternative.

@parth-gr not sure if I’m the right person to mention in this issue … or it is just a mistake!

@parth-gr

Yes, you need to use both the -n "mds.$MDS_ID" and export CEPH_ARGS="--keyring $KEYRING" to get rid of all the errors, as you showed.

Thank you for investigating it further @parth-gr ! I appreciate it

So the summary is inside the container,

This command is failing

ceph fs dump -n "mds.$MDS_ID" --mon-host="$ROOK_CEPH_MON_HOST" --mon-initial-members="$ROOK_CEPH_MON_INITIAL_MEMBERS" --keyring "$KEYRING" --format json

with

2025-02-05T13:51:35.190+0000 7fe51e5e1700 -1 AuthRegistry(0x7fe518064ab8) no keyring found at /var/lib/ceph/mds/ceph-kube-cephfs-a/keyring, disabling cephx

But the below way it worked

export CEPH_ARGS="--keyring $KEYRING"
ceph fs dump -n "mds.$MDS_ID" --mon-host="$ROOK_CEPH_MON_HOST" --mon-initial-members="$ROOK_CEPH_MON_INITIAL_MEMBERS" --format json

I would like to get some initial suggestions from Ceph team if they have seen this earlier cc @batrick @rzarzynski

Hi @BlaineEXE !

I am actually running rook-ceph in StarlingX, so it is not easy to change versions for rook or ceph. StarlingX has vanilla kubernetes.

I found this issue reported in the past about the same problem:

As you can see in my demonstration running the script inside the mds pod, it fails because it cannot find a keyring referencing “client.admin.keyring” even though we are specifying a keyring as an argument: --keyring "$KEYRING"

Since it is a keyring problem when running the ceph fs dump command, all we need to do to fix it, is add the mds client name as an argument: -n "mds.$MDS_ID"

This makes the keyring specified to be accepted for this specific mds client, as we can see below.

Script with the mds client name:

[root@rook-ceph-mds-kube-cephfs-a-6548bbb564-tjsgp ceph]# cat livenessprobe-test.sh 
#!/usr/bin/env bash
# do not use 'set -e -u' etc. because it is important to only fail this probe when failure is certain
# spurious failures risk destabilizing ceph or the filesystem

MDS_ID="kube-cephfs-a"
FILESYSTEM_NAME="kube-cephfs"
KEYRING="/etc/ceph/keyring-store/keyring"

outp="$(ceph fs dump -n "mds.$MDS_ID" --mon-host="$ROOK_CEPH_MON_HOST" --mon-initial-members="$ROOK_CEPH_MON_INITIAL_MEMBERS" --keyring "$KEYRING" --format json)"
rc=$?
if [ $rc -ne 0 ]; then
    echo "ceph MDS dump check failed with the following output:"
    echo "$outp"
    echo "passing probe to avoid restarting MDS. cannot determine if MDS is unhealthy. restarting MDS risks destabilizing ceph/filesystem, which is likely unreachable or in error state"
    exit 0
fi

# get the active and standby MDS in the fs map
standbyMds=$(echo "$outp" | jq ".standbys | map(.name) | any(.[]; . == \"$MDS_ID\")")
activeMds=$(echo "$outp" | jq ".filesystems[] | select(.mdsmap.fs_name == \"$FILESYSTEM_NAME\") | .mdsmap.info | map(.name) | any(.[]; . == \"$MDS_ID\")")

if [[ $standbyMds == true || $activeMds == true ]]; then
    echo "MDS ID present in MDS map, no need to re-start the container"
    exit 0
fi

echo "Error: MDS ID not present in MDS map"
exit 1

Running the script:

[root@rook-ceph-mds-kube-cephfs-a-6548bbb564-tjsgp ceph]# ./livenessprobe-test.sh 
2025-02-05T13:51:35.190+0000 7fe51e5e1700 -1 auth: unable to find a keyring on /var/lib/ceph/mds/ceph-kube-cephfs-a/keyring: (2) No such file or directory
2025-02-05T13:51:35.190+0000 7fe51e5e1700 -1 AuthRegistry(0x7fe518064ab8) no keyring found at /var/lib/ceph/mds/ceph-kube-cephfs-a/keyring, disabling cephx
dumped fsmap epoch 7
MDS ID present in MDS map, no need to re-start the container

Notice that even though it works, first the ceph fs dump command looks for another keyring, fails, and than proceeds to use the keyring specified by us in --keyring "$KEYRING", that is why we see errors.

To bypass these errors and force ceph fs dump to look for our specified keyring, we have to place the --keyringoption in the CEPH_ARGS variable and export it, so that ceph fs dump can use it in this context. This makes the ceph command to use our argument as its first keyring option and not as a backup option.

I hope this helps to clarify the issue and why I tried this type of fix.

Thank you for your quick response!