rook: MDS liveness probe script failing to execute
Is this a bug report or feature request?
- Bug Report
Deviation from expected behavior: The livenessprobe.sh script used by the mds container always fails to execute correctly due to incorrect authentication when calling the “ceph fs dump” command.
Expected behavior: The livenessprobe.sh should correctly execute with proper authentication so the liveness can be asserted. The “ceph fs dump” command in the mds liveness probe script should be successful when the cluster is healthy.
How to reproduce it (minimal and precise):
- Describe the MDS pod and copy the livenessprobe command/script called with exec.
- Enter the mds container and create that same script into a file:
[root@rook-ceph-mds-kube-cephfs-a-65f8485847-wrnsm ceph]# cat livenessprobe-test.sh
#!/usr/bin/env bash
# do not use 'set -e -u' etc. because it is important to only fail this probe when failure is certain
# spurious failures risk destabilizing ceph or the filesystem
MDS_ID="kube-cephfs-a"
FILESYSTEM_NAME="kube-cephfs"
KEYRING="/etc/ceph/keyring-store/keyring"
outp="$(ceph fs dump --mon-host="$ROOK_CEPH_MON_HOST" --mon-initial-members="$ROOK_CEPH_MON_INITIAL_MEMBERS" --keyring "$KEYRING" --format json)"
rc=$?
if [ $rc -ne 0 ]; then
echo "ceph MDS dump check failed with the following output:"
echo "$outp"
echo "passing probe to avoid restarting MDS. cannot determine if MDS is unhealthy. restarting MDS risks destabilizing ceph/filesystem, which is likely unreachable or in error state"
exit 0
fi
# get the active and standby MDS in the fs map
standbyMds=$(echo "$outp" | jq ".standbys | map(.name) | any(.[]; . == \"$MDS_ID\")")
activeMds=$(echo "$outp" | jq ".filesystems[] | select(.mdsmap.fs_name == \"$FILESYSTEM_NAME\") | .mdsmap.info | map(.name) | any(.[]; . == \"$MDS_ID\")")
if [[ $standbyMds == true || $activeMds == true ]]; then
echo "MDS ID present in MDS map, no need to re-start the container"
exit 0
fi
echo "Error: MDS ID not present in MDS map"
exit 1
- Execute the script inside the container:
[root@rook-ceph-mds-kube-cephfs-a-65f8485847-wrnsm ceph]# ./livenessprobe-test.sh
2025-02-05T14:50:45.548+0000 7f23686ac700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
2025-02-05T14:50:45.548+0000 7f23686ac700 -1 AuthRegistry(0x7f23600647f8) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx
2025-02-05T14:50:45.557+0000 7f2366448700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
[errno 13] RADOS permission denied (error connecting to the cluster)
ceph MDS dump check failed with the following output:
passing probe to avoid restarting MDS. cannot determine if MDS is unhealthy. restarting MDS risks destabilizing ceph/filesystem, which is likely unreachable or in error state
Notice it fails the “ceph fs dump” command while looking fir the keyring and shows no output inside of $outp.
- Execute the “ceph fs dump” command outside the container to verify that the mds is actually ok:
$ ceph fs dump
e6
enable_multiple, ever_enabled_multiple: 1,1
default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'kube-cephfs' (1)
fs_name kube-cephfs
epoch 6
flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay
created 2025-02-05T13:53:38.856768+0000
modified 2025-02-05T13:53:39.894174+0000
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
required_client_features {}
last_failure 0
last_failure_osd_epoch 0
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds 1
in 0
up {0=4293}
failed
damaged
stopped
data_pools [4]
metadata_pool 3
inline_data disabled
balancer
bal_rank_mask -1
standby_count_wanted 1
[mds.kube-cephfs-b{0:4293} state up:active seq 5 join_fscid=1 addr [v2:172.16.192.64:6800/4126807878,v1:172.16.192.64:6801/4126807878] compat {c=[1],r=[1],i=[7ff]}]
[mds.kube-cephfs-a{0:4281} state up:standby-replay seq 5 join_fscid=1 addr [v2:172.16.192.69:6800/3599992135,v1:172.16.192.69:6801/3599992135] compat {c=[1],r=[1],i=[7ff]}]
dumped fsmap epoch 6
How to fix it:
- Inside the created livenessprobe-test.sh script, add the mds client name to the “ceph fs dump” command and pass in the “–keyring $KEYRING” argument as part of the exported variable CEPH_ARGS, resulting in the following:
[root@rook-ceph-mds-kube-cephfs-a-65f8485847-wrnsm ceph]# cat livenessprobe-test.sh
#!/usr/bin/env bash
# do not use 'set -e -u' etc. because it is important to only fail this probe when failure is certain
# spurious failures risk destabilizing ceph or the filesystem
MDS_ID="kube-cephfs-a"
FILESYSTEM_NAME="kube-cephfs"
KEYRING="/etc/ceph/keyring-store/keyring"
export CEPH_ARGS="--keyring $KEYRING"
outp="$(ceph fs dump -n "mds.$MDS_ID" --mon-host="$ROOK_CEPH_MON_HOST" --mon-initial-members="$ROOK_CEPH_MON_INITIAL_MEMBERS" --format json)"
rc=$?
if [ $rc -ne 0 ]; then
echo "ceph MDS dump check failed with the following output:"
echo "$outp"
echo "passing probe to avoid restarting MDS. cannot determine if MDS is unhealthy. restarting MDS risks destabilizing ceph/filesystem, which is likely unreachable or in error state"
exit 0
fi
# get the active and standby MDS in the fs map
standbyMds=$(echo "$outp" | jq ".standbys | map(.name) | any(.[]; . == \"$MDS_ID\")")
activeMds=$(echo "$outp" | jq ".filesystems[] | select(.mdsmap.fs_name == \"$FILESYSTEM_NAME\") | .mdsmap.info | map(.name) | any(.[]; . == \"$MDS_ID\")")
if [[ $standbyMds == true || $activeMds == true ]]; then
echo "MDS ID present in MDS map, no need to re-start the container"
exit 0
fi
echo "Error: MDS ID not present in MDS map"
exit 1
- Execute the script and verify that it works correctly now:
[root@rook-ceph-mds-kube-cephfs-a-65f8485847-wrnsm ceph]# ./livenessprobe-test.sh
dumped fsmap epoch 6
MDS ID present in MDS map, no need to re-start the container
Cluster Status to submit:
$ ceph status
cluster:
id: e410b695-e6b7-4f65-b162-a11e37bf1057
health: HEALTH_OK
services:
mon: 1 daemons, quorum a (age 68m)
mgr: a(active, since 66m)
mds: 1/1 daemons up, 1 hot standby
osd: 1 osds: 1 up (since 67m), 1 in (since 67m)
data:
volumes: 1/1 healthy
pools: 4 pools, 81 pgs
objects: 22 objects, 596 KiB
usage: 27 MiB used, 9.7 GiB / 9.8 GiB avail
pgs: 81 active+clean
io:
client: 1.2 KiB/s rd, 2 op/s rd, 0 op/s wr
Environment:
- OS (e.g. from /etc/os-release): Debian GNU/Linux 11 (bullseye)
- Kernel (e.g.
uname -a):Linux controller-0 6.6.0-1-amd64 #1 SMP PREEMPT_DYNAMIC - Rook version (use
rook versioninside of a Rook Pod): 1.13.7 - Storage backend version (e.g. for ceph do
ceph -v): ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) - Kubernetes version (use
kubectl version): v1.29.2 - Storage backend status (e.g. for Ceph use
ceph healthin the Rook Ceph toolbox): HEALTH_OK
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 30 (25 by maintainers)
Commits related to this issue
- mds: fix mds liveness probe mds liveness probe call ceph cmd with mons and keyring, but ceph actually looks for the ceph keyring at default folder first and then use the cmd line keyring path, So to ... — committed to parth-gr/rook by parth-gr 2 years ago
- mds: fix mds liveness probe mds liveness probe call ceph cmd with mons and keyring, but ceph actually looks for the ceph keyring at default folder first and then use the cmd line keyring path, So to ... — committed to parth-gr/rook by parth-gr 2 years ago
- mds: fix mds liveness probe mds liveness probe call ceph cmd with mons and keyring, but ceph actually looks for the ceph keyring at default folder first and then use the cmd line keyring path, So to ... — committed to parth-gr/rook by parth-gr 2 years ago
- mds: fix mds liveness probe mds liveness probe call ceph cmd with mons and keyring, but ceph actually looks for the ceph keyring at default folder first and then use the cmd line keyring path, So to ... — committed to parth-gr/rook by parth-gr 2 years ago
- mds: fix mds liveness probe mds liveness probe call ceph cmd with mons and keyring, but ceph actually looks for the ceph keyring at default folder first and then use the cmd line keyring path, So to ... — committed to parth-gr/rook by parth-gr 2 years ago
- mds: fix mds liveness probe mds liveness probe call ceph cmd with mons and keyring, but ceph actually looks for the ceph keyring at default folder first and then use the cmd line keyring path, So to ... — committed to parth-gr/rook by parth-gr 2 years ago
- mds: fix mds liveness probe mds liveness probe call ceph cmd with mons and keyring, but ceph actually looks for the ceph keyring at default folder first and then use the cmd line keyring path, So to ... — committed to rook/rook by parth-gr 2 years ago
- mds: fix mds liveness probe mds liveness probe call ceph cmd with mons and keyring, but ceph actually looks for the ceph keyring at default folder first and then use the cmd line keyring path, So to ... — committed to rook/rook by parth-gr 2 years ago
- mds: fix mds liveness probe mds liveness probe call ceph cmd with mons and keyring, but ceph actually looks for the ceph keyring at default folder first and then use the cmd line keyring path, So to ... — committed to patrostkowski/rook by parth-gr 2 years ago
I think it might be an alternative.
@parth-gr not sure if I’m the right person to mention in this issue … or it is just a mistake!
@parth-gr
Yes, you need to use both the
-n "mds.$MDS_ID"andexport CEPH_ARGS="--keyring $KEYRING"to get rid of all the errors, as you showed.Thank you for investigating it further @parth-gr ! I appreciate it
So the summary is inside the container,
This command is failing
with
But the below way it worked
I would like to get some initial suggestions from Ceph team if they have seen this earlier cc @batrick @rzarzynski
Hi @BlaineEXE !
I am actually running rook-ceph in StarlingX, so it is not easy to change versions for rook or ceph. StarlingX has vanilla kubernetes.
I found this issue reported in the past about the same problem:
As you can see in my demonstration running the script inside the mds pod, it fails because it cannot find a keyring referencing “client.admin.keyring” even though we are specifying a keyring as an argument:
--keyring "$KEYRING"Since it is a keyring problem when running the
ceph fs dumpcommand, all we need to do to fix it, is add the mds client name as an argument:-n "mds.$MDS_ID"This makes the keyring specified to be accepted for this specific mds client, as we can see below.
Script with the mds client name:
Running the script:
Notice that even though it works, first the
ceph fs dumpcommand looks for another keyring, fails, and than proceeds to use the keyring specified by us in--keyring "$KEYRING", that is why we see errors.To bypass these errors and force
ceph fs dumpto look for our specified keyring, we have to place the--keyringoption in theCEPH_ARGSvariable and export it, so thatceph fs dumpcan use it in this context. This makes the ceph command to use our argument as its first keyring option and not as a backup option.I hope this helps to clarify the issue and why I tried this type of fix.
Thank you for your quick response!