rook: rook v1.15.5 and v1.15.0 adds wrong keyring to osds repeatedly

  • Bug Report

Deviation from expected behavior:

rook adds a keyring into a newly created osd.1. osd.1 is a reused number - more on this soon.

Expected behavior:

rook to add correct keyring.

How to reproduce it (minimal and precise):

What I did:

  • find the osd / host
  • update the keyring on the host path to the key returned by ceph auth ls

What happened:

  • a few minutes later, the hostpath file was updated again with the incorrect key.

Additional information:

  • osd.1 is an old osd number, of an osd that was deleted
  • I did manually delete the keyring multiple times, but something (likely the rook operator) creates a new keyring
  • That keyring is always different when checking ceph auth ls - so the behaviour is correct
  • However the keyring that is stored inside osd.1 is always the SAME, incorrect one

Further Additional information:

  • Many osds where newly added at the same time on different hosts
  • There used to be an osd.1 deployment on server118, while the current deployment targets server116
  • This might be a race condition when multiple prepare jobs create osds at the same time

Cluster Status to submit:

Environment:

  • OS (e.g. from /etc/os-release): alpine-linux
  • Kernel (e.g. uname -a):6.6.61-0-lts
  • Cloud provider or hardware configuration: hardware
  • Rook version (use rook version inside of a Rook Pod):v1.15.0 and v1.15.5 both confirmed
  • Storage backend version (e.g. for ceph do ceph -v): ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)
  • Kubernetes version (use kubectl version): 1.31.1
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): kubeadm

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 34 (9 by maintainers)

Most upvoted comments

Is there any hotfix for this issue?

Not yet, it is still under investigation. To help the ceph team understand the priority/severity, please do comment on the ceph tracker: https://tracker.ceph.com/issues/69173 (looks like https://tracker.ceph.com/issues/69697 is a dup of that one)

@sp98 The OSD is being created, it’ a new one:

% kubectl -n rook-ceph get pods | grep rook-ceph-osd    
rook-ceph-osd-0-6f9b4b968c-plcjk                      1/1     Running            0               22h
rook-ceph-osd-1-5684ddd5c5-xzxvq                      0/1     CrashLoopBackOff   6 (3m47s ago)   9m40s
rook-ceph-osd-10-d8755f996-2glmd                      1/1     Running            0               22h
rook-ceph-osd-11-6b7c5fd4cc-zrjnh                     1/1     Running            0               22h
rook-ceph-osd-12-8677c78db4-s9ttc                     1/1     Running            0               22h
rook-ceph-osd-13-688495b96f-mjwlg                     1/1     Running            0               22h
rook-ceph-osd-14-6b94b7767f-45tvb                     1/1     Running            0               22h
rook-ceph-osd-15-74659bdb6d-cdxbr                     1/1     Running            0               22h
rook-ceph-osd-16-549c5d45f8-ncshf                     1/1     Running            0               22h
rook-ceph-osd-17-5d96d99b5f-6rkpv                     1/1     Running            0               22h
rook-ceph-osd-18-5dbf4948d9-k7p5n                     1/1     Running            0               22h
rook-ceph-osd-19-85fb6f7d95-t7prj                     1/1     Running            0               22h
rook-ceph-osd-2-6f9c9d55c7-qhvs6                      1/1     Running            0               22h
rook-ceph-osd-20-85f956948b-nm8d2                     1/1     Running            0               22h
rook-ceph-osd-21-7f964846cb-wtl2n                     1/1     Running            0               22h
rook-ceph-osd-22-6948c84-ksf29                        1/1     Running            0               22h
rook-ceph-osd-23-747b449644-4fw2v                     1/1     Running            0               22h
rook-ceph-osd-24-74957d7489-57b6m                     1/1     Running            0               22h
rook-ceph-osd-25-74cf95b745-nmwfb                     1/1     Running            0               22h
rook-ceph-osd-26-f57777946-r9s8h                      1/1     Running            0               22h
rook-ceph-osd-27-7c88cd8fcc-5cwlb                     1/1     Running            0               22h
rook-ceph-osd-28-7b4488d6b-m2gwg                      1/1     Running            0               22h
rook-ceph-osd-29-84b8bf99f4-dlwj7                     1/1     Running            0               22h
rook-ceph-osd-31-7bcc5c5977-c5sph                     1/1     Running            0               22h
rook-ceph-osd-32-5f65c5667f-v99ww                     1/1     Running            0               22h
rook-ceph-osd-33-5f6d7d5f88-vdx98                     1/1     Running            0               22h
rook-ceph-osd-34-576b77b88b-l4tpl                     1/1     Running            0               22h
rook-ceph-osd-35-7fc6d9474-6mxpm                      1/1     Running            0               22h
rook-ceph-osd-36-66fb64d58b-vdlfl                     1/1     Running            0               22h
rook-ceph-osd-4-58bdf5b8ff-ndcw8                      1/1     Running            0               22h
rook-ceph-osd-5-5479779f57-vfdzb                      1/1     Running            0               22h
rook-ceph-osd-6-849c5885f5-2v9t7                      1/1     Running            0               22h
rook-ceph-osd-7-76cb6bd7f5-xpnwd                      1/1     Running            0               22h
rook-ceph-osd-8-b45f66f6c-cthxd                       1/1     Running            0               22h
rook-ceph-osd-9-57f59f5bcc-dtkzr                      1/1     Running            0               22h
rook-ceph-osd-prepare-apu3-xzmn2                      0/1     Completed          0               21h
rook-ceph-osd-prepare-server101-t89h7                 0/1     Completed          0               21h
rook-ceph-osd-prepare-server116-k8q7n                 0/1     Completed          0               21h
rook-ceph-osd-prepare-server96-mghzk                  0/1     Completed          0               21h
rook-ceph-osd-prepare-server97-nfkdx                  0/1     Completed          0               21h
rook-ceph-osd-prepare-server98-nd2tb                  0/1     Completed          0               21h
rook-ceph-osd-prepare-server99-95trl                  0/1     Completed          0               21h

osd output

% kubectl -n rook-ceph logs -c osd  rook-ceph-osd-1-5684ddd5c5-xzxvq        
debug 2024-11-19T12:07:57.111+0000 7f7e5892a640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2024-11-19T12:07:57.111+0000 7f7e58129640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2024-11-19T12:07:57.111+0000 7f7e57928640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
failed to fetch mon config (--no-mon-config to skip)

activate output (showing the correct key, redacted):

kubectl -n rook-ceph logs -c activate rook-ceph-osd-1-5684ddd5c5-xzxvq
+ OSD_ID=1
+ CEPH_FSID=bd3061a0-ecf3-4af6-9017-51b63c90b526
+ OSD_UUID=2cad7f5b-6d78-430b-a989-232319118bf3
+ OSD_STORE_FLAG=--bluestore
+ OSD_DATA_DIR=/var/lib/ceph/osd/ceph-1
+ CV_MODE=raw
+ DEVICE=/dev/sdg
+ cp --no-preserve=mode /etc/temp-ceph/ceph.conf /etc/ceph/ceph.conf
+ python3 -c '
import configparser

config = configparser.ConfigParser()
config.read('\''/etc/ceph/ceph.conf'\'')

if not config.has_section('\''global'\''):
    config['\''global'\''] = {}

if not config.has_option('\''global'\'','\''fsid'\''):
    config['\''global'\'']['\''fsid'\''] = '\''bd3061a0-ecf3-4af6-9017-51b63c90b526'\''

with open('\''/etc/ceph/ceph.conf'\'', '\''w'\'') as configfile:
    config.write(configfile)
'
+ ceph -n client.admin auth get-or-create osd.1 mon 'allow profile osd' mgr 'allow profile osd' osd 'allow *' -k /etc/ceph/admin-keyring-store/keyring
[osd.1]
	key = AQCE...
+ [[ raw == \l\v\m ]]
++ mktemp
+ OSD_LIST=/tmp/tmp.RnI5UFXLr2
+ ceph-volume raw list /dev/sdg
+ cat /tmp/tmp.RnI5UFXLr2
{
    "2cad7f5b-6d78-430b-a989-232319118bf3": {
        "ceph_fsid": "bd3061a0-ecf3-4af6-9017-51b63c90b526",
        "device": "/dev/sdg",
        "osd_id": 1,
        "osd_uuid": "2cad7f5b-6d78-430b-a989-232319118bf3",
        "type": "bluestore"
    }
}
+ find_device
+ python3 -c '
import sys, json
for _, info in json.load(sys.stdin).items():
	if info['\''osd_id'\''] == 1:
		print(info['\''device'\''], end='\'''\'')
		print('\''found device: '\'' + info['\''device'\''], file=sys.stderr) # log the disk we found to stderr
		sys.exit(0)  # don'\''t keep processing once the disk is found
sys.exit('\''no disk found with OSD ID 1'\'')
'
/dev/sdgfound device: /dev/sdg
+ [[ -z /dev/sdg ]]
+ OSD_BLOCK_PATH=/var/lib/ceph/osd/ceph-1/block
++ readlink /var/lib/ceph/osd/ceph-1/block
+ '[' -L /var/lib/ceph/osd/ceph-1/block -a /dev/sdg '!=' /dev/sdg ']'
+ ceph-volume raw activate --device /dev/sdg --no-systemd --no-tmpfs
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1
Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/sdg
Running command: /usr/bin/chown -R ceph:ceph /dev/sdg
Running command: /usr/bin/ln -s /dev/sdg /var/lib/ceph/osd/ceph-1/block
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1
--> ceph-volume raw activate successful for osd ID: 1