rook: ceph_assert error on rgw start in rook-ceph-rgw-ceph-objectstore pod

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior:

After cluster restart, the rook-ceph-rgw-ceph-objectstore-xxx pod was noticed to run into CrashLoopBackOff. All crash logs since then look the same: an assertion error is reported for /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.1/rpm/el8/BUILD/ceph-18.2.1/src/rgw/rgw_amqp.cc: 840: FAILED ceph_assert(rc==0)

I first assumed some corruption and tried to fix with pod re-creations without and with having removed all(?) rgw/objectstore resources (pool, zones, zonegroups, etc, at least one time incl the .rgw.root pool). Symptoms never changed.

(The objectstore was un-used)

Expected behavior:

rook-ceph-rgw-ceph-objectstore-xxx starts nominally.

How to reproduce it (minimal and precise):

The cluster runs in microk8s (v1.25) cluster of 4 bare metal nodes (48 cores, 384GB ram each) with 4x 4TB ssds each inside, 2 of them on each node contain one partition configured for osds for ceph.

Ceph was installed using helm:

helm repo add rook-release https://charts.rook.io/release
helm repo update

helm upgrade --install rook-ceph \
    --namespace rook-ceph \
    --create-namespace \
    --values rook-ceph-operator-values.yaml \
    rook-release/rook-ceph

helm upgrade --install rook-ceph-cluster \
    --namespace rook-ceph \
    --create-namespace \
    --values rook-ceph-cluster-values.yaml \
    rook-release/rook-ceph-cluster

kubectl -n rook-ceph exec -it $(kubectl get pod -n rook-ceph -o name | grep ceph-tools) -- /bin/bash
ceph mgr module enable rook
ceph orch set backend rook
exit

(no idea if this triggers the problem again)

File(s) to submit:

  • Cluster CR

rook-ceph-operator-values.yaml:

csi:
  kubeletDirPath: "/var/snap/microk8s/common/var/lib/kubelet"

rook-ceph-cluster-values.yaml:

storage:
  useAllNodes: false
  useAllDevices: false
  config:
    osdsPerDevice: 4
    metadataDevice: "/dev/sdc1"
  nodes:
    - name: "hpc-srv1.xxx.yyy"
      devices:
        - name: "/dev/sdb1"
    - name: "hpc-srv2.xxx.yyy"
      devices:
        - name: "/dev/sdb1"
    - name: "hpc-srv3.xxx.yyy"
      devices:
        - name: "/dev/sdb1"
    - name: "hpc-srv4.xxx.yyy"
      devices:
        - name: "/dev/sdb1"

toolbox:
  enabled: true

Logs to submit:

Ceph crash info:

bash-4.4$ ceph crash info 2024-01-23T13:09:25.023127Z_f9c5ce06-f7c6-4a2c-9504-a3f30fa27794
{
    "assert_condition": "rc==0",
    "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.1/rpm/el8/BUILD/ceph-18.2.1/src/rgw/rgw_amqp.cc",
    "assert_func": "rgw::amqp::Manager::Manager(size_t, size_t, size_t, long int, unsigned int, unsigned int, ceph::common::CephContext*)",
    "assert_line": 840,
    "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.1/rpm/el8/BUILD/ceph-18.2.1/src/rgw/rgw_amqp.cc: In function 'rgw::amqp::Manager::Manager(size_t, size_t, size_t, long int, unsigned int, unsigned int, ceph::common::CephContext*)' thread 7f81c5804a80 time 2024-01-23T13:09:25.016871+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.1/rpm/el8/BUILD/ceph-18.2.1/src/rgw/rgw_amqp.cc: 840: FAILED ceph_assert(rc==0)\n",
    "assert_thread_name": "radosgw",
    "backtrace": [
        "/lib64/libpthread.so.0(+0x12d20) [0x7f81cacc1d20]",
        "gsignal()",
        "abort()",
        "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f81cdc09e6f]",
        "/usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f81cdc09fdb]",
        "(rgw::amqp::init(ceph::common::CephContext*)+0x261) [0x55e4aa4a3961]",
        "(rgw::AppMain::init_notification_endpoints()+0x38) [0x55e4a9e00d98]",
        "main()",
        "__libc_start_main()",
        "_start()"
    ],
    "ceph_version": "18.2.1",
    "crash_id": "2024-01-23T13:09:25.023127Z_f9c5ce06-f7c6-4a2c-9504-a3f30fa27794",
    "entity_name": "client.rgw.ceph.objectstore.a",
    "os_id": "centos",
    "os_name": "CentOS Stream",
    "os_version": "8",
    "os_version_id": "8",
    "process_name": "radosgw",
    "stack_sig": "e082d3eaabfd1f88602d9630b283a2e38f24470b2a2ab68506ca57702cc1bcc2",
    "timestamp": "2024-01-23T13:09:25.023127Z",
    "utsname_hostname": "rook-ceph-rgw-ceph-objectstore-a-6c95458fb-gcb4d",
    "utsname_machine": "x86_64",
    "utsname_release": "3.10.0-1160.105.1.el7.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP Thu Dec 7 15:39:45 UTC 2023"
}
  • Operator’s logs

The operator show repeatedly blocks like this. With ending in failing to contact the rgw service in the end - which is expected as this did not come up:

2024-01-23 12:55:14.652586 I | ceph-spec: parsing mon endpoints: b=10.152.183.153:6789,a=10.152.183.228:6789,d=10.152.183.75:6789
2024-01-23 12:55:14.652655 I | ceph-spec: detecting the ceph image version for image quay.io/ceph/ceph:v18.2.1...
2024-01-23 12:55:18.411951 I | ceph-spec: detected ceph image version: "18.2.1-0 reef"
2024-01-23 12:55:29.360660 I | ceph-object-controller: reconciling object store deployments
2024-01-23 12:55:29.407902 I | ceph-object-controller: ceph object store gateway service running at 10.152.183.71
2024-01-23 12:55:29.407931 I | ceph-object-controller: reconciling object store pools
2024-01-23 12:55:34.581398 I | cephclient: application "rook-ceph-rgw" is already set on pool "ceph-objectstore.rgw.control"
2024-01-23 12:55:34.581429 I | cephclient: reconciling replicated pool ceph-objectstore.rgw.control succeeded
2024-01-23 12:55:38.251090 I | cephclient: creating a new crush rule for changed deviceClass on crush rule "ceph-objectstore.rgw.control_host"
2024-01-23 12:55:38.251129 I | cephclient: updating pool "ceph-objectstore.rgw.control" failure domain from "host" to "host" with new crush rule "ceph-objectsto
re.rgw.control_host"
2024-01-23 12:55:38.251143 I | cephclient: crush rule "ceph-objectstore.rgw.control_host" will no longer be used by pool "ceph-objectstore.rgw.control"
2024-01-23 12:55:46.835021 I | cephclient: Successfully updated pool "ceph-objectstore.rgw.control" failure domain to "host"
2024-01-23 12:55:46.835058 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph-objectstore.rgw.control"
2024-01-23 12:55:54.964801 I | cephclient: application "rook-ceph-rgw" is already set on pool "ceph-objectstore.rgw.meta"
2024-01-23 12:55:54.964830 I | cephclient: reconciling replicated pool ceph-objectstore.rgw.meta succeeded
2024-01-23 12:55:58.541081 I | cephclient: creating a new crush rule for changed deviceClass on crush rule "ceph-objectstore.rgw.meta_host"
2024-01-23 12:55:58.541117 I | cephclient: updating pool "ceph-objectstore.rgw.meta" failure domain from "host" to "host" with new crush rule "ceph-objectstore.
rgw.meta_host"
2024-01-23 12:55:58.541131 I | cephclient: crush rule "ceph-objectstore.rgw.meta_host" will no longer be used by pool "ceph-objectstore.rgw.meta"
2024-01-23 12:56:02.665487 I | cephclient: Successfully updated pool "ceph-objectstore.rgw.meta" failure domain to "host"
2024-01-23 12:56:02.665524 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph-objectstore.rgw.meta"
2024-01-23 12:56:09.834902 I | cephclient: application "rook-ceph-rgw" is already set on pool "ceph-objectstore.rgw.log"
2024-01-23 12:56:09.834935 I | cephclient: reconciling replicated pool ceph-objectstore.rgw.log succeeded
2024-01-23 12:56:16.150305 I | cephclient: creating a new crush rule for changed deviceClass on crush rule "ceph-objectstore.rgw.log_host"
2024-01-23 12:56:16.150339 I | cephclient: updating pool "ceph-objectstore.rgw.log" failure domain from "host" to "host" with new crush rule "ceph-objectstore.r
gw.log_host"
2024-01-23 12:56:16.150381 I | cephclient: crush rule "ceph-objectstore.rgw.log_host" will no longer be used by pool "ceph-objectstore.rgw.log"
2024-01-23 12:56:20.086484 I | cephclient: Successfully updated pool "ceph-objectstore.rgw.log" failure domain to "host"
2024-01-23 12:56:20.086517 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph-objectstore.rgw.log"
2024-01-23 12:56:27.485478 I | cephclient: application "rook-ceph-rgw" is already set on pool "ceph-objectstore.rgw.buckets.index"
2024-01-23 12:56:27.485508 I | cephclient: reconciling replicated pool ceph-objectstore.rgw.buckets.index succeeded
2024-01-23 12:56:33.461033 I | cephclient: creating a new crush rule for changed deviceClass on crush rule "ceph-objectstore.rgw.buckets.index_host"
2024-01-23 12:56:33.461070 I | cephclient: updating pool "ceph-objectstore.rgw.buckets.index" failure domain from "host" to "host" with new crush rule "ceph-obj
ectstore.rgw.buckets.index_host"
2024-01-23 12:56:33.461086 I | cephclient: crush rule "ceph-objectstore.rgw.buckets.index_host" will no longer be used by pool "ceph-objectstore.rgw.buckets.ind
ex"
2024-01-23 12:56:37.714552 I | cephclient: Successfully updated pool "ceph-objectstore.rgw.buckets.index" failure domain to "host"
2024-01-23 12:56:37.714587 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph-objectstore.rgw.buckets.index"
2024-01-23 12:56:45.148772 I | cephclient: application "rook-ceph-rgw" is already set on pool "ceph-objectstore.rgw.buckets.non-ec"
2024-01-23 12:56:45.148800 I | cephclient: reconciling replicated pool ceph-objectstore.rgw.buckets.non-ec succeeded
2024-01-23 12:56:49.453871 I | cephclient: creating a new crush rule for changed deviceClass on crush rule "ceph-objectstore.rgw.buckets.non-ec_host"
2024-01-23 12:56:49.453904 I | cephclient: updating pool "ceph-objectstore.rgw.buckets.non-ec" failure domain from "host" to "host" with new crush rule "ceph-ob
jectstore.rgw.buckets.non-ec_host"
2024-01-23 12:56:49.453919 I | cephclient: crush rule "ceph-objectstore.rgw.buckets.non-ec_host" will no longer be used by pool "ceph-objectstore.rgw.buckets.no
n-ec"
2024-01-23 12:56:57.847024 I | cephclient: Successfully updated pool "ceph-objectstore.rgw.buckets.non-ec" failure domain to "host"
2024-01-23 12:56:57.847063 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph-objectstore.rgw.buckets.non-ec"
2024-01-23 12:57:08.037921 I | cephclient: application "rook-ceph-rgw" is already set on pool "ceph-objectstore.rgw.otp"
2024-01-23 12:57:08.037952 I | cephclient: reconciling replicated pool ceph-objectstore.rgw.otp succeeded
2024-01-23 12:57:11.664909 I | cephclient: creating a new crush rule for changed deviceClass on crush rule "ceph-objectstore.rgw.otp_host"
2024-01-23 12:57:11.664942 I | cephclient: updating pool "ceph-objectstore.rgw.otp" failure domain from "host" to "host" with new crush rule "ceph-objectstore.r
gw.otp_host"
2024-01-23 12:57:11.664956 I | cephclient: crush rule "ceph-objectstore.rgw.otp_host" will no longer be used by pool "ceph-objectstore.rgw.otp"
2024-01-23 12:57:15.710670 I | cephclient: Successfully updated pool "ceph-objectstore.rgw.otp" failure domain to "host"
2024-01-23 12:57:15.710710 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph-objectstore.rgw.otp"
2024-01-23 12:57:24.811457 I | cephclient: reconciling replicated pool .rgw.root succeeded
2024-01-23 12:57:28.038639 I | cephclient: creating a new crush rule for changed deviceClass on crush rule ".rgw.root_host"
2024-01-23 12:57:28.038673 I | cephclient: updating pool ".rgw.root" failure domain from "host" to "host" with new crush rule ".rgw.root_host"
2024-01-23 12:57:28.038685 I | cephclient: crush rule ".rgw.root_host" will no longer be used by pool ".rgw.root"
2024-01-23 12:57:31.694532 I | cephclient: Successfully updated pool ".rgw.root" failure domain to "host"
2024-01-23 12:57:31.694568 I | cephclient: setting pool property "pg_num_min" to "8" on pool ".rgw.root"
2024-01-23 12:57:41.464564 I | cephclient: setting pool property "allow_ec_overwrites" to "true" on pool "ceph-objectstore.rgw.buckets.data"
2024-01-23 12:57:45.445573 I | cephclient: application "rook-ceph-rgw" is already set on pool "ceph-objectstore.rgw.buckets.data"
2024-01-23 12:57:45.445608 I | cephclient: creating EC pool ceph-objectstore.rgw.buckets.data succeeded
2024-01-23 12:57:45.445623 I | ceph-object-controller: setting multisite settings for object store "ceph-objectstore"
2024-01-23 12:57:47.046709 I | ceph-object-controller: there are no changes to commit for RGW configuration period for CephObjectStore "rook-ceph/ceph-objectsto
re"
2024-01-23 12:57:47.046742 I | ceph-object-controller: Multisite for object-store: realm=ceph-objectstore, zonegroup=ceph-objectstore, zone=ceph-objectstore
2024-01-23 12:57:47.046754 I | ceph-object-controller: multisite configuration for object-store ceph-objectstore is complete
2024-01-23 12:57:47.046767 I | ceph-object-controller: creating object store "ceph-objectstore" in namespace "rook-ceph"
2024-01-23 12:57:47.085345 I | cephclient: getting or creating ceph auth key "client.rgw.ceph.objectstore.a"
2024-01-23 12:57:49.288104 I | ceph-object-controller: setting rgw config flags
2024-01-23 12:57:49.288142 I | op-config: setting "client.rgw.ceph.objectstore.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2024-01-23 12:57:51.155725 I | op-config: successfully set "client.rgw.ceph.objectstore.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration d
atabase
2024-01-23 12:57:51.155763 I | op-config: setting "client.rgw.ceph.objectstore.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2024-01-23 12:57:54.747863 I | op-config: successfully set "client.rgw.ceph.objectstore.a"="rgw_log_object_name_utc"="true" option to the mon configuration data
base
2024-01-23 12:57:54.747889 I | op-config: setting "client.rgw.ceph.objectstore.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2024-01-23 12:57:57.366744 I | op-config: successfully set "client.rgw.ceph.objectstore.a"="rgw_enable_usage_log"="true" option to the mon configuration databas
e
2024-01-23 12:57:57.366811 I | op-config: setting "client.rgw.ceph.objectstore.a"="rgw_zone"="ceph-objectstore" option to the mon configuration database
2024-01-23 12:58:00.746514 I | op-config: successfully set "client.rgw.ceph.objectstore.a"="rgw_zone"="ceph-objectstore" option to the mon configuration databas
e
2024-01-23 12:58:00.746557 I | op-config: setting "client.rgw.ceph.objectstore.a"="rgw_zonegroup"="ceph-objectstore" option to the mon configuration database
2024-01-23 12:58:04.751537 I | op-config: successfully set "client.rgw.ceph.objectstore.a"="rgw_zonegroup"="ceph-objectstore" option to the mon configuration da
tabase
2024-01-23 12:58:04.751568 I | op-config: setting "client.rgw.ceph.objectstore.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2024-01-23 12:58:08.257558 I | op-config: successfully set "client.rgw.ceph.objectstore.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2024-01-23 12:58:08.257892 I | ceph-object-controller: object store "ceph-objectstore" deployment "rook-ceph-rgw-ceph-objectstore-a" created
2024-01-23 12:58:08.288756 I | ceph-object-controller: object store "ceph-objectstore" deployment "rook-ceph-rgw-ceph-objectstore-a" already exists. updating if
 needed
2024-01-23 12:58:08.305345 I | op-k8sutil: deployment "rook-ceph-rgw-ceph-objectstore-a" did not change, nothing to update
2024-01-23 12:58:08.311499 I | ceph-object-controller: config map "rook-ceph-rgw-ceph-objectstore-mime-types" for object store "ceph-objectstore" already exists
, not overwriting
2024-01-23 12:58:08.346702 I | ceph-object-controller: enabling rgw dashboard
2024-01-23 12:58:12.495601 I | ceph-object-controller: created object store "ceph-objectstore" in namespace "rook-ceph"
2024-01-23 12:58:14.287460 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/ceph-objectstore". failed to create object store deploymen
ts: failed to get COSI user "cosi": Get "http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:80/admin/user?format=json&uid=cosi": dial tcp 10.152.183.71:80: con
nect: connection refused
  • Crashing pod(s) logs

rook-ceph-rgw-ceph-objectstore-a-xxx (complete in attached log file):

debug   -767> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command assert hook 0x55e4abcf6c50
debug   -766> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command abort hook 0x55e4abcf6c50
debug   -765> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command leak_some_memory hook 0x55e4abcf6c50
debug   -764> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command perfcounters_dump hook 0x55e4abcf6c50
debug   -763> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command 1 hook 0x55e4abcf6c50
debug   -762> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command perf dump hook 0x55e4abcf6c50
debug   -761> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command perfcounters_schema hook 0x55e4abcf6c50
debug   -760> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command perf histogram dump hook 0x55e4abcf6c50
debug   -759> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command 2 hook 0x55e4abcf6c50
debug   -758> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command perf schema hook 0x55e4abcf6c50
debug   -757> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command counter dump hook 0x55e4abcf6c50
debug   -756> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command counter schema hook 0x55e4abcf6c50
debug   -755> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command perf histogram schema hook 0x55e4abcf6c50
debug   -754> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command perf reset hook 0x55e4abcf6c50
debug   -753> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command config show hook 0x55e4abcf6c50
debug   -752> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command config help hook 0x55e4abcf6c50
debug   -751> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command config set hook 0x55e4abcf6c50
debug   -750> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command config unset hook 0x55e4abcf6c50
debug   -749> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command config get hook 0x55e4abcf6c50
debug   -748> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command config diff hook 0x55e4abcf6c50
debug   -747> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command config diff get hook 0x55e4abcf6c50
debug   -746> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command injectargs hook 0x55e4abcf6c50
debug   -745> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command log flush hook 0x55e4abcf6c50
debug   -744> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command log dump hook 0x55e4abcf6c50
debug   -743> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command log reopen hook 0x55e4abcf6c50
debug   -742> 2024-01-23T13:08:55.645+0000 7f81c5804a80  5 asok(0x55e4abfda000) register_command dump_mempools hook 0x55e4aca9e068
debug   -741> 2024-01-23T13:08:55.659+0000 7f81c5804a80 10 monclient: get_monmap_and_config
debug   -740> 2024-01-23T13:08:55.659+0000 7f81c5804a80 10 monclient: build_initial_monmap
debug   -739> 2024-01-23T13:08:55.659+0000 7f81c5804a80  1 build_initial for_mkfs: 0
debug   -738> 2024-01-23T13:08:55.659+0000 7f81c5804a80 10 monclient: monmap:
epoch 0
fsid f31bf636-769f-423b-bc43-5ccf3d1197b1
last_changed 2024-01-23T13:08:55.660821+0000
created 2024-01-23T13:08:55.660821+0000
min_mon_release 0 (unknown)
election_strategy: 1
0: [v2:10.152.183.75:3300/0,v1:10.152.183.75:6789/0] mon.noname-b
1: [v2:10.152.183.153:3300/0,v1:10.152.183.153:6789/0] mon.noname-c
2: [v2:10.152.183.228:3300/0,v1:10.152.183.228:6789/0] mon.noname-a

debug   -737> 2024-01-23T13:08:55.660+0000 7f81c5804a80  5 AuthRegistry(0x55e4aca9a140) adding auth protocol: cephx
debug   -736> 2024-01-23T13:08:55.660+0000 7f81c5804a80  5 AuthRegistry(0x55e4aca9a140) adding auth protocol: cephx
debug   -735> 2024-01-23T13:08:55.660+0000 7f81c5804a80  5 AuthRegistry(0x55e4aca9a140) adding auth protocol: cephx
debug   -734> 2024-01-23T13:08:55.660+0000 7f81c5804a80  5 AuthRegistry(0x55e4aca9a140) adding con mode: secure
debug   -733> 2024-01-23T13:08:55.660+0000 7f81c5804a80  5 AuthRegistry(0x55e4aca9a140) adding con mode: crc
debug   -732> 2024-01-23T13:08:55.660+0000 7f81c5804a80  5 AuthRegistry(0x55e4aca9a140) adding con mode: secure
debug   -731> 2024-01-23T13:08:55.660+0000 7f81c5804a80  5 AuthRegistry(0x55e4aca9a140) adding con mode: crc
debug   -730> 2024-01-23T13:08:55.660+0000 7f81c5804a80  5 AuthRegistry(0x55e4aca9a140) adding con mode: secure
debug   -729> 2024-01-23T13:08:55.660+0000 7f81c5804a80  5 AuthRegistry(0x55e4aca9a140) adding con mode: crc
debug   -728> 2024-01-23T13:08:55.660+0000 7f81c5804a80  5 AuthRegistry(0x55e4aca9a140) adding con mode: secure
debug   -727> 2024-01-23T13:08:55.660+0000 7f81c5804a80  5 AuthRegistry(0x55e4aca9a140) adding con mode: crc
debug   -726> 2024-01-23T13:08:55.660+0000 7f81c5804a80  5 AuthRegistry(0x55e4aca9a140) adding con mode: secure
debug   -725> 2024-01-23T13:08:55.660+0000 7f81c5804a80  5 AuthRegistry(0x55e4aca9a140) adding con mode: crc
debug   -724> 2024-01-23T13:08:55.660+0000 7f81c5804a80  5 AuthRegistry(0x55e4aca9a140) adding con mode: secure
debug   -723> 2024-01-23T13:08:55.660+0000 7f81c5804a80  2 auth: KeyRing::load: loaded key file /etc/ceph/keyring-store/keyring
debug   -722> 2024-01-23T13:08:55.661+0000 7f81c5804a80 10 monclient: init
debug   -721> 2024-01-23T13:08:55.661+0000 7f81c5804a80  5 AuthRegistry(0x7ffcbf5e0400) adding auth protocol: cephx

...

debug    -19> 2024-01-23T13:09:24.987+0000 7f8093c91700 10 monclient: _finish_auth 0
debug    -18> 2024-01-23T13:09:24.988+0000 7f8093c91700 10 monclient: _check_auth_tickets
debug    -17> 2024-01-23T13:09:24.988+0000 7f8093c91700 10 monclient: handle_config config(11 keys) v1
debug    -16> 2024-01-23T13:09:24.988+0000 7f8093c91700 10 monclient: handle_monmap mon_map magic: 0 v1
debug    -15> 2024-01-23T13:09:24.988+0000 7f8093c91700 10 monclient:  got monmap 6 from mon.b (according to old e6)
debug    -14> 2024-01-23T13:09:24.988+0000 7f8095494700  4 set_mon_vals no callback set
debug    -13> 2024-01-23T13:09:24.988+0000 7f8093c91700 10 monclient: dump:
epoch 6
fsid f31bf636-769f-423b-bc43-5ccf3d1197b1
last_changed 2024-01-21T22:56:22.019951+0000
created 2024-01-11T20:17:19.172397+0000
min_mon_release 18 (reef)
election_strategy: 1
0: [v2:10.152.183.228:3300/0,v1:10.152.183.228:6789/0] mon.a
1: [v2:10.152.183.153:3300/0,v1:10.152.183.153:6789/0] mon.b
2: [v2:10.152.183.75:3300/0,v1:10.152.183.75:6789/0] mon.d

debug    -12> 2024-01-23T13:09:24.988+0000 7f81c5804a80  5 monclient: authenticate success, global_id 1778332
debug    -11> 2024-01-23T13:09:24.988+0000 7f81c5804a80 10 monclient: _renew_subs
debug    -10> 2024-01-23T13:09:24.988+0000 7f81c5804a80 10 monclient: _send_mon_message to mon.b at v2:10.152.183.153:3300/0
debug     -9> 2024-01-23T13:09:24.988+0000 7f81c5804a80 10 monclient: _renew_subs
debug     -8> 2024-01-23T13:09:24.988+0000 7f81c5804a80 10 monclient: _send_mon_message to mon.b at v2:10.152.183.153:3300/0
debug     -7> 2024-01-23T13:09:24.988+0000 7f81c5804a80  1 librados: init done
debug     -6> 2024-01-23T13:09:24.992+0000 7f8093c91700  4 mgrc handle_mgr_map Got map version 842
debug     -5> 2024-01-23T13:09:24.992+0000 7f8093c91700  4 mgrc handle_mgr_map Active mgr is now [v2:10.1.23.96:6800/3090971858,v1:10.1.23.96:6801/3090971858]
debug     -4> 2024-01-23T13:09:24.992+0000 7f8093c91700  4 mgrc reconnect Starting new session with [v2:10.1.23.96:6800/3090971858,v1:10.1.23.96:6801/3090971858
]
debug     -3> 2024-01-23T13:09:24.992+0000 7f81c2b91700 10 monclient: get_auth_request con 0x55e4af1ca000 auth_method 0
debug     -2> 2024-01-23T13:09:24.993+0000 7f81c3b93700 10 monclient: get_auth_request con 0x55e4ace2a000 auth_method 0
debug     -1> 2024-01-23T13:09:25.019+0000 7f81c5804a80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/cento
s8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.1/rpm/el8/BUILD/ceph-18.2.1/src/rgw/rgw_amqp.cc: In function 'rgw::amqp::Manager::Manager(size_t, size_t, siz
e_t, long int, unsigned int, unsigned int, ceph::common::CephContext*)' thread 7f81c5804a80 time 2024-01-23T13:09:25.016871+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.1/rpm/el
8/BUILD/ceph-18.2.1/src/rgw/rgw_amqp.cc: 840: FAILED ceph_assert(rc==0)

 ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f81cdc09e15]
 2: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f81cdc09fdb]
 3: (rgw::amqp::init(ceph::common::CephContext*)+0x261) [0x55e4aa4a3961]
 4: (rgw::AppMain::init_notification_endpoints()+0x38) [0x55e4a9e00d98]
 5: main()
 6: __libc_start_main()
 7: _start()

debug      0> 2024-01-23T13:09:25.022+0000 7f81c5804a80 -1 *** Caught signal (Aborted) **
 in thread 7f81c5804a80 thread_name:radosgw

 ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
 1: /lib64/libpthread.so.0(+0x12d20) [0x7f81cacc1d20]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f81cdc09e6f]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f81cdc09fdb]
 6: (rgw::amqp::init(ceph::common::CephContext*)+0x261) [0x55e4aa4a3961]
 7: (rgw::AppMain::init_notification_endpoints()+0x38) [0x55e4a9e00d98]
 8: main()
 9: __libc_start_main()
 10: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 rbd_pwl
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 immutable_obj_cache
   0/ 5 client
   1/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 0 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 1 reserver
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/ 5 rgw_sync
   1/ 5 rgw_datacache
   1/ 5 rgw_access
   1/ 5 rgw_dbstore
   1/ 5 rgw_flight
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 compressor
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   1/ 5 fuse
   2/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
   1/ 5 prioritycache
   0/ 5 test
   0/ 5 cephfs_mirror
   0/ 5 cephsqlite
   0/ 5 seastore
   0/ 5 seastore_onode
   0/ 5 seastore_odata
   0/ 5 seastore_omap
   0/ 5 seastore_tm
   0/ 5 seastore_t
   0/ 5 seastore_cleaner
   0/ 5 seastore_epm
   0/ 5 seastore_lba
   0/ 5 seastore_fixedkv_tree
   0/ 5 seastore_cache
   0/ 5 seastore_journal
   0/ 5 seastore_device
   0/ 5 seastore_backref
   0/ 5 alienstore
   1/ 5 mclock
   0/ 5 cyanstore
   1/ 5 ceph_exporter
   1/ 5 memstore
  -2/-2 (syslog threshold)
  99/99 (stderr threshold)
--- pthread ID / name mapping for recent threads ---
  7f8093c91700 / ms_dispatch
  7f8095494700 / io_context_pool
  7f819bb43700 / lifecycle_thr_2
  7f819db47700 / lifecycle_thr_1
  7f819fb4b700 / lifecycle_thr_0
  7f81a6358700 / rgw_obj_expirer
  7f81a6b59700 / rgw_gc
  7f81a8b5d700 / ms_dispatch
  7f81aa360700 / io_context_pool
  7f81aab61700 / rgw_dt_lg_renew
  7f81bbb83700 / safe_timer
  7f81bcb85700 / ms_dispatch
  7f81bd386700 / ceph_timer
  7f81be388700 / io_context_pool
  7f81c1b8f700 / admin_socket
  7f81c2390700 / service
  7f81c2b91700 / msgr-worker-2
  7f81c3392700 / msgr-worker-1
  7f81c3b93700 / msgr-worker-0
  7f81c5804a80 / radosgw
  max_recent     10000
  max_new         1000
  log_file /var/lib/ceph/crash/2024-01-23T13:09:25.023127Z_f9c5ce06-f7c6-4a2c-9504-a3f30fa27794/log
--- end dump of recent events ---

Cluster Status to submit:

  cluster:
    id:     f31bf636-769f-423b-bc43-5ccf3d1197b1
    health: HEALTH_WARN
            739 daemons have recently crashed
            11 mgr modules have recently crashed

  services:
    mon: 3 daemons, quorum a,b,d (age 8h)
    mgr: a(active, since 2h), standbys: b
    mds: 1/1 daemons up, 1 hot standby
    osd: 8 osds: 8 up (since 8h), 8 in (since 8d)
    rgw: 1 daemon active (1 hosts, 1 zones)

  data:
    volumes: 1/1 healthy
    pools:   15 pools, 281 pgs
    objects: 31.02k objects, 8.3 GiB
    usage:   26 GiB used, 3.5 TiB / 3.5 TiB avail
    pgs:     281 active+clean

  io:
    client:   1.2 KiB/s rd, 2 op/s rd, 0 op/s wr
  • Output of kubectl commands, if necessary

Environment:

  • OS: CentOS 7.9
  • Kernel: 3.10.0-1160.105.1.el7.x86_64 #1 SMP Thu Dec 7 15:39:45 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
  • Cloud provider or hardware configuration: bare metal, see above
  • Rook version: rook: v1.13.2 / go: go1.21.5
  • Storage backend version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
  • Kubernetes version: Client Version: v1.25.16 / Kustomize Version: v4.5.7 / Server Version: v1.25.16
  • Kubernetes cluster type: microk8s v1.25.16 revision 6254

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 34 (15 by maintainers)

Most upvoted comments

I ran into this issue today with SOME of my osds on a single host. Anyone found a work around?

Node is Ubuntu 22.04 kernel 5.15, small cluster with 2 control planes.

will reopen. and prioritize the ceph tracker