rook: daemon_id used for RGW pods is not unique in case of multiple instances

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior:

Just to give some background related to this issue. With the introduction of ceph-exporter some of the performance metrics are generated by this new component (instead of the prometheus mgr module). The problem is that this component doesn’t have some metadata related to the metrics such as: hostname, daemon, etc which is available only on the mgr prometheus module. To fix this problem, ceph-exporter uses a unique label instance_id as key to join the metrics generated by ceph-exporter with the metadata coming from the prometheus module rgw_metadata.

In practice ceph-exporter only has access to the rgw daemon socket filename which it is used in this case to extract the instance_id. In cephadm deployments for example this value is set to hrgsea from daemon_name=rgw.foo.ceph-node-00.hrgsea.2.94739968030880 which is a unique value across all the rgw daemons. However, in a Rook deployment this is not the case. Following are the socket names generated for an object store with the name my-store and two instances:

$ ls -l /var/lib/rook/exporter/
total 0
srwxr-xr-x 1 167 167 0 Feb  6 08:38 ceph-client.ceph-exporter.asok
srwxr-xr-x 1 167 167 0 Feb  6 08:39 ceph-client.rgw.my.store.a.1.94334785460384.asok
srwxr-xr-x 1 167 167 0 Feb  6 08:39 ceph-client.rgw.my.store.a.1.94820349827232.asok

In this case ceph-exporter will ends up setting instance_id to a which is not unique.

On the rook side, daemon_id label generated for each rgw instance is not unique either. For example, using the following spec gives place to two instances with daemon_id set to my-store

apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: my-store
  namespace: rook-ceph # namespace:cluster
spec:
  metadataPool:
    replicated:
      size: 1
  dataPool:
    replicated:
      size: 1
  preservePoolsOnDelete: false
  gateway:
    port: 80
    # securePort: 443
    instances: 2

This daemon id is used to generate later the daemon socket name. So we need it to be unique otherwise ceph-exporter cannot extract this information from the filename.

Expected behavior:

Rook should append the corresponding letter (a,b, c… ) to the daemon_id to make it unique

How to reproduce it (minimal and precise):

Use the provided spec to create RGW instances and observe the daemon_id for each one of them.

File(s) to submit:

  • Cluster CR (custom resource), typically called cluster.yaml, if necessary

Logs to submit:

  • Operator’s logs, if necessary

  • Crashing pod(s) logs, if necessary

    To get logs, use kubectl -n <namespace> logs <pod name> When pasting logs, always surround them with backticks or use the insert code button from the Github UI. Read GitHub documentation if you need help.

Cluster Status to submit:

  • Output of kubectl commands, if necessary

    To get the health of the cluster, use kubectl rook-ceph health To get the status of the cluster, use kubectl rook-ceph ceph status For more details, see the Rook kubectl Plugin

Environment:

  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Cloud provider or hardware configuration:
  • Rook version (use rook version inside of a Rook Pod):
  • Storage backend version (e.g. for ceph do ceph -v):
  • Kubernetes version (use kubectl version):
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):
  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 2
  • Comments: 25 (15 by maintainers)

Most upvoted comments

With newer versions of Prometheus and/or kube-prometheus-stack, this causes issues. When two RGW pods are scheduled on the same node we now see the following in our Prometheus logs:

ts=2024-06-18T21:34:01.914Z caller=scrape.go:1738 level=warn component="scrape manager" scrape_pool=serviceMonitor/rook-ceph/rook-ceph-exporter/0 target=http://10.62.1.5:9926/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=45

This also leads to PrometheusDuplicateTimestamps alerts.

Based on conversation with rgw team, we will need a new cli arg to pass the unique id, that is independent from the auth settings (keyring and daemon name). Then Rook can pass the pod name suffix for this id.

I’m curious about Redo’s thoughts.

Otherwise, I’ll summarize an idea from huddle Deepika gave: we could see if RGW would be willing to add another sub-id field that would allow Rook to have its cake and eat it too so to speak.

The instance_id allows Rook to give each RGW the same keyring, and that in turn allows easy scale out using the k8s horizontal pod autoscaler (HPA), which is important for some users and something we’re planning to invest more in soonish. An additional ID would then allow each RGW to be further identified so as to not have naming/id collisions in the metrics.

My guess is that this could be easier to implement than a solution that would allow RGWs to all share the same keyring but still have different IDs, but I can’t be 100% sure.

We discussed in huddle, but I wanted to add a couple notes here. It may be possible to get RGWs to share a keyring while still issuing each one a unique ID using the Pod ID. Looking through the cephx doc (link below), I notice that the mons (which share a keyring) seem to use the principal mon., with a trailing dot. Conversely mgr, OSD, etc. use principals like osd.{$id}. I wonder if it’s possible to add a trailing dot to the RGW principal we use for its auth to allow us to reuse the same key for all RGWs.

https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/

I’ve been looking around for ceph examples, and they are hard to find. This doc (below) uses ceph tell osd.*, so maybe we can try using the * in the principal as well.

https://docs.ceph.com/en/pacific/rados/configuration/ceph-conf/#override-values

This doc may also have helpful info, but I can’t say for sure: https://docs.ceph.com/en/latest/rados/operations/user-management/

After looking deeply, I am becoming concerned that the rgw.<store-name>.a. strategy may not work, but it is worth trying at least.

The only way we can continue to use a single RGW deployment, is if we can use a single keyring even with different pod names at runtime. If the operator creates separate keyrings for each RGW instance, this means we have to create separate deployments, which means we have to revert back to the old behavior and we miss out on benefits like the pod autoscaler. To avoid that regression, we would really need a way for the metrics to have some unique id without a separate keyring.

We discussed in huddle using the random pod ID as each RGWs individual ID. Redo says cephadm uses a random ID for each RGW (though I’m not clear exactly as to when the ID changes with cephadm). I think this suggests that Rook can safely use the randomized pod ID to achieve a similar thing without much risk.

I think the solution will be pretty easy and involve 2 small changes:

  • use the downward API to apply the pod ID into an env var: POD_ID
    • can use POD_NAME to see a similar example
  • use that env var in the flag that applies the RGW’s daemon name
    • something like --daemon-id=rook-ceph-rgw-my-store.a.$(POD_ID) (I don’t remember the flag name or full details, so don’t take this as a spec, just a hint)

The reason why RGWs don’t have unique IDs are for a few reasons.

In our experience, RGWs generally don’t need to have unique IDs.

  • RGWs use a standard HTTP(S) web protocol and don’t have the same hard requirements from Ceph for them to have individual IDs.
  • Usually, if a user wants to change an RGW setting, they want to change all RGWs and not just one. Thus, users also don’t need them to have unique IDs.
  • As a side note, this also means that RGWs are the only component in Rook that we are able to use a single Service for, versus all other daemons needing individual Services.

Technically speaking, in Kubernetes, the reason comes down to using a Deployment with Replicas >1 to run RGWs. This is a much more simplified deployment model than tracking individual deployments for RGWs. If Ceph weren’t so picky about the IDs of things like mons and OSDs, Rook would prefer to use the same model for deploying those as well.

I want to understand more about why you are flagging this as an issue? Is this blocking some feature? Or is it just that it’s confusing that things are different for RGWs?

Another deployment model that offers more ID management is StatefulSets. These work like Deployments, but each replica has a unique ID added. This could be used for RGWs if we do find that we need unique IDs for each.

However, I would still suggest that we try to look into making changes to Ceph such that it doesn’t care about RGW individual identity as much, if possible. It was a win for the Rook project when we could use a single Deployment for all RGWs to simplify the management.

We used to manage the RGWs like we do other daemons, where each has a different deployment. I think the code for handling this might still be in Rook, but I’m unsure.

However, this is not the most desired design. We would like to make the fullest use of k8s primitives as possible, and in this case, continuing to be able to set a number of replicas for an RGW deployment would be ideal. Following on this idea, it would be best if RGWs continued to not need individual IDs.

If it does become necessary, perhaps something we could do is see whether each RGW can take its ID using the Pod downward API if necessary. Or it could be better to use a StatefulSet for RGWs to do a similar thing.