rook: Suspected memory leak in the Rook operator

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior: Rook operator memory usage continues to increase. The graph below shows the container_memory_working_set_bytes metrics for the Rook operator. The metrics have been reset as the node was restarted around 10:00 on 8 April.

rook operator memory used

Expected behavior: Memory usage does not continue to increase.

How to reproduce it (minimal and precise):

File(s) to submit:

  • Cluster CR (custom resource), typically called cluster.yaml, if necessary

Logs to submit:

Cluster Status to submit:

  • Result of ceph status command:
      cluster:
        id:     71c19227-1fd2-44ba-a883-00f8248ad926
        health: HEALTH_OK
     
      services:
        mon: 3 daemons, quorum a,b,c (age 4d)
        mgr: a(active, since 4d), standbys: b
        osd: 3 osds: 3 up (since 5m), 3 in (since 4d)
     
      data:
        pools:   2 pools, 64 pgs
        objects: 3 objects, 577 KiB
        usage:   24 MiB used, 3.0 TiB / 3 TiB avail
        pgs:     64 active+clean
    

Environment:

  • OS (e.g. from /etc/os-release): Ubuntu 22.04.1
  • Kernel (e.g. uname -a): 5.15.0-58-generic
  • Rook version (use rook version inside of a Rook Pod): 1.13.1
  • Storage backend version (e.g. for ceph do ceph -v): 17.2.6
  • Kubernetes version (use kubectl version): 1.27.10

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 22 (22 by maintainers)

Most upvoted comments

@travisn I don’t have that error anymore. I got that error once, about 20 minutes after the upgrade, but I haven’t had that error for many days since then. (It may be that currentNamespaceOnly is now in effect due to the upgrade.) The problem has been resolved and this issue is closed. Thank you for your cooperation.

Based on my reading, I think this is controller-runtime behaving as expected. This thread (from 2022) [1] talks about how c-r can filter what objects get cached. But because Rook watches all namespaces and because of the number of different types of resources Rook managers, I think this tends to be a lot of stuff. It seems like there are probably some improvements Rook could make to help keep the cache size down, but I think that may be a bit of a tall order.

For example, the dapr project did so here (big PR): https://github.com/dapr/dapr/pull/5944 Elastic did so here: https://github.com/elastic/cloud-on-k8s/pull/5187/files There are others, but these 2 seem like the most relevant to Rook.

Each project I look at has done something slightly different, so we need to make sure the fix is tailored to Rook and considers all configurations that users might use (like multi-namespace CephObjectStoreUsers, OBCs, etc.). This isn’t something that we can implement without detailed discussions and good proof-of-concept investigations.

[1] https://groups.google.com/g/operator-framework/c/AIiDgRPJc00

Your operator log is full of error messages such as these:

2024-04-10 04:26:26.731566 E | ceph-cluster-controller: failed to reconcile CephCluster "ceph-canary-object-store/ceph-canary-object-store". failed to get cephCluster: unable to get: ceph-canary-object-store/ceph-canary-object-store because of unknown namespace for the cache
2024-04-10 04:26:26.731593 E | ceph-cluster-controller: failed to reconcile CephCluster "ceph-dotcom-blob-0/ceph-dotcom-blob-0". failed to get cephCluster: unable to get: ceph-dotcom-blob-0/ceph-dotcom-blob-0 because of unknown namespace for the cache
2024-04-10 04:26:26.731616 E | ceph-cluster-controller: failed to reconcile CephCluster "ceph-object-store/ceph-object-store". failed to get cephCluster: unable to get: ceph-object-store/ceph-object-store because of unknown namespace for the cache

Did you delete these clusters or some other unexpected operation? If you clean up these errors, it likely will stop growing in memory like that.