rook: Suspected memory leak in the Rook operator
Is this a bug report or feature request?
- Bug Report
Deviation from expected behavior:
Rook operator memory usage continues to increase. The graph below shows the container_memory_working_set_bytes metrics for the Rook operator. The metrics have been reset as the node was restarted around 10:00 on 8 April.
Expected behavior: Memory usage does not continue to increase.
How to reproduce it (minimal and precise):
File(s) to submit:
- Cluster CR (custom resource), typically called
cluster.yaml, if necessary
Logs to submit:
- Operator’s logs
Cluster Status to submit:
- Result of
ceph statuscommand:cluster: id: 71c19227-1fd2-44ba-a883-00f8248ad926 health: HEALTH_OK services: mon: 3 daemons, quorum a,b,c (age 4d) mgr: a(active, since 4d), standbys: b osd: 3 osds: 3 up (since 5m), 3 in (since 4d) data: pools: 2 pools, 64 pgs objects: 3 objects, 577 KiB usage: 24 MiB used, 3.0 TiB / 3 TiB avail pgs: 64 active+clean
Environment:
- OS (e.g. from /etc/os-release): Ubuntu 22.04.1
- Kernel (e.g.
uname -a): 5.15.0-58-generic - Rook version (use
rook versioninside of a Rook Pod): 1.13.1 - Storage backend version (e.g. for ceph do
ceph -v): 17.2.6 - Kubernetes version (use
kubectl version): 1.27.10
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 22 (22 by maintainers)
@travisn I don’t have that error anymore. I got that error once, about 20 minutes after the upgrade, but I haven’t had that error for many days since then. (It may be that
currentNamespaceOnlyis now in effect due to the upgrade.) The problem has been resolved and this issue is closed. Thank you for your cooperation.Based on my reading, I think this is controller-runtime behaving as expected. This thread (from 2022) [1] talks about how c-r can filter what objects get cached. But because Rook watches all namespaces and because of the number of different types of resources Rook managers, I think this tends to be a lot of stuff. It seems like there are probably some improvements Rook could make to help keep the cache size down, but I think that may be a bit of a tall order.
For example, the dapr project did so here (big PR): https://github.com/dapr/dapr/pull/5944 Elastic did so here: https://github.com/elastic/cloud-on-k8s/pull/5187/files There are others, but these 2 seem like the most relevant to Rook.
Each project I look at has done something slightly different, so we need to make sure the fix is tailored to Rook and considers all configurations that users might use (like multi-namespace CephObjectStoreUsers, OBCs, etc.). This isn’t something that we can implement without detailed discussions and good proof-of-concept investigations.
[1] https://groups.google.com/g/operator-framework/c/AIiDgRPJc00
Your operator log is full of error messages such as these:
Did you delete these clusters or some other unexpected operation? If you clean up these errors, it likely will stop growing in memory like that.