status-go: Memory leak at least since version v10.9.0
I have recently deployed the v10.9.0 release on our notify.prod fleet, which has been running v0.166.4 before.
Here is memory usage for status-go node on node-01.do-ams3.notify.prod for the last 20 days:
You can clearly see the change in memory usage on the 3rd of March when the Docker image was upgraded.
This is not sustainable and needs to be investigated.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 22 (20 by maintainers)
Commits related to this issue
- fix: remove references to logger's core children References prevented GC to remove unused loggers. Potentially fixes #6402 — committed to status-im/status-go by osmaczko 2 years ago
- fix_: remove references to logger's core children References prevented GC to remove unused loggers. Potentially fixes #6402 — committed to status-im/status-go by osmaczko 2 years ago
- fix_: remove references to logger's core children References prevented GC to remove unused loggers. fixes: #6402 — committed to status-im/status-go by osmaczko 2 years ago
- fix_: remove references to logger's core children References prevented GC to remove unused loggers. fixes: #6402 — committed to status-im/status-go by osmaczko 2 years ago
- fix_: remove references to logger's core children References prevented GC to remove unused loggers. fixes: #6402 — committed to status-im/status-go by osmaczko 2 years ago
- fix_: remove references to logger's core children References prevented GC to remove unused loggers. fixes: #6402 — committed to status-im/status-go by osmaczko 2 years ago
- fix_: remove references to logger's core children (#6465) References prevented GC to remove unused loggers. fixes: #6402 — committed to status-im/status-go by osmaczko 2 years ago
- fix_: remove references to logger's core children References prevented GC to remove unused loggers. fixes: #6402 — committed to status-im/status-go by osmaczko 2 years ago
- fix_: remove references to logger's core children (#6510) References prevented GC to remove unused loggers. fixes: #6402 — committed to status-im/status-go by osmaczko 2 years ago
So thereโs a 50% chance itโs @osmaczko commit ๐
Looks good except for the large interim spike.
Such spikes seem to appear before the upgrade to v10.9.0, so I consider it โnormal.โ
Done: https://ci.infra.status.im/job/status-go/job/deploy-notify-prod/55/
Thanks @jakubgs for blame. Iโll try to fix ASAP, sorry for inconvenience ๐
Weโre getting closer, f80042c is broken:
Weโre down to 6 commits:
It seems
v3.7.0is fine:Next should be
v7.1.0from2024-12-06.Indeed, it does look better. I have no idea what this spike is tho. But if itโs only temporary we can live with it, until it becomes too big of a problem. Thanks for looking into the issue.
Looks promising, will give it few more hours.
@jakubgs The neatness of your investigation is one for the books and deserves a round of applause, thank you! ๐
v3.10.0is fine:So thereโs no more releases to check, just individual commits.
And
v4.0.0is a dud too:Next up
v3.10.0.It appears
v3.0.0is free from the memory leak based on one day of metrics:Nest closest release between that and
v10.9.0isv8.0.0from2024-12-23.Iโve decided to try to bisect it a bit. Since
v0.166.4is from30 Aug 2023andv10.9.0is from27 Feb 2025, which means the middle happens to be on2024-05-29:v0.166.42023-08-30 15:47:29unknownv0.179.132024-05-07 10:17:252025-03-10v3.0.02024-10-07 14:05:372025-03-11v3.7.02024-11-19 08:11:522025-03-13v3.10.02024-11-22 02:23:172025-03-17-2024-11-22 08:23:422025-03-20-2024-11-22 09:03:222025-03-22-2024-11-22 12:32:492025-03-21-2024-11-22 12:32:492025-03-19-2024-11-22 12:32:492025-03-18v4.0.02024-12-03 21:20:262025-03-16v6.0.02024-11-28 10:55:022025-03-15v7.1.02024-12-06 14:16:542025-03-14v8.0.02024-12-23 08:32:482025-03-12v10.9.02025-02-27 16:29:202025-03-03