status-go: Memory leak at least since version v10.9.0

I have recently deployed the v10.9.0 release on our notify.prod fleet, which has been running v0.166.4 before.

Here is memory usage for status-go node on node-01.do-ams3.notify.prod for the last 20 days:

Image

You can clearly see the change in memory usage on the 3rd of March when the Docker image was upgraded.

This is not sustainable and needs to be investigated.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 22 (20 by maintainers)

Commits related to this issue

Most upvoted comments

So thereโ€™s a 50% chance itโ€™s @osmaczko commit ๐Ÿ˜‚

Looks good except for the large interim spike.

Image

Such spikes seem to appear before the upgrade to v10.9.0, so I consider it โ€œnormal.โ€

Image

Image

Thanks @jakubgs for blame. Iโ€™ll try to fix ASAP, sorry for inconvenience ๐Ÿ˜“

Weโ€™re getting closer, f80042c is broken:

Image

Weโ€™re down to 6 commits:

 > g sl v3.10.0~1..f80042c
f80042c5c 2024-11-22 Sale Djenic          > chore(wallet)_: ens resolver identified under ens api  (origin/test-mem-leak, test-mem-leak)
c5dede93b 2024-11-22 Alex Jbanca          > feat(browser connect)_: Implementing signTypedData_V4 
c5c28cc56 2024-11-22 Patryk Osmaczko      > chore(logging)_: switch to zap.Logger as central logger 
d18dfeffc 2024-11-22 Patryk Osmaczko      > chore(logging)_: introduce geth adapter 
ed36d6054 2024-11-22 Alex Jbanca          > fix(BC)_: Fix personal_sign RPC event processing 
3912e7645 2024-11-22 Patryk Osmaczko      > chore(logging)_: add benchmark for custom zap core 
eae034aea 2024-11-22 Patryk Osmaczko      > feat(logging)_: introduce custom zap.Core enabling runtime changes 
3824691f7 2024-11-22 frank                > chore_: remove CLI option  MEDIA_HTTPS (#6112)  (tag: v3.10.0)

It seems v3.7.0 is fine:

Image

Next should be v7.1.0 from 2024-12-06.

Indeed, it does look better. I have no idea what this spike is tho. But if itโ€™s only temporary we can live with it, until it becomes too big of a problem. Thanks for looking into the issue.

Looks promising, will give it few more hours.

Image

@jakubgs The neatness of your investigation is one for the books and deserves a round of applause, thank you! ๐Ÿ‘

v3.10.0 is fine:

Image

So thereโ€™s no more releases to check, just individual commits.

And v4.0.0 is a dud too:

Image

Next up v3.10.0.

It appears v3.0.0 is free from the memory leak based on one day of metrics:

Image

Nest closest release between that and v10.9.0 is v8.0.0 from 2024-12-23.

Iโ€™ve decided to try to bisect it a bit. Since v0.166.4 is from 30 Aug 2023 and v10.9.0 is from 27 Feb 2025, which means the middle happens to be on 2024-05-29:

Version Commit Commit Date Deploy Date Leak?
v0.166.4 bf748de216a2061bf14afb2a410599485ee1ee7c 2023-08-30 15:47:29 unknown ๐Ÿ†—
v0.179.13 4f4b7a9ce4a1b314e41ba4ea7ab9c4ad36b3f4df 2024-05-07 10:17:25 2025-03-10 ๐Ÿ†—
v3.0.0 a08319f6151467100ad4f65f15158227e90527bc 2024-10-07 14:05:37 2025-03-11 ๐Ÿ†—
v3.7.0 a08319f6151467100ad4f65f15158227e90527bc 2024-11-19 08:11:52 2025-03-13 ๐Ÿ†—
v3.10.0 3824691f782719112ea55f5a4bb405ebe7370f9b 2024-11-22 02:23:17 2025-03-17 ๐Ÿ†—
- ed36d60540f4cc96101ca47bacdf841e38e40bfd 2024-11-22 08:23:42 2025-03-20 ๐Ÿ†—
- d18dfeffca7a7dbae1c23d9c86173d630799b83c 2024-11-22 09:03:22 2025-03-22 ๐Ÿ†—
- c5c28cc5687963491ce2815c4523ce6248ff1adb 2024-11-22 12:32:49 2025-03-21 ๐ŸŽฏ
- f80042c5cbdbf69425b745bd24840602d1f36a17 2024-11-22 12:32:49 2025-03-19 โŒ
- 5fa57c80ec1baf9687828d089ed8a9848684ac1c 2024-11-22 12:32:49 2025-03-18 โŒ
v4.0.0 ad28f15531c76c216f08118d4055a140ee52f80f 2024-12-03 21:20:26 2025-03-16 โŒ
v6.0.0 991d5dfde5e49eaff3ae628798b01d74d9506cf6 2024-11-28 10:55:02 2025-03-15 โŒ
v7.1.0 d07e61f6157dce56324a1c6814f31515b62eee50 2024-12-06 14:16:54 2025-03-14 โŒ
v8.0.0 a4e36d49cd67935cc4e2a26236e7ded4877e3eed 2024-12-23 08:32:48 2025-03-12 โŒ
v10.9.0 fb7023693318aa997b0b3fbce5e9537f06593889 2025-02-27 16:29:20 2025-03-03 โŒ