binaryninja-api: Something is triggering invisible work and sometimes re-analysis and leaking memory
Version and Platform (required):
- Binary Ninja Version: 5.0.7195-dev (2abafca6)
- OS: macOS
- OS Version: 15.2
- CPU Architecture: M1
Bug Description: Clicking around a bunch in HLIL of a DSC view seems to cause invisible work to be triggered (Binary Ninja doesn’t tell you its doing anything but starts running a bunch of threads for a long perfiod of time). In the steps to reproduce below it doesn’t seem to start doing re-analysis but I have had that happen before. Also it will start consuming a bunch of RAM and then not give it all back once its done.
This bug has been griefing quite a bit in larger database because the invisible work and re-analysis can take many hours and it can be triggered multiple times in the same session.
Steps To Reproduce: Please provide all steps required to reproduce the behavior:
- Open the DYLD Shared Cache from the IPSW for iPhone17,2 iOS 18.1.1 22B91
- Wait for initial auto analysis to finish.
- Load the image
/System/Library/Frameworks/Foundation.framework/Foundation. - Wait for auto analysis to finish.
- Go to the function at
0x182300964called-[NSFilePresenterXPCMessenger _makePresenter:accommodateEvictionWithSubitemURL:completionHandler:]in HLIL. No particular reason for that function other than I was looking for references toobjc_msgSend. - Click at the following locations in order: the red box (the reference to
___stack_chk_guardthen the blue box (the reference to the field_queue) then the green box (the call to_objc_msgSend) and then the line below.I think the main problem here is
_objc_msgSendwhich just clicking on it by itself seems to trigger the issue. I’ve found clicking around more makes things worse. So if you were to click on many different references to things and intermingle clicking on_objc_msgSendthe duration of the invisible work and the memory consumption gets worse. - Observe Binary Ninja will start doing a lot of work in the background.
bv.analysis_infowill not show anything. It will also rapidly start consuming memory, some of which will be given back once it is done but a bunch will remain allocated. I observed 10s of GBs of memory being consumed after all analysis was completed just by clicking around like this. - Repeat the steps again and observe that again lots of background work will begin and eventually memory consumption will start to go up again.
Additional Steps: Things get weirder
- Do something in the Python console to trigger an analysis update which will result in extra work being done. For some reason the analysis stopped short. You’ll notice in the screenshot above that the
objc_msgSendcalls have no parameters even though the function type is set to take 2. After analysis completes the function updates to look like this:Also these are the before and after memory usages when triggering the analysis update:
>>> get_memory_usage_info()
{'Active BasicBlock objects': 2312467, 'Active BinaryView objects': 6, 'Active Component objects': 0, 'Active FileMetadata objects': 3, 'Active FlowGraph objects': 0, 'Active Function objects': 162925, 'Active LowLevelILFunction objects': 13122, 'Active MediumLevelILFunction objects': 17400, 'Active Type objects': 3054578, 'Active TypeArchive objects': 0}
>>> get_memory_usage_info()
{'Active BasicBlock objects': 4606588, 'Active BinaryView objects': 6, 'Active Component objects': 0, 'Active FileMetadata objects': 3, 'Active FlowGraph objects': 0, 'Active Function objects': 162925, 'Active LowLevelILFunction objects': 52602, 'Active MediumLevelILFunction objects': 70040, 'Active Type objects': 3564068, 'Active TypeArchive objects': 0}
There’s a huge number more IL function objects.
2. Do the clicking around again and observe the same invisible work issue
3. Call get_memory_usage_info() and see the stats have changed weirdly again:
>>> get_memory_usage_info()
{'Active BasicBlock objects': 2456535, 'Active BinaryView objects': 6, 'Active Component objects': 0, 'Active FileMetadata objects': 3, 'Active FlowGraph objects': 0, 'Active Function objects': 162925, 'Active LowLevelILFunction objects': 15453, 'Active MediumLevelILFunction objects': 20508, 'Active Type objects': 3099131, 'Active TypeArchive objects': 0}
Also this will trigger another analysis update that will do a meaningful amount of work. Memory usage after that analysis update looks like this:
>>> get_memory_usage_info()
{'Active BasicBlock objects': 2398970, 'Active BinaryView objects': 6, 'Active Component objects': 0, 'Active FileMetadata objects': 3, 'Active FlowGraph objects': 0, 'Active Function objects': 162925, 'Active LowLevelILFunction objects': 13353, 'Active MediumLevelILFunction objects': 17708, 'Active Type objects': 3104037, 'Active TypeArchive objects': 0}
- You can keep doing this over and over; clicking around, waiting for the CPU usage to drop, triggering auto-analysis through the Python console, waiting for the analysis update to finish, rinse and repeat.
Additional Information:
- You can probably easily replicate this using any copy of the DSC.
- If it matters I have the DYLD Shared Cache in a project.
- I’m not sure the memory issue was quite as bad in prior dev versions but maybe it was.
- The more images loaded the longer the invisible work will run for.
- Would not surprise me if this is related to #6458 and/or #6598
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 39 (18 by maintainers)
Okay thanks for double checking the original cross reference issue. Sounds like we can proceed with fixing that now that the other issues and regressions from trying to resolve the other cross-linked ticket have been fixed.
Yeah, I was wondering if that would happen. The
RequestAdvancedAnalysisDatachange was part of that fix. I’ll take another stab at trying to repro the issue here.Haha so I did, my bad, dyslexic moment.
Thanks for persisting on this, much appreciated.
I realise you said try
7140, I assume you meant7320? Which I tried and things appear to be fixed. There doesn’t seem to be any memory leak and the background work runs for a bit (in this case ofobjc_msgSendwith 59,000 references it runs for roughly 30 seconds) but then stops, instead of carrying on indefinitely. So it seems thats fixed this issue.No problem. I reverted the RequestAdvancedAnalysisData change to see if that is the issue. Try
7140out in about an hour.If its helpful to fully replicate my setup then these are all the settings I have modified:
No, the invisible/background work, seems to be, as you pointed out above, related to IL preview generation for cross references. That stuff still occurs and still has a memory leak.
I don’t know whats going on with the memory leak but I’m guessing the extensive work being done is due to a high number of cross references and it not being cancelled in the case that the user has moved on from highlighting the thing that cross references are being generated for. In addition, to what you mentioned above, about only generating IL previews for cross references in the viewport.
Found the regression that shows up in
5.1.7295. Will be fixed in the next build which is5.1.7303-dev