binaryninja-api: Something is triggering invisible work and sometimes re-analysis and leaking memory

Version and Platform (required):

  • Binary Ninja Version: 5.0.7195-dev (2abafca6)
  • OS: macOS
  • OS Version: 15.2
  • CPU Architecture: M1

Bug Description: Clicking around a bunch in HLIL of a DSC view seems to cause invisible work to be triggered (Binary Ninja doesn’t tell you its doing anything but starts running a bunch of threads for a long perfiod of time). In the steps to reproduce below it doesn’t seem to start doing re-analysis but I have had that happen before. Also it will start consuming a bunch of RAM and then not give it all back once its done.

This bug has been griefing quite a bit in larger database because the invisible work and re-analysis can take many hours and it can be triggered multiple times in the same session.

Steps To Reproduce: Please provide all steps required to reproduce the behavior:

  1. Open the DYLD Shared Cache from the IPSW for iPhone17,2 iOS 18.1.1 22B91
  2. Wait for initial auto analysis to finish.
  3. Load the image /System/Library/Frameworks/Foundation.framework/Foundation.
  4. Wait for auto analysis to finish.
  5. Go to the function at 0x182300964 called -[NSFilePresenterXPCMessenger _makePresenter:accommodateEvictionWithSubitemURL:completionHandler:] in HLIL. No particular reason for that function other than I was looking for references to objc_msgSend.
  6. Click at the following locations in order: the red box (the reference to ___stack_chk_guard then the blue box (the reference to the field _queue) then the green box (the call to _objc_msgSend) and then the line below. Image I think the main problem here is _objc_msgSend which just clicking on it by itself seems to trigger the issue. I’ve found clicking around more makes things worse. So if you were to click on many different references to things and intermingle clicking on _objc_msgSend the duration of the invisible work and the memory consumption gets worse.
  7. Observe Binary Ninja will start doing a lot of work in the background. bv.analysis_info will not show anything. It will also rapidly start consuming memory, some of which will be given back once it is done but a bunch will remain allocated. I observed 10s of GBs of memory being consumed after all analysis was completed just by clicking around like this.
  8. Repeat the steps again and observe that again lots of background work will begin and eventually memory consumption will start to go up again.

Additional Steps: Things get weirder

  1. Do something in the Python console to trigger an analysis update which will result in extra work being done. For some reason the analysis stopped short. You’ll notice in the screenshot above that the objc_msgSend calls have no parameters even though the function type is set to take 2. After analysis completes the function updates to look like this: Image Also these are the before and after memory usages when triggering the analysis update:
>>> get_memory_usage_info()
{'Active BasicBlock objects': 2312467, 'Active BinaryView objects': 6, 'Active Component objects': 0, 'Active FileMetadata objects': 3, 'Active FlowGraph objects': 0, 'Active Function objects': 162925, 'Active LowLevelILFunction objects': 13122, 'Active MediumLevelILFunction objects': 17400, 'Active Type objects': 3054578, 'Active TypeArchive objects': 0}
>>> get_memory_usage_info()
{'Active BasicBlock objects': 4606588, 'Active BinaryView objects': 6, 'Active Component objects': 0, 'Active FileMetadata objects': 3, 'Active FlowGraph objects': 0, 'Active Function objects': 162925, 'Active LowLevelILFunction objects': 52602, 'Active MediumLevelILFunction objects': 70040, 'Active Type objects': 3564068, 'Active TypeArchive objects': 0}

There’s a huge number more IL function objects. 2. Do the clicking around again and observe the same invisible work issue 3. Call get_memory_usage_info() and see the stats have changed weirdly again:

>>> get_memory_usage_info()
{'Active BasicBlock objects': 2456535, 'Active BinaryView objects': 6, 'Active Component objects': 0, 'Active FileMetadata objects': 3, 'Active FlowGraph objects': 0, 'Active Function objects': 162925, 'Active LowLevelILFunction objects': 15453, 'Active MediumLevelILFunction objects': 20508, 'Active Type objects': 3099131, 'Active TypeArchive objects': 0}

Also this will trigger another analysis update that will do a meaningful amount of work. Memory usage after that analysis update looks like this:

>>> get_memory_usage_info()
{'Active BasicBlock objects': 2398970, 'Active BinaryView objects': 6, 'Active Component objects': 0, 'Active FileMetadata objects': 3, 'Active FlowGraph objects': 0, 'Active Function objects': 162925, 'Active LowLevelILFunction objects': 13353, 'Active MediumLevelILFunction objects': 17708, 'Active Type objects': 3104037, 'Active TypeArchive objects': 0}
  1. You can keep doing this over and over; clicking around, waiting for the CPU usage to drop, triggering auto-analysis through the Python console, waiting for the analysis update to finish, rinse and repeat.

Additional Information:

  • You can probably easily replicate this using any copy of the DSC.
  • If it matters I have the DYLD Shared Cache in a project.
  • I’m not sure the memory issue was quite as bad in prior dev versions but maybe it was.
  • The more images loaded the longer the invisible work will run for.
  • Would not surprise me if this is related to #6458 and/or #6598

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 39 (18 by maintainers)

Most upvoted comments

Okay thanks for double checking the original cross reference issue. Sounds like we can proceed with fixing that now that the other issues and regressions from trying to resolve the other cross-linked ticket have been fixed.

Yeah, I was wondering if that would happen. The RequestAdvancedAnalysisData change was part of that fix. I’ll take another stab at trying to repro the issue here.

Oh yeah, you mentioned 7139 in the previous comment…threw me off 😃

Haha so I did, my bad, dyslexic moment.

Okay thanks for the verification. Glad it’s fixed and we are back to a stable point. I’ll have to investigate further why the change to RequestAdvancedAnalysisData was problematic.

Thanks for persisting on this, much appreciated.

I realise you said try 7140, I assume you meant 7320? Which I tried and things appear to be fixed. There doesn’t seem to be any memory leak and the background work runs for a bit (in this case of objc_msgSend with 59,000 references it runs for roughly 30 seconds) but then stops, instead of carrying on indefinitely. So it seems thats fixed this issue.

No problem. I reverted the RequestAdvancedAnalysisData change to see if that is the issue. Try 7140 out in about an hour.

If its helpful to fully replicate my setup then these are all the settings I have modified:

Image

Image

No, the invisible/background work, seems to be, as you pointed out above, related to IL preview generation for cross references. That stuff still occurs and still has a memory leak.

I don’t know whats going on with the memory leak but I’m guessing the extensive work being done is due to a high number of cross references and it not being cancelled in the case that the user has moved on from highlighting the thing that cross references are being generated for. In addition, to what you mentioned above, about only generating IL previews for cross references in the viewport.

Found the regression that shows up in 5.1.7295. Will be fixed in the next build which is 5.1.7303-dev