Dopamine: Random Reboots (Spinlock Timeout Panic)

Here is an attempt at a more indepth explanation of the issue, to my best current understanding of it. Keep in mind it is based on assumptions that are basically impossible to verify.

So in a multithreaded system “locks” are used to prevent two threads from interfering with each other. By that one thread can acquire a lock, make the modification and unlock it. While locked, another thread trying to acquire the lock will wait until the object has been unlocked again.

A spinlock is essentially the same thing, just used for performance relevant stuff and the main difference is that a spinlock can time out if something takes the lock too long while another thread is trying to acquire the lock. So when acquiring a lock and the object is already locked, it would wait for a few ticks and if the object doesn’t get unlocked in that time frame, it will time out.

This mechanism by itself is not the issue, the issue has to do with memory pages. Every memory page (which describes an area of 16kB of RAM) has a spinlock so that there are no issues when multiple processes try to acquire the same page at the same time.

Specific pages can be mapped into multiple processes (e.g. if both load the same library), they reuse the same page in order to save memory. Tweaks want to overwrite such memory on a per-process basis, so they have to first make a process-specific copy of the existing mapping and map it on top of it, so that e.g. one page can be modified in one process while remaining stock in the other processes. The issue seems to specifically happen when mapping on top of a page that resides inside the dyld_shared_cache.

The problem is now that Apple probably never tested this kind of hooking and apparently when you do it in a lot of processes, it can cause the original page (the one of the shared mapping) to be paged out, because it’s not actively being used. Paging out a page essentially removes it from RAM and when it is accessed again it will be loaded again. On a stock system this will not happen because nothing has been hooked.

Now the root cause appears to be something trying to page a previously paged out shared/executable page back in, this triggers a preemption issue where one thread takes the spinlock and while it has that, it gets preempted to a different context which also takes the same spinlock (Preemption essentially is a mechanism that allows one thread to be used for something else even if it’s currently busy, code has to explicitely disable and reenable it if there is a piece of code that should always be executed in one go). So there seems to be one code path which is only invoked from this particular behaviour where Apple does not correctly disable preemption, leading to one thread taking the same spinlock two times, which makes it time out because the old context isn’t executing anymore and can’t unlock the spinlock again.

As for mitigating it, I tried messing with spinlock related variables to make the threshold that it takes for it to time out higher, unfortunately Apple screwed us over because everything related to that is KTRR protected, for which we do not have a bypass. I guess the proper fix would be to “wire down” (wiring down a page prevents it from being paged out) every to-be-hooked page before it’s overwritten to ensure that the page out never happens and therefore the code path involved in the issue doesn’t trigger, I tried a bunch of stuff so far but it seems it’s straight up impossible to acquire such a wiring from userspace, so it has to be done inside the kernel. Unfortunately the structures involved in this specific shared mapping that causes the issue are very convuluted and I have yet to find a way to get the correct page object to apply the wiring to.

So the next step to try and fix it would be to find the vm_page structure of a DSC page in kernel memory, so far all my attempts at finding such a structure have failed.