taiHEN: crash when hooking sceIoClose() in a kernel module

Issue & replication

As the title indicates, I’m encountering a crash whenever I try to hook sceIoClose() that I am beginning to think might be a taiHEN issue rather than something with my code.

You can find a complete simple project to demonstrate the issue, with a simple kernel module and a test application that takes care of loading/unloading it (so that you don’t have to crash the whole boot process by adding the module to tai/config.txt) at: https://github.com/VitaSmith/sceIoClose

The instructions on how to recompile and run the module and test application to replicate the crash are in the Readme.

I have been testing with taiHen on fw 3.60 and I really can’t figure out what it is I might be doing wrong in my code to produce the crash.

Code

Basically, the code for the hook is:

int hook_user_close(SceUID fd)
{
    printf("sceIoClose(0x%08X) (Before TAI_CONTINUE)\n", fd);
    int r = TAI_CONTINUE(int, close_ref, fd);
    printf("r = 0x%08X (After TAI_CONTINUE)\n", r);
    return r;
}

And the installation code:

    // sceIoClose Hook
    close_id = taiHookFunctionExportForKernel(KERNEL_PID, &close_ref,
                                   "SceIofilemgr", TAI_ANY_LIBRARY,
                                   0xC70B8886, hook_user_close);
    printf("- sceIoClose Hook ID: 0x%08X\n", close_id);

However, the end result is that a crash occurs in TAI_CONTINUE as per the log output below:

Loading IO logger kernel driver...
- sceIoOpen Hook ID: 0x00052C8D
- sceIoClose Hook ID: 0x00052BEF
sceIoOpen('ux0:app/IO_LOGGER/eboot.bin') = 0x400100A3
sceIoClose(0x400100A3) (Before TAI_CONTINUE)

Things I tried

Removing logging altogether, in case the problem had to do with calling ksceIoClose() → Still crashes!
Overriding sceIoCloseForDriver() instead sceIoClose() → This doesn’t crash but of course the override I need is sceIoClose(). This does seem to confirm that the problem appears to be only with sceIoClose() override however.
Using the actual SceIofilemgr NID (0xF2FF276E) instead of TAI_ANY_LIBRARY → Same issue.
Updating VitaSDK to latest → Still crashes!
Not calling sceIoClose() in the test app, but waiting to unload the module → Still crashes as soon as a background app calls sceIoClose().
Adding void *args as extra parameters to sceIoClose() and TAI_CONTINUE() just in case → Same issue.
Using SceUID as return value instead of int → Same issue.

After having spent the last 24 hours trying to figure this out, I wouldn’t mind having some expert input on this issue, as I really can’t figure out what may be wrong in the sceIoClose() override when the sceIoOpen() one is working just fine. At this stage, I have reason to believe that the crash might have to do with taiHEN itself, which is why I am logging this issue.

About this issue

Original URL
State: open
Created 6 years ago
Comments: 20 (8 by maintainers)

Commits related to this issue

add sceIoOpen/sceIoClose hooks This was tricky, as there seems to be a taiHEN bug when hooking sceIoClose() See https://github.com/yifanlu/taiHEN/issues/84 Also make sure the hooks are guarded by ENT... — committed to VitaSmith/vpfs by VitaSmith 6 years ago

Most upvoted comments

Things I tried

Using the actual SceIofilemgr NID (0xF2FF276E) instead of TAI_ANY_LIBRARY → Same issue.

Can you please read the original issue?

I must admit I’m starting to get a bit annoyed having to dispell all of the obvious stuff that I already tried, which I made sure to document when I opened the original issue.

That’s not to say I don’t appreciate your willingness to help (I really do, coz you are a lot more familiar with the system than I am), but at this stage, if you think you have an idea about what in the code is causing the issue with sceIoClose(), I would prefer if you simply forked the sample project, tested the .skprx with your added modification, as a tai/config.txt plugin, and then post your code modifications to make it work either here or as a pull request on the project.

Note that I don’t care if the sceIoClose() hook logs anything, as long as it does close the user fds that it is presented with.

VitaSmith on Sep 6, 2018

Thanks for looking into this. It’s not non-compliant to save R3 even if it doesn’t have to be saved. However I agree it’s a suspicious piece of code. If you make TAI_CONTINUE take 4 args, make sure your patched function also takes 4 args and no other hooks are installed on it. Someone may want to try this out to confirm.

yifanlu on Sep 9, 2018

I’m pretty sure you can achieve non ARM ABI compliant behaviour with vanilla gcc through inline assembly.

The fact is that nobody here knows why r3 is being pushed and popped in sceIoClose(), in an apparent violation of the ARM ABI specifications.

My understanding is that, if Sony’s code was compiled with an ABI compliant toolchain, you should never observe a push/pop r3 at the beginning and end of a function call. Yet, this is what we are seeing. So, unfortunately, there is a nonstandard behaviour that we have no choice but to take into account.

Also, I can confirm that when I apply my own hooker (Boy is that a complete pain in the ass to replicate! - I have even more respect for what you must have gone through, when developing taiHEN, now that I’ve seen just how much work is required to simply get hooks that work), I get a stable sceIoClose() override, with a kernel module that can be loaded from tai/config.txt and doesn’t make the Vita crash or freeze.

Thus, as far as I am concerned, there is something that should be improved or fixed in the manner taiHEN applies hooks, because I can demonstrate that, if you use an alternate method to create your hooks, you won’t observe the crash that you observe with taiHEN… and I doubt everybody will want to go through the ordeal of creating their own custom hooker, just to work around what seems to be a problematic taiHEN behaviour.

I should also point out that one thing I tried, without success, was to create a taiHEN hook, with TAI_CONTINUE for an sceIoClose that takes 4 parameters, in the hope that maybe the taiHEN hooker would preserve r3 by the time it jumped back to the original code. However, that doesn’t seem to work either…

I guess the one way to prove or disprove the r3 hypothesis is to just change all the the push {r3, r4, r5, lr} and pop {r3, r4, r5, pc} in sceIoClose() to push {r4, r5, lr} / pop {r4, r5, pc}, and see what happens. I haven’t tried that, first because it’s a bit tricky (there are a few caveats to doing that properly so that you don’t get a crash for other reasons), and second because, now that I’ve lost about a week on this issue, I’d rather get back to making some progress with what I was doing before I got stuck with this, and I estimate that I have provided more than enough information for whoever is willing to investigate this further…

VitaSmith on Sep 9, 2018