eigen: dyld crash at launch

• A colleague ran into a situation where the app would crash on launch. Important to note is that she was also seeing it with other apps, such as LinkedIn.app.
• A restart of her device ‘fixed’ the issue.
• These crashes were not being reported to HockeyApp, meaning it was definitely happening very early in the process.
• Extracting the crash reports directly from the device revealed a dyld error.
• Checking iTunesConnect for crash reports revealed that we had ~80 similar crash reports.

TL;DR

This appears to happen when the device is out of memory, a restart of the device ‘fixes’ it. Presumably this neat trick does so as well.

Info about the errors

Dyld Error Message:
Dyld Message: Library not loaded: @rpath/CocoaLumberjack.framework/CocoaLumberjack
  Referenced from: /var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Artsy
  Reason: no suitable image found.  Did find:
    /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack:
    mremap_encrypted() => -1, errno=12 for /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack

    /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack:
    mremap_encrypted() => -1, errno=12 for /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack

    /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/C
  Dyld Version: 370.1

• The error message and codes are always the same.
• The error is not specific to a framework, it happens with various frameworks.
• It appears to have started from iOS 9.1 and up (but that could very well be iOS 9).

Known affected versions

These are the various versions for which we have crash reports.

iOS versions

~/D/dyld crash » ack -h 'OS Version:' **/*.crash | sort | uniq
OS Version:          iOS 9.1 (13B143)
OS Version:          iOS 9.2 (13C75)
OS Version:          iOS 9.2.1 (13D15)
OS Version:          iOS 9.2.1 (13D20)
OS Version:          iOS 9.3 (13E5214d)
OS Version:          iOS 9.3 (13E5225a)

dyld versions

~/D/dyld crash » ack -h 'Dyld Version:' **/*.crash | sort | uniq
  Dyld Version: 370.1
  Dyld Version: 370.6
  Dyld Version: 390.7

Device versions

~/D/dyld crash » ack -h 'Hardware Model:' **/*.crash | sort | uniq
Hardware Model:      iPad3,4
Hardware Model:      iPad4,4
Hardware Model:      iPad4,7
Hardware Model:      iPad5,3
Hardware Model:      iPad5,4
Hardware Model:      iPad6,7
Hardware Model:      iPad6,8
Hardware Model:      iPhone6,1
Hardware Model:      iPhone7,1
Hardware Model:      iPhone7,2
Hardware Model:      iPhone8,1
Hardware Model:      iPhone8,2

Retracing

1. The error is raised here in dyld.
2. Only place where mremap_encrypted could return -1 is here.
3. vn_getpath calls build_path, which in turn has a few possible locations where it could be returning -1:
   • https://github.com/opensource-apple/xnu/blob/977178fc3e01a280624e2095838cb9530763cda9/bsd/vfs/vfs_cache.c#L404
   • https://github.com/opensource-apple/xnu/blob/977178fc3e01a280624e2095838cb9530763cda9/bsd/vfs/vfs_cache.c#L446
   • https://github.com/opensource-apple/xnu/blob/977178fc3e01a280624e2095838cb9530763cda9/bsd/vfs/vfs_cache.c#L509
4. But trying to hunt that further down would take a lot more time, suffice to say that:
   • if mremap_encrypted returns -1, it most likely corresponds to ERESTART: https://github.com/opensource-apple/xnu/blob/2fa84067f6cdeb23267f877ca4fd26201316da1b/bsd/sys/errno.h#L270
   • if errno is 12, it most likely corresponds to ENOMEM: https://github.com/opensource-apple/xnu/blob/2fa84067f6cdeb23267f877ca4fd26201316da1b/bsd/sys/errno.h#L102

Caveats

Alas, the latest dyld sources to be released (at the time of writing) appear to be for version 360.18.

Taking a cursory look at the decompiled code for ImageLoaderMachOCompressed::registerEncryption from a firmware image for iOS 9.2 for iPad3,4, does not appear to have any real changes.

~/t/ios-9.2-iPad3,4 » unzip iPad3,4_9.2_13C75_Restore.ipsw 
Archive:  iPad3,4_9.2_13C75_Restore.ipsw
  inflating: 058-25903-078.dmg       
  inflating: 058-26242-078.dmg       
  inflating: 058-26268-078.dmg
[…]

~/t/ios-9.2-iPad3,4 » ~/Code/ObjectiveC/Debugging/xpwn/dmg/dmg extract 058-26268-078.dmg extracted.dmg -k 7ccd940dfda4b304c87050997e9fd70b458f21b5f3845262d985730b23f41e84aa25dae3
Writing out data..
[…]

~/t/ios-9.2-iPad3,4 » open extracted.dmg
~/t/ios-9.2-iPad3,4 » hopper -e /Volumes/Castlerock13C75.P101OS/usr/lib/dyld

int ImageLoaderMachOCompressed::registerEncryption(encryption_info_command const*, ImageLoader::LinkContext const&)(void * arg0, void * arg1) {
    r0 = arg0;
    r7 = (sp - 0x4 - 0x4 - 0x4 - 0x4 - 0x4) + 0xc;
    sp = sp - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4;
    r6 = arg1;
    r8 = r2;
    r4 = r0;
    if (r6 != 0x0) {
            r5 = 0x0;
            do {
                    r2 = *(*r4 + 0xe8);
                    if ((r2)(r4, r5, r2) != 0x0) goto (null);
                    r5 = r5 + 0x1;
            } while (true);
            r1 = r5;
            r2 = *(*r4 + 0x104);
            r0 = (r2)(r4, r1, r2);
            r10 = r6 + 0x8;
            asm{ ldm.w      sl, {r1, r5, sl} };
            r2 = *(r8 + 0xb2);
            r8 = *(r0 + 0x4);
            r6 = r0 + r1;
            r11 = *(r0 + 0x8);
            if (r2 != 0x0) {
                    dyld::log("                      0x%08lX->0x%08lX configured for FairPlay decryption\n");
            }
            r0 = _mremap_encrypted();
            r5 = r0;
            if (r5 != 0x0) {
                    r0 = ___error();
                    r3 = *(r4 + 0x4);
                    r2 = *r0;
                    r1 = r5;
                    r0 = dyld::throwf("mremap_encrypted() => %d, errno=%d for %s\n");
                    // …
            }
    }
    return r0;
}

Conclusion

The crash at launch definitely feels like a very bad experience and I feel like iOS should handle this more gracefully. Maybe show an alert? Will have a think about this and possibly file a radar.

I’m not sure what/if we can do about this. We could probably improve it somewhat by trimming some framework fat from the bundle, but I don’t think these are large anyways.

It would be interesting to know if such failures occurred prior to iOS 9 and if they were labelled differently.

Status

• A couple of radar tickets have already been filed:
  • http://www.openradar.me/radar?id=4952667885404160 (mine)
  • http://www.openradar.me/radar?id=5054516256833536
  • http://www.openradar.me/radar?id=4992459683659776
• A DTS ticket has been filed http://twitter.com/senior/status/707996353964728321
• A guess has been made as to the issue being a jetsam change https://twitter.com/Catfish_Man/status/707955795493687296 (jetsam is the process in which the kernel basically kills apps to free up memory)