dd-sdk-ios: iOS 15 crash on MobileDevice.swift

The crash

Random crash on iOS 15 MobileDevice.swift / isLowPowerModeEnabled

crash_info_entry_0
BUG IN CLIENT OF LIBPLATFORM: Trying to recursively lock an os_unfair_lock

Crashed: com.datadoghq.ios-sdk-rum-upload
0  libsystem_platform.dylib       0x1f3c620c0 _os_unfair_lock_recursive_abort + 36
1  libsystem_platform.dylib       0x1f3c5ca10 _os_unfair_lock_lock_slow + 304
2  Foundation                     0x184fea730 -[NSProcessInfo(NSProcessInfoHardwareState) isLowPowerModeEnabled] + 68
3  WebCore                        0x19329d004 <redacted> + 56
4  CoreFoundation                 0x1837c1ee8 __CFNOTIFICATIONCENTER_IS_CALLING_OUT_TO_AN_OBSERVER__ + 28
5  CoreFoundation                 0x18385db9c ___CFXRegistrationPost_block_invoke + 52
6  CoreFoundation                 0x183830f54 _CFXRegistrationPost + 456
7  CoreFoundation                 0x1837d7d54 _CFXNotificationPost + 716
8  Foundation                     0x184fdc028 -[NSNotificationCenter postNotificationName:object:userInfo:] + 96
9  Foundation                     0x1850549d4 NSProcessInfoNotifyPowerState + 188
10 Foundation                     0x184fea768 -[NSProcessInfo(NSProcessInfoHardwareState) isLowPowerModeEnabled] + 124
11 Datadog                        0x105285390 partial apply for closure #3 in MobileDevice.init(uiDevice:processInfo:) + 67 (MobileDevice.swift:67)
12 Datadog                        0x105287d50 DataUploadConditions.blockersForUpload() + 31 (BatteryStatusProvider.swift:31)
13 Datadog                        0x105288838 closure #1 in DataUploadWorker.scheduleNextUpload(after:) + 56 (DataUploadWorker.swift:56)
14 Datadog                        0x1052ad290 thunk for @escaping @callee_guaranteed () -> () + 4406776464 (<compiler-generated>:4406776464)
15 libdispatch.dylib              0x1834a2950 _dispatch_client_callout + 20
16 libdispatch.dylib              0x1834a5e04 _dispatch_continuation_pop + 504
17 libdispatch.dylib              0x1834b8d60 _dispatch_source_invoke + 1356
18 libdispatch.dylib              0x1834a9f84 _dispatch_lane_serial_drain + 368
19 libdispatch.dylib              0x1834aac10 _dispatch_lane_invoke + 392
20 libdispatch.dylib              0x1834b5318 _dispatch_workloop_worker_thread + 656
21 libsystem_pthread.dylib        0x1f3c651b0 _pthread_wqthread + 288
22 libsystem_pthread.dylib        0x1f3c64f50 start_wqthread + 8

Datadog SDK versions:

1.6.0

Last stable Datadog SDK version:

Related to iOS version

Volume:

Minor

OS version:

iOS 15

Deployment Target:

iOS 12, iPhone + iPad

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 15 (3 by maintainers)

Commits related to this issue

#609 observe changes to isLowPowerMode and cache the values as they change rather than polling for them on every log submission — committed to GoodNotes/dd-sdk-ios by earltedly 3 years ago
#609 observe changes to isLowPowerMode and cache the values as they change rather than polling for them on every log submission — committed to GoodNotes/dd-sdk-ios by earltedly 3 years ago
#609 observe changes to isLowPowerMode and cache the values as they change rather than polling for them on every log submission — committed to GoodNotes/dd-sdk-ios by earltedly 3 years ago
#609 - CR feedback (final class and use private dispatch queue) — committed to GoodNotes/dd-sdk-ios by earltedly 3 years ago
#609 Apply suggestions from code review Co-authored-by: Mert Buran <buranmert@gmail.com> — committed to GoodNotes/dd-sdk-ios by earltedly 3 years ago

Most upvoted comments

1.7.2 was just released and it should fix this problem. I’m closing this issue, hoping it has been ultimately solved 🤞. Please keep us posted if this crash disappeared from your radars or if we need to take any further actions on our side.

I also received an update to my Apple radar (FB9741207), confirming it’s an Apple bug: Screenshot 2021-11-08 at 12 31 48

ncreated on Nov 8, 2021

I actually just came in here to report this issue and submit a fix 😃.

We use a third-party pre-compiled framework that registers for updates to NSProcessInfoPowerStateDidChangeNotification and then when receiving a callback will immediately query ProcessInfo.isLowPowerModeEnabled. From what I’ve seen, this is a very common use case.

I’ve attempted to reproduce it myself, but have been unable. I’ve got some stats though…

DataDog is polling this every five seconds
1/1000 of our users on iOS 15 are experiencing this at least once (10% of those will see it again)
Our users tend to have quite long sessions

That points towards it being a new race condition in iOS 15 - sometimes calling isLowPowerModeEnabled will trigger an update which then notifies the observers before returning. Those observers will then re-enter the method within the same stack trace.

Unfortunately the current implementation in MobileDevice.swift will poll this value on every log submission. That gives a lot of opportunity for this to happen. I put out an interim fix which disabled it being called altogether which got rid of the majority of our crashes.

In the next hour or so I’ll submit a PR request here which switches it to registering for notifications and caching the changes rather than polling for them all the time. That should prevent re-entering on the same callstack and getting this deadlock.

earltedly on Sep 28, 2021

Thank you @pingd. This was very helpful as it explains WebKit symbols presence* in the backtrace 👌. Although I haven’t managed to reproduce the crash, we made a patch in #655. It will be released on Monday as 1.7.2 hotfix. I also submitted another feedback to Apple (FB9741207) so we can keep track they response and hopefully have it fixed in next iOS releases.

* In my tests, using NSAttributedString and .documentType for rendering HTML leads to loading WebKit stack under the hood. WebKit in turn, leverages power mode status for its own optimisation (reducing impact on LPM enabled). Don’t know if that’s relevant, but there were some changes in WebKit around the LMP logic on Dec 2020.

ncreated on Nov 5, 2021

Fixed in 1.7.1 - we mitigated the issue on our side, although the root cause needs to be fixed by Apple in ProcessInfo.isLowPowerModeEnabled implementation (FB9661108).

ncreated on Oct 4, 2021