firebase-ios-sdk: -[GDTCORFlatFileStorage batchWithEventSelector:batchExpiration:onComplete:] causes High CPU Usage Terminations
Description
We’ve been tracking an issue via Xcode Organizer for a bit where GoogleDataTransport is billed for the majority of our app’s high CPU terminations.
We’ve been unable to reproduce this in-house, so all we have to go on are the Xcode organizer Energy reports, and some specific user complaints. Our own app logs don’t show anything interesting either. I am not able to include the entire xcprocessorusagelog as it contains IP from my employer.
There are two distinct stacks, but they both generate this type of watchdog termination:
48 seconds cpu time over 56 seconds (86% cpu average), exceeding limit of 80% cpu over 60 seconds
High CPU Usage Termination
From the threads that are not related to GoogleDataTransport, we can tell this is happening very soon after applicationDidFinishLaunching
Sampled Backtrace (0)#0 (null) in __rename ()
#1 (null) in -[NSFilesystemItemMoveOperation main] ()
#2 (null) in __NSOPERATION_IS_INVOKING_MAIN__ ()
#3 (null) in -[NSOperation start] ()
#4 (null) in -[NSFileManager moveItemAtPath:toPath:error:] ()
#5 0xb760a68 in __75-[GDTCORFlatFileStorage batchWithEventSelector:batchExpiration:onComplete:]_block_invoke_2 at /Users/leonjuan/sources/connect-mobile-ios/Pods/GoogleDataTransport/GoogleDataTransport/GDTCORLibrary/GDTCORFlatFileStorage.m:226
#6 (null) in _dispatch_call_block_and_release ()
#7 (null) in _dispatch_client_callout ()
#8 (null) in _dispatch_lane_serial_drain ()
#9 (null) in _dispatch_lane_invoke ()
#10 (null) in _dispatch_workloop_worker_thread ()
#11 (null) in _pthread_wqthread ()
#12 (null) in start_wqthread ()
Video of stack With system libs: https://user-images.githubusercontent.com/115565643/228986078-42d3c516-b89a-45fc-9185-da120d0f8806.mov
Without system libs:
Similar stack:
Apologies for not being able to include the original xcprocessorusagelog
files from Xcode Organizer, but they contain IP from my employer.
Reproducing the issue
Unable to repro
Firebase SDK Version
8.15
Xcode Version
14.2
Installation Method
CocoaPods
Firebase Product(s)
Analytics, Crashlytics, Performance, Remote Config
Targeted Platforms
iOS
Relevant Log Output
energyDetail:
"48 seconds cpu time over 58 seconds (83% cpu average), exceeding limit of 80% cpu over 60 seconds"
If using Swift Package Manager, the project’s Package.resolved
n/a
If using CocoaPods, the project’s Podfile.lock
- Firebase (8.15.0):
- Firebase/Core (= 8.15.0)
- Firebase/Analytics (8.15.0):
- Firebase/Core
- Firebase/Core (8.15.0):
- Firebase/CoreOnly
- FirebaseAnalytics (~> 8.15.0)
- Firebase/CoreOnly (8.15.0):
- FirebaseCore (= 8.15.0)
- Firebase/Crashlytics (8.15.0):
- Firebase/CoreOnly
- FirebaseCrashlytics (~> 8.15.0)
- Firebase/Performance (8.15.0):
- Firebase/CoreOnly
- FirebasePerformance (~> 8.15.0)
- Firebase/RemoteConfig (8.15.0):
- Firebase/CoreOnly
- FirebaseRemoteConfig (~> 8.15.0)
- FirebaseABTesting (8.15.0):
- FirebaseCore (~> 8.0)
- FirebaseAnalytics (8.15.0):
- FirebaseAnalytics/AdIdSupport (= 8.15.0)
- FirebaseCore (~> 8.0)
- FirebaseInstallations (~> 8.0)
- GoogleUtilities/AppDelegateSwizzler (~> 7.7)
- GoogleUtilities/MethodSwizzler (~> 7.7)
- GoogleUtilities/Network (~> 7.7)
- “GoogleUtilities/NSData+zlib (~> 7.7)”
- nanopb (~> 2.30908.0)
- FirebaseAnalytics/AdIdSupport (8.15.0):
- FirebaseCore (~> 8.0)
- FirebaseInstallations (~> 8.0)
- GoogleAppMeasurement (= 8.15.0)
- GoogleUtilities/AppDelegateSwizzler (~> 7.7)
- GoogleUtilities/MethodSwizzler (~> 7.7)
- GoogleUtilities/Network (~> 7.7)
- “GoogleUtilities/NSData+zlib (~> 7.7)”
- nanopb (~> 2.30908.0)
- FirebaseCore (8.15.0):
- FirebaseCoreDiagnostics (~> 8.0)
- GoogleUtilities/Environment (~> 7.7)
- GoogleUtilities/Logger (~> 7.7)
- FirebaseCoreDiagnostics (8.15.0):
- GoogleDataTransport (~> 9.1)
- GoogleUtilities/Environment (~> 7.7)
- GoogleUtilities/Logger (~> 7.7)
- nanopb (~> 2.30908.0)
- FirebaseCrashlytics (8.15.0):
- FirebaseCore (~> 8.0)
- FirebaseInstallations (~> 8.0)
- GoogleDataTransport (~> 9.1)
- GoogleUtilities/Environment (~> 7.7)
- nanopb (~> 2.30908.0)
- PromisesObjC (< 3.0, >= 1.2)
- FirebaseInstallations (8.15.0):
- FirebaseCore (~> 8.0)
- GoogleUtilities/Environment (~> 7.7)
- GoogleUtilities/UserDefaults (~> 7.7)
- PromisesObjC (< 3.0, >= 1.2)
- FirebasePerformance (8.15.0):
- FirebaseCore (~> 8.0)
- FirebaseInstallations (~> 8.0)
- FirebaseRemoteConfig (~> 8.0)
- GoogleDataTransport (~> 9.1)
- GoogleUtilities/Environment (~> 7.7)
- GoogleUtilities/ISASwizzler (~> 7.7)
- GoogleUtilities/MethodSwizzler (~> 7.7)
- nanopb (~> 2.30908.0)
- FirebaseRemoteConfig (8.15.0):
- FirebaseABTesting (~> 8.0)
- FirebaseCore (~> 8.0)
- FirebaseInstallations (~> 8.0)
- GoogleUtilities/Environment (~> 7.7)
- “GoogleUtilities/NSData+zlib (~> 7.7)”
- GoogleAppMeasurement (8.15.0):
- GoogleAppMeasurement/AdIdSupport (= 8.15.0)
- GoogleUtilities/AppDelegateSwizzler (~> 7.7)
- GoogleUtilities/MethodSwizzler (~> 7.7)
- GoogleUtilities/Network (~> 7.7)
- “GoogleUtilities/NSData+zlib (~> 7.7)”
- nanopb (~> 2.30908.0)
- GoogleAppMeasurement/AdIdSupport (8.15.0):
- GoogleAppMeasurement/WithoutAdIdSupport (= 8.15.0)
- GoogleUtilities/AppDelegateSwizzler (~> 7.7)
- GoogleUtilities/MethodSwizzler (~> 7.7)
- GoogleUtilities/Network (~> 7.7)
- “GoogleUtilities/NSData+zlib (~> 7.7)”
- nanopb (~> 2.30908.0)
- GoogleAppMeasurement/WithoutAdIdSupport (8.15.0):
- GoogleUtilities/AppDelegateSwizzler (~> 7.7)
- GoogleUtilities/MethodSwizzler (~> 7.7)
- GoogleUtilities/Network (~> 7.7)
- “GoogleUtilities/NSData+zlib (~> 7.7)”
- nanopb (~> 2.30908.0)
- GoogleDataTransport (9.2.0):
- GoogleUtilities/Environment (~> 7.7)
- nanopb (< 2.30910.0, >= 2.30908.0)
- PromisesObjC (< 3.0, >= 1.2)
- GoogleSignIn (5.0.2):
- AppAuth (~> 1.2)
- GTMAppAuth (~> 1.0)
- GTMSessionFetcher/Core (~> 1.1)
- GoogleUtilities/AppDelegateSwizzler (7.10.0):
- GoogleUtilities/Environment
- GoogleUtilities/Logger
- GoogleUtilities/Network
- GoogleUtilities/Environment (7.10.0):
- PromisesObjC (< 3.0, >= 1.2)
- GoogleUtilities/ISASwizzler (7.10.0)
- GoogleUtilities/Logger (7.10.0):
- GoogleUtilities/Environment
- GoogleUtilities/MethodSwizzler (7.10.0):
- GoogleUtilities/Logger
- GoogleUtilities/Network (7.10.0):
- GoogleUtilities/Logger
- “GoogleUtilities/NSData+zlib”
- GoogleUtilities/Reachability
- “GoogleUtilities/NSData+zlib (7.10.0)”
- GoogleUtilities/Reachability (7.10.0):
- GoogleUtilities/Logger
- GoogleUtilities/UserDefaults (7.10.0):
- GoogleUtilities/Logger
Firebase: 5f8193dff4b5b7c5d5ef72ae54bb76c08e2b841d FirebaseABTesting: 10cbce8db9985ae2e3847ea44e9947dd18f94e10 FirebaseAnalytics: 7761cbadb00a717d8d0939363eb46041526474fa FirebaseCore: 5743c5785c074a794d35f2fff7ecc254a91e08b1 FirebaseCoreDiagnostics: 92e07a649aeb66352b319d43bdd2ee3942af84cb FirebaseCrashlytics: feb07e4e9187be3c23c6a846cce4824e5ce2dd0b FirebaseInstallations: 40bd9054049b2eae9a2c38ef1c3dd213df3605cd FirebasePerformance: 66eb58c3e3568a0501a9be271c8ff424dea0ff34 FirebaseRemoteConfig: 2d6e2cfdb49af79535c8af8a80a4a5009038ec2b
GoogleAppMeasurement: 4c19f031220c72464d460c9daa1fb5d1acce958e GoogleDataTransport: 1c8145da7117bd68bbbed00cf304edb6a24de00f GoogleMaps: eb03e327edfd70b06de1e6e321653f73712df7ad GoogleSignIn: 7137d297ddc022a7e0aa4619c86d72c909fa7213 GoogleUtilities: bad72cb363809015b1f7f19beb1f1cd23c589f95
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 18 (7 by maintainers)
Hi @jotai-coder, sorry for the delay here. Nothing firm yet but I did do a deep dive to better understand things. I’m going to follow-up early next week with any next steps I indentify.
@ncooke3 These are not traditional crash reports. These are
Energy Reports
from the AppStore / Xcode Organizer feature.They correspond to High CPU Usage terminations (watchdogs as we used to call 'em), but the original crash report is not included. Instead, they provide a series of JSON files with the stack information.
@ncooke3 we call it in applicationDidFinishLaunching
@ncooke3: Another tidbit: we have both background and foreground watchdogs for this, so my previous comment about background launches may not be significant.
If there’s any more data we can provide, perhaps something from Firebase, please let us know.
Thanks – I took a look at that issue before posting, and after taking a quick peek into the Uploader, some wild guesses I could come up with:
1- Something causes a large number of events to pile up – could this happen if users use something to block certain domains? 2- The OS doesn’t like the tight loop iterating through the directory contents 2- The async op somehow is allowing multiple concurrent batches to be processed 3- The uploader_queue priority may need to be lower
Again, just wild guesses.
Good to know, thanks. I’ll take a look into trying to reproduce. There may be lifecycle related lock on filesystem access that is causing that API (or an underlying one it calls into) to spin its wheels and force a timeout.
We have crash logs from anything from 15.5 to 16.3 with about even distribution.