firebase-ios-sdk: -[GDTCORFlatFileStorage batchWithEventSelector:batchExpiration:onComplete:] causes High CPU Usage Terminations

Description

We’ve been tracking an issue via Xcode Organizer for a bit where GoogleDataTransport is billed for the majority of our app’s high CPU terminations.

We’ve been unable to reproduce this in-house, so all we have to go on are the Xcode organizer Energy reports, and some specific user complaints. Our own app logs don’t show anything interesting either. I am not able to include the entire xcprocessorusagelog as it contains IP from my employer.

There are two distinct stacks, but they both generate this type of watchdog termination: 48 seconds cpu time over 56 seconds (86% cpu average), exceeding limit of 80% cpu over 60 seconds High CPU Usage Termination

From the threads that are not related to GoogleDataTransport, we can tell this is happening very soon after applicationDidFinishLaunching

Sampled Backtrace (0)#0	(null) in __rename ()
#1	(null) in -[NSFilesystemItemMoveOperation main] ()
#2	(null) in __NSOPERATION_IS_INVOKING_MAIN__ ()
#3	(null) in -[NSOperation start] ()
#4	(null) in -[NSFileManager moveItemAtPath:toPath:error:] ()
#5	0xb760a68 in __75-[GDTCORFlatFileStorage batchWithEventSelector:batchExpiration:onComplete:]_block_invoke_2 at /Users/leonjuan/sources/connect-mobile-ios/Pods/GoogleDataTransport/GoogleDataTransport/GDTCORLibrary/GDTCORFlatFileStorage.m:226
#6	(null) in _dispatch_call_block_and_release ()
#7	(null) in _dispatch_client_callout ()
#8	(null) in _dispatch_lane_serial_drain ()
#9	(null) in _dispatch_lane_invoke ()
#10	(null) in _dispatch_workloop_worker_thread ()
#11	(null) in _pthread_wqthread ()
#12	(null) in start_wqthread ()

Video of stack With system libs: https://user-images.githubusercontent.com/115565643/228986078-42d3c516-b89a-45fc-9185-da120d0f8806.mov

Without system libs: Screen Shot 2023-03-30 at 5 58 37 PM

Similar stack: Screen Shot 2023-03-30 at 5 58 16 PM

Apologies for not being able to include the original xcprocessorusagelog files from Xcode Organizer, but they contain IP from my employer.

Reproducing the issue

Unable to repro

Firebase SDK Version

8.15

Xcode Version

14.2

Installation Method

CocoaPods

Firebase Product(s)

Analytics, Crashlytics, Performance, Remote Config

Targeted Platforms

iOS

Relevant Log Output

energyDetail:
"48 seconds cpu time over 58 seconds (83% cpu average), exceeding limit of 80% cpu over 60 seconds"

If using Swift Package Manager, the project’s Package.resolved

n/a

If using CocoaPods, the project’s Podfile.lock

Firebase (8.15.0):
- Firebase/Core (= 8.15.0)
Firebase/Analytics (8.15.0):
- Firebase/Core
Firebase/Core (8.15.0):
- Firebase/CoreOnly
- FirebaseAnalytics (~> 8.15.0)
Firebase/CoreOnly (8.15.0):
- FirebaseCore (= 8.15.0)
Firebase/Crashlytics (8.15.0):
- Firebase/CoreOnly
- FirebaseCrashlytics (~> 8.15.0)
Firebase/Performance (8.15.0):
- Firebase/CoreOnly
- FirebasePerformance (~> 8.15.0)
Firebase/RemoteConfig (8.15.0):
- Firebase/CoreOnly
- FirebaseRemoteConfig (~> 8.15.0)
FirebaseABTesting (8.15.0):
- FirebaseCore (~> 8.0)
FirebaseAnalytics (8.15.0):
- FirebaseAnalytics/AdIdSupport (= 8.15.0)
- FirebaseCore (~> 8.0)
- FirebaseInstallations (~> 8.0)
- GoogleUtilities/AppDelegateSwizzler (~> 7.7)
- GoogleUtilities/MethodSwizzler (~> 7.7)
- GoogleUtilities/Network (~> 7.7)
- “GoogleUtilities/NSData+zlib (~> 7.7)”
- nanopb (~> 2.30908.0)
FirebaseAnalytics/AdIdSupport (8.15.0):
- FirebaseCore (~> 8.0)
- FirebaseInstallations (~> 8.0)
- GoogleAppMeasurement (= 8.15.0)
- GoogleUtilities/AppDelegateSwizzler (~> 7.7)
- GoogleUtilities/MethodSwizzler (~> 7.7)
- GoogleUtilities/Network (~> 7.7)
- “GoogleUtilities/NSData+zlib (~> 7.7)”
- nanopb (~> 2.30908.0)
FirebaseCore (8.15.0):
- FirebaseCoreDiagnostics (~> 8.0)
- GoogleUtilities/Environment (~> 7.7)
- GoogleUtilities/Logger (~> 7.7)
FirebaseCoreDiagnostics (8.15.0):
- GoogleDataTransport (~> 9.1)
- GoogleUtilities/Environment (~> 7.7)
- GoogleUtilities/Logger (~> 7.7)
- nanopb (~> 2.30908.0)
FirebaseCrashlytics (8.15.0):
- FirebaseCore (~> 8.0)
- FirebaseInstallations (~> 8.0)
- GoogleDataTransport (~> 9.1)
- GoogleUtilities/Environment (~> 7.7)
- nanopb (~> 2.30908.0)
- PromisesObjC (< 3.0, >= 1.2)
FirebaseInstallations (8.15.0):
- FirebaseCore (~> 8.0)
- GoogleUtilities/Environment (~> 7.7)
- GoogleUtilities/UserDefaults (~> 7.7)
- PromisesObjC (< 3.0, >= 1.2)
FirebasePerformance (8.15.0):
- FirebaseCore (~> 8.0)
- FirebaseInstallations (~> 8.0)
- FirebaseRemoteConfig (~> 8.0)
- GoogleDataTransport (~> 9.1)
- GoogleUtilities/Environment (~> 7.7)
- GoogleUtilities/ISASwizzler (~> 7.7)
- GoogleUtilities/MethodSwizzler (~> 7.7)
- nanopb (~> 2.30908.0)
FirebaseRemoteConfig (8.15.0):
- FirebaseABTesting (~> 8.0)
- FirebaseCore (~> 8.0)
- FirebaseInstallations (~> 8.0)
- GoogleUtilities/Environment (~> 7.7)
- “GoogleUtilities/NSData+zlib (~> 7.7)”

GoogleAppMeasurement (8.15.0):
- GoogleAppMeasurement/AdIdSupport (= 8.15.0)
- GoogleUtilities/AppDelegateSwizzler (~> 7.7)
- GoogleUtilities/MethodSwizzler (~> 7.7)
- GoogleUtilities/Network (~> 7.7)
- “GoogleUtilities/NSData+zlib (~> 7.7)”
- nanopb (~> 2.30908.0)
GoogleAppMeasurement/AdIdSupport (8.15.0):
- GoogleAppMeasurement/WithoutAdIdSupport (= 8.15.0)
- GoogleUtilities/AppDelegateSwizzler (~> 7.7)
- GoogleUtilities/MethodSwizzler (~> 7.7)
- GoogleUtilities/Network (~> 7.7)
- “GoogleUtilities/NSData+zlib (~> 7.7)”
- nanopb (~> 2.30908.0)
GoogleAppMeasurement/WithoutAdIdSupport (8.15.0):
- GoogleUtilities/AppDelegateSwizzler (~> 7.7)
- GoogleUtilities/MethodSwizzler (~> 7.7)
- GoogleUtilities/Network (~> 7.7)
- “GoogleUtilities/NSData+zlib (~> 7.7)”
- nanopb (~> 2.30908.0)
GoogleDataTransport (9.2.0):
- GoogleUtilities/Environment (~> 7.7)
- nanopb (< 2.30910.0, >= 2.30908.0)
- PromisesObjC (< 3.0, >= 1.2)

GoogleSignIn (5.0.2):
- AppAuth (~> 1.2)
- GTMAppAuth (~> 1.0)
- GTMSessionFetcher/Core (~> 1.1)
GoogleUtilities/AppDelegateSwizzler (7.10.0):
- GoogleUtilities/Environment
- GoogleUtilities/Logger
- GoogleUtilities/Network
GoogleUtilities/Environment (7.10.0):
- PromisesObjC (< 3.0, >= 1.2)
GoogleUtilities/ISASwizzler (7.10.0)
GoogleUtilities/Logger (7.10.0):
- GoogleUtilities/Environment
GoogleUtilities/MethodSwizzler (7.10.0):
- GoogleUtilities/Logger
GoogleUtilities/Network (7.10.0):
- GoogleUtilities/Logger
- “GoogleUtilities/NSData+zlib”
- GoogleUtilities/Reachability
“GoogleUtilities/NSData+zlib (7.10.0)”
GoogleUtilities/Reachability (7.10.0):
- GoogleUtilities/Logger
GoogleUtilities/UserDefaults (7.10.0):
- GoogleUtilities/Logger

Firebase: 5f8193dff4b5b7c5d5ef72ae54bb76c08e2b841d FirebaseABTesting: 10cbce8db9985ae2e3847ea44e9947dd18f94e10 FirebaseAnalytics: 7761cbadb00a717d8d0939363eb46041526474fa FirebaseCore: 5743c5785c074a794d35f2fff7ecc254a91e08b1 FirebaseCoreDiagnostics: 92e07a649aeb66352b319d43bdd2ee3942af84cb FirebaseCrashlytics: feb07e4e9187be3c23c6a846cce4824e5ce2dd0b FirebaseInstallations: 40bd9054049b2eae9a2c38ef1c3dd213df3605cd FirebasePerformance: 66eb58c3e3568a0501a9be271c8ff424dea0ff34 FirebaseRemoteConfig: 2d6e2cfdb49af79535c8af8a80a4a5009038ec2b

GoogleAppMeasurement: 4c19f031220c72464d460c9daa1fb5d1acce958e GoogleDataTransport: 1c8145da7117bd68bbbed00cf304edb6a24de00f GoogleMaps: eb03e327edfd70b06de1e6e321653f73712df7ad GoogleSignIn: 7137d297ddc022a7e0aa4619c86d72c909fa7213 GoogleUtilities: bad72cb363809015b1f7f19beb1f1cd23c589f95

About this issue

Original URL
State: open
Created a year ago
Comments: 18 (7 by maintainers)

Most upvoted comments

Hi @jotai-coder, sorry for the delay here. Nothing firm yet but I did do a deep dive to better understand things. I’m going to follow-up early next week with any next steps I indentify.

ncooke3 on May 4, 2023

@ncooke3 These are not traditional crash reports. These are Energy Reports from the AppStore / Xcode Organizer feature.

They correspond to High CPU Usage terminations (watchdogs as we used to call 'em), but the original crash report is not included. Instead, they provide a series of JSON files with the stack information.

{
  "sourceFileLineNumber" : 226,
  "userData" : {
    "name" : "GoogleDataTransport: __75-[GDTCORFlatFileStorage batchWithEventSelector:batchExpiration:onComplete:]_block_invoke_2 + 492",
    "resolved" : false,
    "comments" : ""
  },
  "processorUsagePointName" : "GoogleDataTransport: __75-[GDTCORFlatFileStorage batchWithEventSelector:batchExpiration:onComplete:]_block_invoke_2 + 492",
  "bugType" : "206",
  "identifier" : {
    "build" : "1",
    "isBeta" : false,
    "rootBuild" : "0",
    "bundleIdentifier" : "com.redacted",
    "rootVersion" : "0",
    "version" : "0",
    "adamIdentifier" : "redacted",
    "platformSDKIdentifier" : "com.apple.platform.iphoneos",
    "processorUsagePointIdentifier" : "ByiYqFtlOUkliEfH8WqbDO"
  },
  "sourceFileName" : "GDTCORFlatFileStorage.m"
}

{
    "osVersion": "15.5",
    "energyDetail": "48 seconds cpu time over 58 seconds (83% cpu average), exceeding limit of 80% cpu over 60 seconds",
    "userActivity": "49 samples Idle, 0 samples Active",
    "sampledExecutionTree":
    [
        [
            {
                "stackFrame": ""
            }
        ]
    ]
}

jotai-coder on Apr 14, 2023

@ncooke3 we call it in applicationDidFinishLaunching

jotai-coder on Apr 8, 2023

@ncooke3: Another tidbit: we have both background and foreground watchdogs for this, so my previous comment about background launches may not be significant.

If there’s any more data we can provide, perhaps something from Firebase, please let us know.

jotai-coder on Mar 31, 2023

Thanks – I took a look at that issue before posting, and after taking a quick peek into the Uploader, some wild guesses I could come up with:

1- Something causes a large number of events to pile up – could this happen if users use something to block certain domains? 2- The OS doesn’t like the tight loop iterating through the directory contents 2- The async op somehow is allowing multiple concurrent batches to be processed 3- The uploader_queue priority may need to be lower

Again, just wild guesses.

jotai-coder on Mar 30, 2023

Good to know, thanks. I’ll take a look into trying to reproduce. There may be lifecycle related lock on filesystem access that is causing that API (or an underlying one it calls into) to spin its wheels and force a timeout.

ncooke3 on Mar 30, 2023

Thanks @jotai-coder and sorry for the trouble. Do you have an idea if this is affecting clients running a particular iOS version?

We have crash logs from anything from 15.5 to 16.3 with about even distribution.

jotai-coder on Mar 30, 2023