firebase-android-sdk: Cloud Firestore crashes with android.database.CursorWindowAllocationException

I’m seeing this happen on my react-native project using react-native-firebase. It seems to happen primarily on Samsung devices (94% of 50 occurrences). None of the devices with this crash had full ram, they all had 300+mb free and were at or below 85% usage. I don’t have r8 enabled. I can’t reliably reproduce this, it only seems to happen in production. This is a tough one!

We stumbled across the exact same problem in our native Android app. Our device is a Galaxy Grand Prime Pro, running Android 7.1.

RAM free: 433.46 MB
Disk free: 770.34 MB 

Our logs may provide an indication of the cause though. Shortly before the android.database.CursorWindowAllocationException occurred, another exception was logged in our app:

java.io.FileNotFoundException: <SOME_FILE> (Too many open files) at 
java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init
(FileOutputStream.java:221) at java.io.FileOutputStream.<init>(FileOutputStream.java:169) at
....

Maybe Firebase can’t open it’s db file because of an Too many open files issue…

@creativecreatorormaybenot Very interesting! I hate to say this, but it’s starting to seem possible that Firestore isn’t really to blame for the crash. It may just be the symptom that shows up first when memory usage gets too high (e.g. once the JVM allocates too much memory, then SQLite’s native memory allocations start failing causing the Firestore crash)… It might be useful for you to play around with Android Studio’s memory profiler tools (https://developer.android.com/studio/profile/memory-profiler). Perhaps Flutter Isolate or Firebase Storage is leaking memory or using an inordinate amount for some reason? If you can optimize the memory usage somehow, perhaps the Firestore crashes will go away.

@chokokatana Thank you for the additional report. Unfortunately, the root cause of this crash has eluded us and we are unsure how to resolve it. If you are somehow able to consistently reproduce the issue or find a pattern that may provide additional context then that would be very helpful.

For quick reference, the latest update from the Firestore team on this issue is in this comment: https://github.com/firebase/firebase-android-sdk/issues/629#issuecomment-710126713.

@creativecreatorormaybenot I’m using @react-native-firebase/firestore": "^7.7.0

Check this out!

@creativecreatorormaybenot Thanks for the added info. I’ll check with the team to see if we think there’s something we could put into a debug APK that would help us. In the meantime, would it be possible for you to (as a test) remove the memory intensive stuff you are doing and see if that helps for this user? That could be a big clue.

Just to update the state of this on our end, @var-const actually spent a bunch more time trying to reproduce this using an Android emulator with 128MB of RAM (the lowest amount Android Studio would allow) and API 23 (which is the API level on one of the two devices reported to have this error). He tried various operations:

• writing many batches of ~1MB documents;
• reading a collection with a huge number of small documents;
• reading a lot of documents with small fields that add up to ~1MB per document;
• reading a lot of documents with a single string field that is ~1MB. But unfortunately was unable to reproduce the issue.

So we’re basically stuck unless somebody can provide us a working repro. Sorry.

The issue with these stacks is that they’re saying that essentially anything in the remote document cache failed while performing nearly any operation in the local store.

  at com.google.firebase.firestore.local.SQLiteRemoteDocumentCache.add (SQLiteRemoteDocumentCache.java)
  or                     .decodeMaybeDocument (SQLiteRemoteDocumentCache.java)
  or                     .get (SQLiteRemoteDocumentCache.java)
  or                     .getAll (SQLiteRemoteDocumentCache.java)
  or                     .getAllDocumentsMatchingQuery (SQLiteRemoteDocumentCache.java)
  or                     .lambda$get$0 (SQLiteRemoteDocumentCache.java)
  or                     .lambda$getAll$1 (SQLiteRemoteDocumentCache.java)
  or                     .lambda$getAllDocumentsMatchingQuery$2 (SQLiteRemoteDocumentCache.java)
  at com.google.firebase.firestore.local.LocalStore.acknowledgeBatch (LocalStore.java)
  or                     .allocateQuery (LocalStore.java)
  or                     .applyRemoteEvent (LocalStore.java)
  or                     .collectGarbage (LocalStore.java)
  or                     .getLastRemoteSnapshotVersion (LocalStore.java)
  or                     .getNextMutationBatch (LocalStore.java)
  or                     .handleUserChange (LocalStore.java)
  or                     .lambda$acknowledgeBatch$2 (LocalStore.java)
  or                     .lambda$allocateQuery$7 (LocalStore.java)
  or                     .lambda$applyRemoteEvent$5 (LocalStore.java)
  or                     .lambda$collectGarbage$9 (LocalStore.java)
  or                     .lambda$notifyLocalViewChanges$6 (LocalStore.java)
  or                     .lambda$rejectBatch$3 (LocalStore.java)
  or                     .lambda$releaseQuery$8 (LocalStore.java)
  or                     .lambda$setLastStreamToken$4 (LocalStore.java)
  or                     .lambda$startMutationQueue$0 (LocalStore.java)
  or                     .lambda$writeLocally$1 (LocalStore.java)
  or                     .notifyLocalViewChanges (LocalStore.java)
  or                     .readDocument (LocalStore.java)
  or                     .setLastStreamToken (LocalStore.java)
  or                     .shouldPersistQueryData (LocalStore.java)

This makes it hard to construct a thesis for the failure. At this point the best lead is try to reproduce the many query.gets very quickly approach. I’ll report back when I have something working.

We’ve had other, complete stack traces and it hasn’t really helped, so I wouldn’t go too far out of my way to try to get one. Then again, another stack could help confirm that this is indeed the same issue as #382 or something else.