expo: [Android] FileSystem.readAsStringAsync for large (JSON) file causes Out of Memory Exception

Summary

Core Issue

When loading relatively large data for a given android system using FileSystem.readAsStringAsync, an OOM Exception is triggered in the following format:

Encountered an exception while calling native method: Exception occurred while executing exported method readAsStringAsync on module ExponentFileSystem: Failed to allocate a XXX byte allocation with 25165824 free bytes and XXMB until OOM, target footprint XX, growth limit XX!

Low end devices experience this error with files as small as 30Mb, whereas flagship models released since 2020 give this error around 75Mb large files and up. Android Emulators with an amped Heap size of 1Gb give this error starting from 150Mb, which is still relatively small concerning the use case of my app.

The exception shows up on android on the following builds:

Expo Go (SDK 45-47)
EAS builds with no special flags
EAS builds with android:largeHeap=true and android:hardwareAccelerated=false enabled, which is usually recommended on StackOverflow when encountering this error in react-native.

Context

I’m developing an offline-first application for some 2 years now using Expo, which is technically concerned with storing rich content using Expo’s FileSystem. The Filesystem is used in two key parts relevant to this critical issue:

persisting data after app restarts, using redux-persist
Importing backup data for offline backups

I found that no one is experiencing a similar error on the internet, whereas the use-case seems to be very relevant in an offline-first scenario.

Is there a better way to load CSVs, JSONs, or other datasets for offline backups, or on-device analysis of data, in React-Native? Any clues, tips, or insights will be helpful.

Please avoid stating your preference for offloading personal user data to servers, and providing them just-in-time when users need them.

What platform(s) does this occur on?

Android

Environment

expo-env-info 1.0.5 environment info: System: OS: macOS 12.6 Shell: 3.2.57 - /bin/bash Binaries: Node: 14.20.1 - ~/.nvm/versions/node/v14.20.1/bin/node Yarn: 1.17.3 - /usr/local/bin/yarn npm: 6.14.17 - ~/.nvm/versions/node/v14.20.1/bin/npm Watchman: 2022.10.17.00 - /usr/local/bin/watchman Managers: CocoaPods: 1.9.1 - /usr/local/bin/pod SDKs: iOS SDK: Platforms: DriverKit 22.1, iOS 16.1, macOS 13.0, tvOS 16.1, watchOS 9.1 IDEs: Android Studio: 2021.3 AI-213.7172.25.2113.9123335 Xcode: 14.1/14B47b - /usr/bin/xcodebuild npmPackages: expo: ~47.0.8 => 47.0.8 react: 18.1.0 => 18.1.0 react-native: 0.70.5 => 0.70.5 npmGlobalPackages: eas-cli: 2.8.0 Expo Workflow: managed

Minimal reproducible example

First init a fresh expo project (SDK45-47, your pick) using expo init > Blank Project.

Then install relevant libraries using expo install expo-file-system expo-document-picker.

In App.js add the following code:

import { StatusBar } from 'expo-status-bar';
import { StyleSheet, Text, View, Pressable } from 'react-native';
import * as FileSystem from 'expo-file-system';
import * as DocumentPicker from 'expo-document-picker';

export default function App() {
  const load_file = async () => {
    try {
      data_obj = await DocumentPicker.getDocumentAsync({copyToCacheDirectory:true, type: '*/*'});
      const data_string = await FileSystem.readAsStringAsync(data_obj.uri, {encoding: FileSystem.EncodingType.UTF8});
      console.log(`Loaded ${data_string.length} bytes`);
    } catch (error) {
      console.log(`Big error in importing: ${error}!`);
    }
  }

  return (
    <View style={styles.container}>
      <Text>Press button and load (large) json backup</Text>
      <Pressable onPress={load_file} style={styles.button}/>
      <StatusBar style="auto" />
    </View>
  );
}

const styles = StyleSheet.create({
  container: { flex: 1, backgroundColor: '#fff', alignItems: 'center', justifyContent: 'center'},
  button: { width: 50, height:50, backgroundColor:'black' }
});

Run app in Expo Go, load a big JSON file after button click, might or might not cause an OOM error.

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 1
Comments: 24 (4 by maintainers)

Most upvoted comments

@JohnyClash I revised the Expo Plugin, which you can now find in this gist. It correctly enables the android:largeHeap flag in AndroidManifest.

I’ve performed a few tests and have encountered no OOM errors thus far (up to 80 Mb files), which is fantastic news. I’ll be testing more next week with heavier files. Please also try in your own use case!

@stefmedjo I don’t know what you mean by that. Could you elaborate?

adamblvck on Mar 2, 2023

@adamblvck I am having the exact same issues as you as it relates to Out of Memory Exceptions. I am attempting to upload an mp4 directly to an s3 bucket. At which point everything works until the file size goes above 190mb then let data = fetch(localUri) fails with “Network Request Failed”, this was odd since I simply wanted to blob the read Stream fetch returns, this is all done locally so ‘Network Request Failed’ is not applicable . When trying to figure out a work around to this I stumbled across the same error you have with FileSystem.readAsStringAsync() on the exact same files that fail to fetch due to size. Leading me to believe fetch fails due to this heap size limit but does not return helpful data on this error. XML also fails in the same way.

Ill start with something I discovered that may help you. In the FileSystem docs it states that {position: x, length:x} only work with FileSystem.EncodingType.Base64. Above another user asked if you tried base64 encoding and you said it made no difference. However, when I tested

base64 = await FileSystem.readAsStringAsync(file, {encoding:FileSystem.EncodingType.Base64, position :0 , length: 10000000});

it does in fact work on files upto 350 mb(max I tested). However, the same file and code with UTF8 it fails.

I noticed in your above response you typed. FileSystem.EncodingType.BASE64 when it is in fact FileSystem.EncodingType.Base64. When I type it in the way you may have BASE64 it does not throw an error however it does still encode it as utf8, and continues to the throw the memory exception error.

I am making a internal distribution app that is running on an s8 ultra with 8gb of ram. I find it extremely surprising that I cannot allocate more of that ram then 205mb, when I followed the tutorial for plugins you linked and added a plugin that changed largeHeap to true. I got the exact same Failed to allocate error at the exact same memory level. How does one test what one’s memoryHeap limit actually is and how does one confirm that "largeHeap": true is actually set?

ExponentFileSystem: Failed to allocate a 537133064 byte allocation with 25165824 free bytes and 205MB until OOM, target footprint 346481488, growth limit 536870912]

It seems like maybe the tutorial was incorrect and one may have to eject from a managed expo workflow. I found many cases on stack overflow in which largeHeap:true worked successfully on react native applications, but your link was the only description of how to do it in a managed workflow.

JohnyClash on Feb 20, 2023

@djaffer hey, thank you for pointing to the plugin and post. I believe I’ve mentioned it in this issue too, and sadly it didn’t set the largeHeap flag correctly (after configuring app config) when built through EAS build.

The gist which I provided a few comments back contains a plugin which correctly sets the largeHeap flag to true. I’ve verified it using ClassyShark, and my app can also sustain more RAM now.

I’ll be testing how far I can push it before hitting OOM.

adamblvck on Mar 3, 2023

this was automatically closed when we ran an issue validation script across all open issues that had the “needs review” label - the reason it was closed is there wasn’t a link to a reproducible example. it’d make this more quickly actionable for someone on the expo side if there was a clonable repro. i’ll reopen nonetheless

brentvatne on Feb 27, 2023

@PhilemonBel the largeHeap flag isn’t set in the ExpoGo app. You can set android:largeHeap=true (or any other flag) via EAS-builds and plugins.

Expo developers have provided a helpful guide here, which involves creating a plugin file, then mentioning your desired flags in app.js:

{
  "expo": {
[...]
    "plugins": [
      [
        "./plugins/withAndroidMainActivityAttributes",
        {
          "android:largeHeap": true,
          "android:hardwareAccelerated": true
        }
      ]
    ]
  }
}

The bad news is that I already tried this option, and I build my apps with forum-recommended flags enabled on both simulator and 3+ Real Android devices. The result is that OOM memories are still being played out.

The underlying issue might be how react-native is being build into android code. On forums I’ve seen native android developers (java) being able to load ridiculous amounts of data into working memory, yet react-native developers pump against JVMs limitation even with largeHeap=true, which might be linked to how JSC or Hermes is handling or paging large memory operations to metal.

I’ve also experienced a similar issue on expo web builds whilst loading 60Mb+ files using the above minimal code, the web app just CRASHES with no error. There too, I suspect, memory handling by the javascript engine misses the boat.

I’m currently reworking the whole data management architecture of my application to comply with this ridiculous 25Mbyte Heap memory size constraint, but it still remains an issue and a barking problem which I suspect is to be addressed at the engine level.

adamblvck on Dec 29, 2022

Hi, I’m having almost exactly the same problem when I’m writing a backup file from my app using writeAsStringAsync and I get the Out Of Memory error on Expo Go on android (both on simulator and real device) when I try to write a file of about 30MB or bigger. I have two questions about the largeHeap flag:

Does anybody know if it’s set to true in the ExpoGo app?
How to edit the android manifest to add android:largeHeap=true in an expo managed workflow?

PhilemonBel on Dec 13, 2022