amplify-js: Storage.put slows application - require background disk based solution

Hi, this is promising, will your solution include support for continuing long uploads when an app is backgrounded? At the moment they seem to be immediately suspended by the OS, but there are underlying APIs that would allow for this on iOS/Android. If not, would you consider providing and implementation of Storage.put that simply returns the pre-signed PUT URL (in a similar fashion that we can get a GET URL from the SDK? If we could access the pre-signed PUT URL for a given S3 key, we could leverage another background-friendly solution to do the actual uploading

Hi @sacrampton , I wanted to reach out and let you know that we are actively working on providing a solution for this.

Hello everyone, we are still looking into this, we will provide an update when we have clarity on our next steps.

Any updates? Currently blocking.

@sacrampton , Thanks for your patience. Below is the summary of my finding so far.

1. fetch straight out does not work for android(for me) when uploading large files. Uploading smaller files(few MBs) works fine. However, when I try to upload a file 200mb+ it throws “Network request failed”. There is a very detailed issue on this. Which I need to further explore. https://github.com/facebook/react-native/issues/28551#

2. When using rn-fetch-blob package , uploading larger files still works but larger files(500 mb - 2 gb) will still hang the UI for around 500ms - 3 seconds. Additionaly, the UI main thread is also frozen when retrieving/verifying/updating credential during each consecutive data block upload.

3. While I couldn’t verify if the high memory usage is caused by fetch when reading larger files (because it throws network error), The main UI thread still hangs when Storage.put is uploading file chunks.

So, I believe there are 2 aspects that we need to work on.

1. While rn-fetch-blob package works fine(with small UI thread hang ups) with reading the large files, we need to find out alternative solution which helps to solve the file reading issue(probably requires us to implement usage of native apis). Our goal would be to only keep chunks of the file in memory while it is uploading and sequentially remove them from memory once they are uploaded.

2. Further explore and solve the issue why the UI thread hangs when retrieving/verifying/updating credential.

I will continue to work on the issue and engage with team to provide a solution for this issue.

+1, looking forward to any solution

Hi @chintannp - just checking back to see if there is any further progress on this issue you can share?

+1

Has anyone successfully used aws-amplify with react-native-background-upload? It seems like others have moved away from aws-amplify altogether for apps where background uploading is a business req

+1

Hi @paulsizer - we have a POC underway right now that we are testing out. We will provide more update when we have clarity on timelines!

Any updates on this? We have users asking us for updates on when bugs will be fixed.

Hi @chintannp - glad to see you are making progress. Thanks for your efforts.

Fetch works fine on iOS providing device has some good power behind it, super slow for older devices. For Android out of memory will throw.

I wasn’t able to ever get upload working for Android using storage.put for videos, even if not so big in size. Tried a couple of times over the years.

The only way we could do so was by:

1. Creating presigned s3 url
2. Use react-native-background-upload to upload to the presigned url in the background - without multipart as haven’t really been able to get that working (iOS upload speeds are slower unless in production version of app)

Note: this seems to upload slow in a development build, but once uploaded to TestFlight the upload speed is much faster.

Sample code:

Generate presigned url Lambda

   const params = {
          Bucket: process.env.STORAGE_XXXXX_BUCKETNAME,
          Key: input.key,
          ContentType: input.contentType,
          Expires: 900,
        };

        return s3.getSignedUrl('putObject', params).promise()

React native background upload

 const url = await generatePreSignedURL(key, contentType); // mutation that calls s3 presigned url lambda

    const options = {
      url,
      path,
      method: "PUT",
      type: "raw", // or raw
      maxRetries: 5, // set retry count (Android only). Default 2
      headers: {
        "Content-Type": contentType, // server requires a content-type header
        "Content-Length": `${fileInfo.size}`, // must be string - size is int
        bucket: awsmobile.aws_user_files_s3_bucket,
        region: awsmobile.aws_user_files_s3_bucket_region,
        key: `public/${key}`,
        // ACL: 'public-read'
      },
    };

    Upload.startUpload(options)
      .then((upId) => {
        setUploadId(upId);
        // event fired when upload progresses
        Upload.addListener("progress", upId, (data) => {
          let progress;
          const currProgress = data.progress / 100;
          console.log("currProgress: ", currProgress);
        });

        // event fired when upload error
        Upload.addListener("error", upId, (err) => {
          console.log("error with upload", err);
          if (err.error === "User cancelled upload") {
               // logic if user cancels upload
          } else {
            showErrorPage();
          }
        });

        // event when upload cancelled
        Upload.addListener("cancelled", upId, (data) => {
          console.log("upload cancelled", data);
        });

        // event when upload completes
        Upload.addListener("completed", upId, (data) => {
          const { responseCode, responseBody } = data;

          if (responseCode === 200) {
               // upload complete logic
          } else {
            console.log("upload response not okay", responseCode, data);
            // error
            showErrorPage();
          }
        });
      })
      .catch((err) => {
        console.log("Upload start error!", err);
        showErrorPage();
      });

Hi @sacrampton - We are still working on this, and have a couple of approaches that we are experimenting with. The most sustainable path has been eluding us for a while, but we are feeling confident that we are marching towards a solution that we can share soon. We will link a PR to this issue when our solution is ready!

@rahulnainwal107 thank you for confirming. We did suspect that the official docs will requires updates as well given our investigation thus far so it is helpful to have that verification.

Hi @cshfang @Ashish-Nanda @elorzafe - I checked with our developers and their answers are listed below…

To get file uri/path we are using

1. react-native-fs
2. react-native-blob-util

There are two ways we used storage.put in our application.

a) Using fetch - but we are not using this currently because we were facing some ANR crash due to it.

Storage.configure({
	AWSS3: {
		bucket: awsMobile.aws_user_files_s3_bucket, //Your bucket name;
		region: awsMobile.aws_user_files_s3_bucket_region, //Specify the region your bucket was created in;
	},
});

const response = await fetch(uri);
const blob = await response.blob(); // format the data for images
const customPrefix = {
	public: '',
	protected: '',
	private: '',
};
const responseAfterUpload = await Storage.put(fileName, blob, {
	customPrefix: customPrefix,
	contentType: 'multipart/form-data',
});

b) By converting images into base64 - This is what we are using before using react-native-background-upload.

Storage.configure({
	AWSS3: {
		bucket: awsMobile.aws_user_files_s3_bucket, //Your bucket name;
		region: awsMobile.aws_user_files_s3_bucket_region, //Specify the region your bucket was created in;
	},
});

const base64 = await fs.readFile(path ? path : uri, 'base64');
const arrayBuffer = decode(base64);

const customPrefix = {
	public: '',
	protected: '',
	private: '',
};