amplify-js: DataStore - DeltaSync Very Slow on SQLite

Before opening, please confirm:

• I have searched for duplicate or closed issues and discussions.
• I have read the guide for submitting bug reports.
• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework

React Native

Amplify APIs

DataStore

Amplify Categories

api

Environment information

# Put output below this line
npx: installed 1 in 1.481s

  System:
    OS: Linux 4.14 Amazon Linux 2
    CPU: (2) x64 Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
    Memory: 6.75 GB / 7.79 GB
    Container: Yes
    Shell: 4.2.46 - /bin/bash
  Binaries:
    Node: 10.23.2 - ~/.nvm/versions/node/v10.23.2/bin/node
    npm: 6.14.10 - ~/.nvm/versions/node/v10.23.2/bin/npm
  npmGlobalPackages:
    @aws-amplify/cli: 5.2.0
    cdk: 1.87.1
    coffeescript: 2.5.1
    esformatter: 0.11.3
    js-beautify: 1.13.5
    npm: 6.14.10
    prettier: 2.2.1
    typescript: 3.7.5

Describe the bug

I am creating this issue as a separate GitHub issue - separate from #8405

https://github.com/aws-amplify/amplify-js/issues/8405#issuecomment-891634004

Hi @iartemiev - want to push further into the slowness we are seeing for DeltaSync

DataStore creates a separate table in DynamoDB to manage the DeltaSync called “AmplifyDataStore-ENV”.

There are no indexes in this table - just the partition key and sort key - where the partition key is table/date and sort key is time/id/version.

Our database is multi-tenanted - and we deal with assets in industrial plants. So I could have hundreds of other users in other plants making massive amounts of changes. But I might not have any users working my plant. The DeltaSync as I see it is going to have to sort through everyone else’s changes just to work out there are zero changes that are going to be applicable to me.

When we initially hydrate the cache we do a base query which uses GSI’s to get a quick response.

At the moment I’m seeing DeltaSync take about the same amount of time as the full sync (20 minutes). Today I know I was doing a lot of bulk updating of data in a few different plants through our web back end. Not an unusually large workload. But I am concerned from the slowness I’m seeing in our database and what I see in the DeltaSync table for DynamoDB has me worried that this is not scalable for a multi-tenanted environment.

You’ve been really good at coming up with solutions to get us moving - hopefully someone else has already come up with a solution to make the DeltaSync run in seconds rather than 20+ minutes.

https://github.com/aws-amplify/amplify-js/issues/8405#issuecomment-891797469

@sacrampton, I think this behavior likely warrants a separate GitHub issue, unless this is somehow related to the on-device database on React Native specifically (AsyncStorage or SQLite).

To better understand what’s going on, I have some follow up questions:

How many total records are in the delta sync table in DynamoDB at the time that you’re seeing the 20 min delta sync time? How many of those records are being synced down to the app? Are you using DataStore.configure to change any of the sync-related settings (e.g., syncPageSize, fullSyncInterval, etc.)? If so, which settings are you using? Are you seeing roughly the same delta sync performance if you test this in a web app?

Expected behavior

DeltaSync processes very quickly - seconds, not minutes

Reproduction steps

DataStore.start

Code Snippet

// Put your code below this line.

Log output

// Put your logs below this line

aws-exports.js

No response

Manual configuration

No response

Additional configuration

No response

Mobile Device

No response

Mobile Operating System

No response

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response