amplify-js: Simple chat app cannot be efficiently implemented using Datastore (ex. whatsapp-like app)

@iartemiev let’s forget everything from discord or above comments. it’s getting complicated.

Just provide a documentation or example app design (only Datastore related thing) for very basic whatsapp-like chat app with just 3 entities - User, Group and Post

(You don’t have to design it completely by yourself. We can discuss each point and design iteratively)

Criteria -

1. User will join by cognito userpool
2. User can create group (so they become admin of that group) or join other group
3. All Users of a group can post messages in group
4. Group admin can delete any one’s post. If any one else than admin deletes it even from local,it should not get deleted at backend.
5. there can be more than one admins. Existing admin can assign other members of group as admins.
6. Admin of one group might be normal user (non admin ) of other group.so he should be able to perform admin function (deleting anyone’s Post ) only in groups where he is admin.
7. Locally, user should have only data of groups he is in and only posts of these groups. Storing other users group or post is security-wise bad practice. User should not be able to query other user data by modifying request from front-end
8. Users can see profile of other users who share atleast one mutual group. Suppose User A and User B dont have any mutual group, they cannot see each other’s profile.
9. User record have a unique field (other than default id field) called as userId. This is basically the sub obtained from Cognito.
10. Users might belong to some groups and might not belong to others
11. Admin of group can delete/remove any user from group. Once user is removed from group, he should be able to send or receive new message to or from that group. So syncing should be near real-time.

Access patterns which will be frequently used :-

1. get single user record
2. get single group record
3. get groups belonging to a particular user
4. get posts belonging to a particular group
5. get single post record
6. get users belonging to a particular group

Design Schemas : - I have created two design schemas for the use case - Relational design and Single table design

Relational schema - User table : id - default PK userId - sub of Cognito - PK of GSI name

Group table : id - default PK - can use this as groupid groupname

GroupUsers table : id - default PK groupId - PK of GSI-A - SK of GSI-B userId - SK of GSI-A - PK of GSI-B

Post table : id - default PK - can use as postId groupId - id of group that the post is created in - PK of GSI createdAt - SK of GSI text - content of post

Single table design - id - default PK - let’s forget this we are keeping but not using it Also,we are using two generic keys - key1 and key2 - two GSI are created on them GSI-A and GSI-B

key1 - PK of GSI-A - SK of GSI-B key2 - PK of GSI-B - SK of GSI-A

For storing user record ; key1 - userId - cognito sub key2 - name (here sort key is not needed but we are storing just to fill value and avoid extra field for name)

For storing group record : key1 - groupId key2 - group name (here sort key is not needed but we are storing just to fill value and avoid extra field for name)

For storing group-users mapping - Remember Groups and Users have m-m relationship key1 - userId key2 - groupId

For storing group-post mapping - Remember groups and post have 1-m relationship key1 - groupId key2 - postId

We will prefix type name before value like User#1233, Group#4344

consider below data according to above mapping - 2 users, 1 group and 2 posts key1 | key2 user#1 | john group#1 | developers user#1 | group#1 group#1 | post#1 group#1 | post#2 user#2 | smith user#2 | group#1

Let’s see how access pattern can work on these 2 GSIs -

1. get single user record - GSI-A : PK as user#1
2. get single group record - GSI-A : PK as group#1
3. get groups belonging to a particular user - GSI-A : PK as user#1 and SK starting with group#
4. get posts belonging to a particular group - GSI-A : PK as group#1 and SK starting with post#
5. get single post record - GSI-B : PK as post#1
6. get users belonging to a particular group - GSI-B : PK as group#1 and SK starting with user#

@iartemiev , cc - @nubpro , @dabit3

Now please specify to satisfy the mentioned criteria - what syncExpression should be (if to use ) or what @ auth directive should be use to protect user data and sync only required data. also where should custom resolvers defined and when it will be enforced ( at syncing time ?)

Use any schema and see if we can implement this using datastore.

If we are able to solve this using Datastore, then it should be added to documentation instead of that very basic notes app.

There is one chat app example on internet but it is a single room or we can say single group chat where all users belong to same group . But in case of whatsapp users might belong to some groups and might not belong to others.

If this is solvable by Datastore, most real world app would be solved. If it fails then I think I have to opt out of Datastore and it means Datastore is still unable to handle such common use cases.

We can deisgn iteratively by discussing on discord .

The AppSync team recently posted an RFC detailing their proposal for Enhanced Subscription Filtering. Once this feature is released, DataStore will be able to take advantage of it to enable use cases such as this one.

@danrivett, @mir1198yusuf - a quick update on this: the AppSync team is actively working on a solution on their end that would (among other things) enable this use case. Once they have completed and released this feature, the Amplify client-side libraries will work on utilizing it on our end, specifically to add the functionality requested in this issue.

It’s prioritized on both the service and client side, but it is a complex feature that is still early in the process.

Makes sense, but if doesn’t facilitate that, e.g. user membership in some shared entity such as members of a project, then DataStore isn’t all that useful. The docs are also misleading, seems more promising than it actually is (as evidenced by a lot of other issues).

Simply put, if DataStore could facilitate these multi-tenancy designs, such as this thread example of a WhatsApp type app, or a project management app with privileged users across varying projects then it would be a great framework.

@iartemiev Can you confirm clearly on the following:

• DataStore cannot currently facilitate privileged multi-tenancy, real-time apps in general, correct?
• If and when is this planned to be supported?

@iartemiev - Any rough estimate when datastore would be able to fully support this use case ? So I can plan my project accordingly. Also I see multi-tenancy questions in many past issues.

@iartemiev with AppSync having added the required enhanced subscriptions support, we’re still looking to be able to share data using DataStore between a list of users for group messaging and other use cases such as team workflows where team members can have read/edit capabilities on shared data.

Is the required DataStore updates still on the roadmap, and any update here, as we’ve been waiting on implementing certain new functionality requiring this, and would really like to start on that, but currently blocked by DataStore limitations.

I agree with you . Even I felt that Datastore seems more promising than it actually is. The response from Ivan in discord was - they don’t have a public roadmap or estimate for this and Datastore cannot handle such cases for now.

If the owner array rule with if-found-in-array is implemented atleast, to some extent Datastore would be useful (though there are some other issues as well). Without this feature it is totally not suitable for real-world apps.

I myself have to shift to Appsync for now (without offline features) and wait for Datastore to be ready for real-world.

@mir1198yusuf I spent some time with this over the weekend and my conclusion is that unfortunately at the moment, we can’t utilize either of these approaches with DataStore.

The relational design won’t work because even though we can configure AppSync resolvers to fetch all the correct data for the base and delta sync queries by leveraging pipeline resolvers, we won’t be able to get the desired result out of the subscriptions that DataStore uses after the initial sync to keep the local store up to date. The reason for this is that request/response resolvers are only applied to AppSync subscriptions at connect time, there’s no way to apply them to each message, which precludes us from being able to use dynamic authorization rules as are required here (i.e., a changing list of authorized users). In other words, subscriptions would be sent either to all users (which is not acceptable due to privacy concerns) or only to the user creating the messages, which defeats the purpose of a messaging app.

As far as the single table design, without field-level authorization in DataStore this won’t be viable either, as there will be no mechanism to prevent a user from modifying certain fields (e.g., key1 and key2 in your example schema) to escalate their privileges and give themselves admin roles in other groups, or access messages they shouldn’t be authorized to read.

The team will work on identifying which features we’ll need to add to DataStore in the near future in order to enable this use case.

I’m wondering, are you deeply involve with the AppSync team? Maybe the best would be to try to solve certain real world use cases together. User based policies and group based polices with transitive permission are pretty common in real world applications.

If you can demonstrate some cases like:

• Team > Channels > Messages
• Forum > Sub Forum > Sub Forum > … > Post > Replies
• Organization > Folder > Document > Comment

In any of these use cases the final user can decide of the access at each node:

• You can be the owner of a Document
• Viewer of a Folder
• Member & Viewer of an Organization

Even more complex and could be allowed in some app

• Member of Organization restricted of some Folder
• Viewer on a Folder but restricted on some Document

And for all that you want to have in cache recent data (let’s say the last 2 month) and subscribe for any update on any DataType that you have access or may have access (when granted) to.

Any updates on this?

@chriszirkel it’s both, because AppSync itself doesn’t allow you to subscribe to updates if the owner field on the type is an array of permitted owners, only a single owner id. When using an owners array field, currently you either have to subscribe without specifying any owner value (and so receive updates for every item, even ones not permitted to read), or subscribe and specify an exact match of the whole owners field array.

What we want to be able to do is subscribe specifying the current user and AppSync returns updates for any record that includes the current user id in the owner field array. This requires a change to AppSync itself to support this. After that, DataStore then needs to be updated to make use of this change.

Hope that helps.

No problem, Dan! We’ll post an update here as soon as the AppSync feature in question gets released

For reference I also ran into similar problems with multi-owner access and wrote it up previously in #7989 as didn’t come across this issue when searching.

My ticket doesn’t contain anything in addition to what Ivan has mentioned here, but it’s heartening for me at least to see that the multi-owner use case seems to be quite a common requirement. Our upcoming roadmap over the next 3 months relies pretty heavily on it - for team-based workflows, messaging between individuals (which currently requires us to duplicate message records one for the author and one per recipient) etc.

Supporting team-based ownership/access rules is my number one request from DataStore, so would love to see it added.

Isn’t an AppSync subscription generally designed to only check auth at connection time, not on each message (and thus a single O(n) call relatively infrequently? Is this not supported by DataStore?

@iyz91, AppSync only evaluates auth for subscriptions at connect time, which makes using an array of owners for the auth rule (like what @mir1198yusuf was exploring as a potential solution) incompatible with AppSync subscriptions. In order to use an array of owners, you would need to be able to check the current user against the changing list of owners for each message.

In other words, AppSync subscriptions will not work as expected for the following schema:

type Model @model @auth(rules: [allow: owner, ownerField: "admins"]) 
{
  admins: [String]
}

What I was talking about with O(1) vs. O(n) was just comparing a single equality check vs iterating through an array, but that’s kind of beside the point. The crux of the issue is evaluating auth rules for each message.

Some of the other proposed solutions (like the one you referenced) may or may not offer appropriate authorization (that depends on the exact use case). But regardless, their implementation hinges on customizing GraphQL subscriptions. Currently, DataStore generates the subscriptions in the library when it starts and it’s not possible to override these/supply your own. So if the schema you want to implement requires you to specify your own subscriptions, it will not be possible to use such a schema with DataStore at the moment.

The changes we’ll need to make are quite complex and require a deep dive from the team. It will take some time to do this right and to enable these use cases. In the meantime, please keep sharing feedback with us about the kinds of applications you would like to build. We take all of this into account as we strive to improve DataStore for all of our customers.

@mir1198yusuf Thank you very much for that info and for following up on it yourself. They would save a lot of pain if they just stated this simply in their docs. I’ll let you know if I come across some intermediate solution or alternative for our similar use cases. Good luck with your work.

Yes, this would be an example of a multi-tenant app

Yes, this is an AppSync constraint. My guess is that it’s done out of a performance consideration. Given that AppSync can be used at extreme scale, iterating through an array for each subscription message is notably more resource-intensive than doing an equality check i.e., O(n) vs O(1).

To illustrate what I mean re: subscription limitations with an array of owners:

When using AppSync subscriptions, the selection set and any subscription arguments have to match the selection set and value for that subscription argument in a mutation in order to for the subscription to receive a message.

E.g., if you establish the subscription with

onUpdateMessage("bob") {
    id
    content
    _version
    _deleted
    _lastChangedAt
    createdAt
    updatedAt
}

Only a mutation that has the same selection set and owner === 'bob' will result in the subscription receiving a message.

When using an array of owners, the value of the owner field is always changing, as users get added/removed.

So at connect time we may have

onUpdateMessage(["bob", "allice"]) {
    id
    content
    _version
    _deleted
    _lastChangedAt
    createdAt
    updatedAt
}

However, once we add or remove a user to this array, the subscription will no longer receive messages. Since DataStore depends on subscriptions in order to keep the local store up to date, this prevents this auth rule from being usable with DataStore at the moment.

So suppose user is scrolling the list, he came to end of list for earlier from date and now I called the change function so it would start syncing with new value. How would I know when the sync is completed so to display it?

If you’re expecting that to be a common use case, I would not recommend utilizing selective sync for this. Instead, I would leave the default behavior where all of the Meetings get persisted to the local store and then use DataStore.query with pagination to update your UI state.

Since DataStore follows an offline-first model, it should contain all of the data that your users may plausibly need in order to comprehensively interact with your application.

If that ^ approach doesn’t work for you, could you please elaborate on your app requirements and use cases in detail?