amplify-js: Pinpoint: Exceeded maximum endpoint per user count: 10

All,

Just wanted to provide an update here to this issue. We are working internally with the Amazon Pinpoint team on this. While we are working on a solution, we may reach out to some of you to understand your use case more. Thank you for your patience on this!

Hi @sammartinez, thanks for asking. I would consider my case temporarily mitigated by rolling back, but the underlying issue re-introduced in @aws-amplify/analytics": "4.0.0" is still a problem. We uninstall / re-install apps so often that we quickly hit that 10 endpoint limit. It could also affect our production users though it would be more of an edge case. With no way for us to manage endpoints, this issue effectively breaks push notifications.

To me, this is a showstopper that precludes us from upgrading to 4.0.0 and beyond. We’re locked into specific versions and might not be able to benefit from future features or security patches. I feel we either need some mechanism to manually manage endpoints, automated management of endpoints in Amplify or (preferably) in Pinpoint, or other guidance that would allow us to continue staying current with Amplify packages while avoiding this issue.

I’ve realized that this issue is a critical security vulnerability. The version of aws-amplify that mitigates the Exceed maximum endpoint per user problem discussed in this issue is 3.3.8. That version of aws-amplify exposes two security problems.

1. @aws-amplify/datastore contains a dependency to immer@6.0.1. That package contains an exploit documented in CVE-2020-28477.

2. @aws-amplify/api-rest and @aws-amplify/storage depend on axios@0.19.0. The vulnerability in that package is documented in CVE-2020-28168.

The affected dependencies have been upgraded to patched versions in the latest release of @aws-amplify, however we are stuck on 3.3.8 because of the endpoint issue.

@sammartinez This is a show-stopping issue for us. Can you please provide some update on progress and/or a mitigation strategy we can implement today?

Hi @abdallahshaban557, any news on this issue?

FWIW, I’ve discovered that you can sidestep this issue by specifying your own endpointId to override the one that Amplify is assigning by default. If you’re using React Native, this could be as simple as:

import DeviceInfo from 'react-native-device-info';

Amplify.configure({
  ...awsmobile,
  Analytics: {
    AWSPinpoint: {
      appId: awsmobile.aws_mobile_analytics_app_id,
      region: awsmobile.aws_mobile_analytics_app_region,
      endpointId: DeviceInfo.getUniqueId(),  // overrides the default Amplify-generated endpoint ID
    },
  },
});

Then your user will get a unique (and unchanging) endpoint for each device they use to access your app, which is desirable for per-device push notifications.

This solution will still break if the user has too many devices, but it works for the overwhelming majority. Futhermore, where it falls down, we are arguably viewing the problem as a Pinpoint limitation rather than an Amplify bug.

Hello everyone, we are working with the Pinpoint team to alleviate this issue. In the future, the Pinpoint team is going to allow events to be recorded at the “User” level rather than at the endpoint, which would mean that the creation of the 15 endpoints would be significantly less prone to happen since an endpoint would not be required for recording analytics events. We will keep you updated when we have a path forward.

@josefaidt - any chance you can take a look at this issue? 🙏🏼

Hi @sammartinez,

It has been 5 months this issue has been open for, pretty much every single application that has aws-amplify installed with a version over 3.3.8+ will eventually have users no longer receive push notifications, I can imagine that could be thousands of projects.

For us we are currently waiting to release a new project because of this blocker and are also facing a fresh round of amplify / push notification problems with an update to an existing application.

When can we expect a resolution to this issue / stability to amplify push notifications?

I am on 3.3.26, and as soon as I call Analytics.updateEndpoint post login, I run into this issue:

ERROR [ERROR] 53:35.387 AWSPinpointProvider - updateEndpoint failed [BadRequestException: Exceeded maximum endpoint per user count:15].

Hello Everyone,

I wanted to provide an update here. We have been working with the Pinpoint team on resolving this issue. On the Pinpoint service, they are now detaching the oldest push endpoint from the user. @kylekirkby @joebernard @dylan-westbury Can you please try the latest of the Analytics category and validate you are not receiving the error? If you are, please share them so we can take any errors back to the Pinpoint team. Thanks again for being patience with us on this.

Hi @wasurocks - we completely agree with you, this is extremely frustrating. We are working closely with the Pinpoint team to get this resolved. We will provide timelines on when we expect this issue to be resolved.

Even the official doc has the same issue prompting in the console

Exceeded maximum endpoint per user count

Any news on this issue? We still suffer from this issue.

Is there any update on this issue? I am still running into this issue with aws-amplify 4.3.8:

Strangely, even after deleting all endpoints for the given user with aws pinpoint delete-user-endpoint ..., the problem persists…

I received this error - exceeded maximum endpoints per user count: 15 just 39 hours ago…I don’t believe it is fixed.

Thanks for this feedback everyone, I am sharing this feedback with the Amazon Pinpoint team since they should handle this within the Service. Will provide an update early next week or once I hear something.

@sammartinez I updated to "aws-amplify": "3.3.27" and am no longer able to reproduce this issue locally.

I installed and re-installed my React Native app onto a test device 15 times and called updateEndpoint after each installation. Each time the result was SUCCESS. I also verified that I could receive a push notification on that device after 15 installation attempts.

I see that others are still running into this issue so I will keep it open and continue testing.

Hi @joebernard

We upgraded aws-amplify and noticed this issue, so we downgraded back to the last working version we had within the app, which was:

"aws-amplify": "3.3.8",

We no longer received the “exceeded maximum endpoint per user” once we downgraded.

FWIW, we have had success by using the following to configure Amplify Analytics:

Amplify.configure({
    ...awsmobile,
    Analytics: {
      AWSPinpoint: {
        appId: awsmobile.aws_mobile_analytics_app_id,
        region: awsmobile.aws_mobile_analytics_app_region,
        endpointId: uniqueId,
      },
    },
  });

where uniqueId is the result of a call to DeviceInfo.getUniqueId() from the react-native-device-info package. This effectively gives you one single endpoint per device.

A full year in and we still need-discussion and investigating…when they had something that worked before…better get the pinpoint team to re-design their entire system instead of leaving in the 3 lines of code that fixed this…

I have noticed for some users:

• They have 14 - 15 “ACTIVE” endpoints, even some dating back more than a year ago to a device that is old / no longer in possession OR an older version of the operating system.

For example, I can see for a single user the following active endpoints for a single device “iPhone X”:

• 1 active endpoint for iOS 13.6
• 6 active endpoints for iOS 14.4.2
• 7 active endpoints for iOS 14.6

I read pinpoint documentation, an endpoint will only change to “INACTIVE” when an existing “address” is added to a new endpoint. This can be problematic because iOS refreshes the push token which is used as the address for an endpoint, so an endpoint with an old address / token can remain ACTIVE.

I also noticed some new endpoints have changed from optOut “NONE” to “ALL” over time, I don’t think that should be happening because each time the token refreshes we update the optOut to “NONE”. Could endpoints be changing from “NONE” to “ALL” ?

This is our code to update endpoint address with push token.

const updatePushToken = async (token) => {
    try {
      const user = await getCurrentAuthenticatedUser(); // calls Auth.currentAuthenticatedUser

      if (user && user.username) {
        const options = {
          optOut: 'NONE',
          address: token,
          channelType: Platform.OS === 'ios' ? 'APNS' : 'GCM',
          userId: user.username
        };

        await Analytics.updateEndpoint(options);
      }
    } catch (err) {
      console.log('error getCurrentAuthenticatedUser ', err);
    }

    await AsyncStorage.setItem('amplifyPushToken', token);
  };

We ask for push notification permissions when user is logged in also, but as a safety we save the token in storage and update the endpoint with that token (if it exists) from async storage in case for whatever reason a new endpoint is created without the address as the token.

We also set the user session to expire after a very long while, so there isn’t any conflict with the user having to log back in and mess up the existing endpoint in some way

any update?

I’m getting this error with @aws-amplify/analytics 4.0.22 so it doesn’t seem to be fixed for me either.

Hi @irfancnk - I am no longer a part of the Amplify team - I am adding @haverchuck /@renebrandel/ @cwomack to help answer this for you.

Hi @nubpro

Amplify version is 4.3.12 @aws-amplify/pushnotification version is 4.3.9

Were you expecting it to be fixed in these versions? This is in dev, but we are facing push notification issue in production.

What we notice is, overtime users will stop receiving push notifications in production. At first they will work, but then after some months they stop. So assuming the latest endpoint is not able to save once limit is reached, occasionally? Perhaps endpoints are changing from optOut NONE to ALL when creating a new endpoint?

As this issue has been ongoing for 2 years and there has been lots of instructions of how to implement on various Github issues that doesn’t reflect what is shown in the documentation and lots of work arounds and hacks posted by everyone, is it possible to provide the exact expected way to implement and we can test that?

For example, do we still need to update the endpoint ourselves like this. For our use case, we request push permission once user is logged in in iOS and update when onRegister is called. But we also store in local storage and update providing it exists as well as the userId, because for Android the user may not be logged in when the token is received.

  Amplify.configure({
    ...awsmobile,
    PushNotification: {
      requestIOSPermissions: false,
      appId: awsmobile.aws_mobile_analytics_app_id,
    },
  });

 PushNotification.onRegister(async (token) => {
    console.log("in app registration", token);

    await updatePushToken(token);
  });

  const updatePushToken = async (token) => {
    try {
      let user;
      try {
        user = await getCurrentAuthenticatedUser();
      } catch (err) {
        console.log("error getCurrentAuthenticatedUser: ", err);
      }

      if (user && user.username) {
        const options = {
          optOut: "NONE",
          address: token,
          userId: user.username
        };

        await Analytics.updateEndpoint(options);
        await AsyncStorage.setItem("xxxxxxx-xxxxxx-amplifyPushToken", token);
      }
    } catch (err) {
      console.log("error updating address for push", err);
    }
  };

Or can we depend on @aws-amplify/pushnotification to do this for us?

Or another preferred suggestion?

Thanks for the clarification. But since when is this active? Because 1 have one EMAIL, 1 TEL and 13 GCM (due to testing), and ran into the error 2 weeks ago…

Hi, thanks for the information. I am still trying to understand the root cause of it by talking to pinpoint team. Can you help me confirm if it is the case:

1. You have one user id linked to 15 endpoints now (13 GCM, 1 TEL, 1 Email) and you cannot add new endpoints to the same user.
2. In testing are you deleting the endpoint id in storage? Are you using react-native or just web js?

For web, the endpoint id is created and cached in local storage.

Normally it will be persisted so you should not run into the issue of creating another endpoint and reach limit, but if you are using Incognito or manually clean the localStorage, then the endpoint id will be recreated.

For now, if the endpoint channel type is “PUSH” ( e.g. ADM, PUSH, GCM, APNS, APNS_VOIP, APNS_VOIP_SANDBOX, BAIDU), then pinpoint will auto evict it if there are more than 15 endpoints associated with the userId (by default, Amplify use the identity id from identity pool as the userId).

The auto eviction won’t apply to the “Email” channel type though. In this case, we need to delete the overflowed endpoints if they reach the limits.

@sammartinez any update on this? Still experiencing this with the latest release…

FYI, I am now seeing 13 endpoints for 1 user. I am not seeing any errors, but I also don’t see a purge of stale endpoints.

@BabyDino on your question, the Amazon Pinpoint team has let me know that it’s based on the last active date they are using for the detach of unused endpoints

Thanks for the update @sammartinez!

@sammartinez Any idea when we will actually be able to use Pinpoint analytics in our Amplify apps?

Not only am I now stuck on this issue, I’ve also had to do some ridiculous workarounds to get Pinpoint working with Group based auth roles (Amplify CLI). It seems that the Amplify team has completely neglected the Pinpoint product within this stack as that group-based auth issue still isn’t solved either…

https://github.com/aws-amplify/amplify-cli/issues/5004 https://github.com/aws-amplify/amplify-cli/issues/4772

Same issue here, waiting for the fix to publish an app.

hey @joebernard

I can confirm this includes the endpoint management code.

You can also check by yourself with, e.g.

diff <(curl -s https://unpkg.com/@aws-amplify/analytics@3.3.12-hotfix.0/lib/Providers/AWSPinpointProvider.js) <(curl -s https://unpkg.com/@aws-amplify/analytics@4.0.9/lib/Providers/AWSPinpointProvider.js)

(This is comparing @aws-amplify/analytics@3.3.12-hotfix.0 and @aws-amplify/analytics@latest)

I’ve realized that this issue is a critical security vulnerability. The version of aws-amplify that mitigates the Exceed maximum endpoint per user problem discussed in this issue is 3.3.8. That version of aws-amplify exposes two security problems.

1. @aws-amplify/datastore contains a dependency to immer@6.0.1. That package contains an exploit documented in CVE-2020-28477.
2. @aws-amplify/api-rest and @aws-amplify/storage depend on axios@0.19.0. The vulnerability in that package is documented in CVE-2020-28168.

The affected dependencies have been upgraded to patched versions in the latest release of @aws-amplify, however we are stuck on 3.3.8 because of the endpoint issue.

@sammartinez This is a show-stopping issue for us. Can you please provide some update on progress and/or a mitigation strategy we can implement today?

Hey @joebernard, thanks for this callout. I will work with the team on seeing about getting an update to version 3.3.8 to update these dependencies specifically. Our ETA for this is later today. I will let you know once we update the version with the callouts above. As for the update on the mitigation, we are working with the Pinpoint team on this and will provide a timeline once we have one for you.