amplify-js: Clock Drift causing client to fire off infinite number of uncatchable 4xx errors to set up GraphQL

Hi @gracektay

I just confirmed this is indeed a bug with our clock drift correction function. Here’s where it goes wrong:

1. When server clock is 19:00:00 but the the client clock is 18:00:00, a request is initiated and failed with BadRequest error
2. Error triggers the drift correction and client offset the clock by 1 hour. The retried request has Date header at 19:00:00 (client clock + offset)
3. When the client clock is back to correct 19:00:00, the client offset is still 1 hour, the consequent request has Date header at 20:00:00 with the offset.
4. The request failed with BadRequest error and trigger the drift correction. The function compares the request date at 20:00:00 and the server clock 19:00:00, and set the offset to -1 hour, then initiate a retry request.
5. The retry request has local date of 19:00:00 and offset -1 hour, so the request Date header is 18:00:00 and hence failed with BadRequest. At this time the drift correction detect and set the offset to 1 hour again and trigger the retry. Then trigger the error in Step 3.
6. As result, the request is in infinite retry because the request Date is flipped back and forth between 18:00:00 and 20:00:00 whereas the server clock is 19:00:00.

We will implement a fix for this issue and let you know when it’s ready.

@gracektay Hi, a fix has been released in aws-amplify@5.3.12. Can you give a try with this version?

It’d be a bit difficult to share the entire code set up but that’s the component and this is the base API call:

import { API, Auth } from "aws-amplify";
import { getSessionToken } from "../../util/functions";
import { AUTH_MODE } from "../../api-config";
import { isAfter } from 'date-fns'
export async function graphqlBaseQuery(args) {
  const { query, variables } = args;
  let authMode
  const sessionToken = await getSessionToken().catch()
  const { identityId, expiration } = await Auth.currentCredentials().catch() ?? {}
  const expirationDate = new Date(expiration)
  const nowDate = new Date()
  const isExpired = isAfter(nowDate, expirationDate)
  if(sessionToken) {
    authMode = AUTH_MODE.AMAZON_COGNITO_USER_POOLS
  } else if(identityId && !isExpired) {
    authMode = AUTH_MODE.AWS_IAM
  }
  let response = {}  
  try {
    if(!authMode) {
      return {
        error: {
          status: 500,
          data: 'Unable to authenticate credentials'
        }
      }
    }
    const result = await API.graphql({ authMode, query, variables });
    if (result?.data) {
      response = result;
    } else {
      throw new Error("Unexpected data format");
    }
  } catch (error) {
    response = { error: { status: 500, data: (error?.errors ?? error) } };
    if(error?.data) {
      response.data = error.data
    }
    return {
      error
    }
  }
  return response;
}

the expired bit was an attempt to fix the issue, but it made no difference.

Reposting these docs in case it’s helpful to understand the polling: https://redux-toolkit.js.org/rtk-query/usage/polling

The polling just offers more opportunities for the infinite failure to occur.

@ashwinkumar6 to an extent that’s true, the minute repeated successful calls are fine. But the failures with clock drift also cause repeated calls, an infinite number of them until the browser is force closed.

@gracektay, have you also enabled CORS and checked the IAM permissions for the public identity pool? We’re having trouble reproducing the issue with GraphQL (but have reproduced it using Pinpoint similar to #9281), and trying to see if we can get the same recurring error you see.

If you have a minimal sample app (or even what the frontend code looks like), that would be helpful too!

There are several previously closed issues related to this. In particular, this issue was resolved by both a library fix, and a fix to the the CLI. @gracektay - can you also share what version of the CLI you are using? Thank you!

@gracektay, thank you for the additional information and pics! We are currently investigating this and will update this issue as soon as we can.

@AllanZhengYP Hi! I’ve updated https://d3rue5wy9qeptt.cloudfront.net. I have tested in Chrome and Safari, so far so good. I see it error once, retry and succeed. I’ll let you know if I see more issues! Thanks SO much!

This only impacts a couple of users (its a handful of people responsible for 50k requests) and for the most part everything is working as expected (except for when we switch back and forth on time) so my gut feeling is that it is not related to CORS or our permissions. If it was then I’d assume it’d be broken for the hundred thousand other users.

Here are some snippets from our IAM permissions in case its useful (with certain values removed just in case)

Trust relationship:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "cognito-identity.amazonaws.com"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "cognito-identity.amazonaws.com:aud": "us-east-2:[[identifier-removed]]"
                },
                "ForAnyValue:StringLike": {
                    "cognito-identity.amazonaws.com:amr": "unauthenticated"
                }
            }
        }
    ]
}
permissions:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "mobileanalytics:PutEvents",
                "cognito-sync:*",
                "cognito-identity:*"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "appsync:GraphQL"
            ],
            "Resource": [
                "arn:aws:appsync:us-east-2:[[accountid]]:apis/[[apiid]]/types/*/fields/*",
                "arn:aws:appsync:us-east-2:[[accountid]]:apis/[[apiid]]"
            ],
            "Effect": "Allow"
        }
    ]
}

Also, if its helpful to know, we had originally been unsure of the cause and my team had a case opened, CASE 14087351881, forwarded to the AppSync team.

@cwomack Hi! here are lots of pictures showing the issue from the client side. Hope this helps! Client side:

{
  "errors" : [ {
    "errorType" : "BadRequestException",
    "message" : "Signature not yet current: 20231027T153914Z is still later than 20231027T153419Z (20231027T152919Z + 5 min.)"
  } ]
}

Here are some pictures:

our call:

import { API } from "aws-amplify"; 
API.graphql({ authMode, query, variables });