amplify-cli: nextToken not accepted and gives error "Invalid pagination token given"

Describe the bug Query is working in development and build environments but when deployed in the Test environment the nextToken is never accepted.

The same error occurs in the code as well as the AppSync console where we do a query with a limit and get a nextToken. Then we do a query straight away with the same token and it gives the following error: com.amazonaws.deepdish.common.pagination.InvalidPaginationTokenException: Invalid pagination token given.

To Reproduce Do following query in the console (same as in the code) FYI There are 15 items in the table.

{
  listUsers(limit:3){
    items{
      name
    }
    nextToken
  }
}

The response is:

{
  "data": {
    "listUsers": {
      "items": [
        {
          "name": "Joergen Von Klopp"
        },
        {
          "name": "Yolanda McAdams"
        },
        {
          "name": "Info Test"
        }
      ],
      "nextToken": "eyJ2ZX...shortened="
    }
  }
}

Then we do it again with the nextToken:

{
  listUsers(limit:3,nextToken:"eyJ2ZX...shortened="){
    items{
      name
    }
    nextToken
  }
}

The error is:

{
  "data": {
    "listUsers": null
  },
  "errors": [
    {
      "path": [
        "listUsers"
      ],
      "data": null,
      "errorType": "DynamoDB:UserIllegalArgumentException",
      "errorInfo": null,
      "locations": [
        {
          "line": 2,
          "column": 3,
          "sourceName": null
        }
      ],
      "message": "com.amazonaws.deepdish.common.pagination.InvalidPaginationTokenException: Invalid pagination token given."
    }
  ]
}

We experience the same behavior with all the tables in this environment. If we increase the limit we of course get all the elements but for some reason nextToken does not work.

Expected behavior Should get more data returned like on the other environments.

Screenshots If applicable, add screenshots to help explain your problem.

Environment
npx envinfo --system --binaries --browsers --npmPackages --npmGlobalPackages
npx: installed 1 in 1.301s

  System:
    OS: Windows 10 10.0.18362
    CPU: (12) x64 Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
    Memory: 8.41 GB / 23.94 GB
  Binaries:
    Node: 12.13.1 - C:\Tools\nodejs\node.EXE
    npm: 6.13.2 - C:\Tools\nodejs\npm.CMD
  Browsers:
    Edge: 44.18362.449.0
    Internet Explorer: 11.0.18362.1

Additional context Amplify --version 4.5.0 “aws-amplify”: “^2.2.2”, “aws-amplify-react”: “^3.1.3”,

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 2
  • Comments: 48 (9 by maintainers)

Commits related to this issue

Most upvoted comments

i also experience this error.

use this query on aws console

query list {
  listUsers {
    items {
      id
    }
    nextToken
  }
}

console return

{
  "data": {
    "listUsers": {
      "items": [
        {
          "id": "692d47ae-ee2e-4e64-90f1-5a9984382f14"
        },
        {
          "id": "9d096d0f-c057-4d29-b30f-de24254a6cf9"
        },
        {
          "id": "a6c937e6-06b3-47be-8306-2d5cde74b0e2"
        },
        {
          "id": "ff5d01b8-a7dd-44d0-8fa2-d0d738233ac1"
        },
        {
          "id": "f7d4960f-bb80-4dc6-a72e-83824afdc0e4"
        },
        {
          "id": "5b2d0722-f38e-40b7-956f-4461f1eae367"
        },
        {
          "id": "2277475b-718c-4582-be91-cc8105ce90ca"
        },
        {
          "id": "71a93b23-742d-4920-84e5-b9f34e77d3c0"
        },
        {
          "id": "15a6bf64-4854-498f-b50e-3d3c5fe26bfe"
        },
        {
          "id": "a6f981a7-50b4-4660-acb1-b297c97503a2"
        }
      ],
      "nextToken": "eyJ2ZXJzaW9uIjoyLCJ0b2tlbiI6IkFRSUNBSGg5OUIvN3BjWU41eE96NDZJMW5GeGM4WUNGeG1acmFOMUpqajZLWkFDQ25BRS9Cc1c1Tks2WTRhVTN3ckZ2Y3N1WUFBQUNIRENDQWhnR0NTcUdTSWIzRFFFSEJxQ0NBZ2t3Z2dJRkFnRUFNSUlCL2dZSktvWklodmNOQVFjQk1CNEdDV0NHU0FGbEF3UUJMakFSQkF4M21KMUYyWTVvMVBTM0x1WUNBUkNBZ2dIUE5rVTdldzdiNFZSVlB2SVkwYW9sTGxXVjJBakRQMlZ5MXBtdW9FQzF0V3F1Rk5YckE3bmcwMFF6bmtuS1JRakZwTnRDeWVkaDZsdFVTUmhZbTRwaXpVVjdPUERJT1RqNTZ6OHZGVzVRVVI5RTVxUGdta3hiNDdjUFJQMjVxdTVpM3RUMFVPaXNBREN1VHRXdTlZYXhqa1NuVzVDWnNoZFdFK3hMd2NBQ3JWa2dzQ2dBZ2hmZ2M1RVp5Mk5EVnB4SWJjc2ZVYW9va2UxWEcwcUt0cGhjN2gzQkVWaC81d1lXVU9iMzVjMEZ4d3lyVGxoRllvSjQzRzZWU1FBbW8rdU5oMEt6Wm9WY0NvZElqMjg0UXM4NXFPZFc4dDhIaHJ6SnJMN0orZ3oxUU95QU1XQ1JXa2V6c1ZkYVFpalRpV0JYaktQS3hUY1Z0NEk4R0F0R1Q4OWI0VnlDdmpnN05RVXZpSTFJWTN6bTl1Vm5ydXNBYXhzT0ZUUEZKZVBaRDVQM3BvWDBueFdxTkkvWDhQNkt6Q1ljdW1jVzUvTUlLYzJXVU9sTXdyRE5TK3F0Wis4cnZFVHBmaVhiQlZEcFYzdmlJdXlnQkwwaURXRE95MzdrTDVFc3A5V2R1TkRHTDZMRklZem43VFdOL0t2cTArQnpZT3RLTFQvakZPZXphcHdRcFM5WXM3a0h3cWVaWFVXRjNQWlZDT0JaQWlMaHI1c3R6RkhwRmtVYlNPYzU2M1g5Tm03dzNsM3JBMVgyUFRkamxFVFN5RjNCS2ltU1YzLzFzSmVhNm93ODNJeGdueG1qeWZqaE5ZeVUrZz09In0="
    }
  }
}

then

query list {
  listUsers(nextToken: "eyJ2ZXJzaW9uIjoyLCJ0b2tlbiI6IkFRSUNBSGg5OUIvN3BjWU41eE96NDZJMW5GeGM4WUNGeG1acmFOMUpqajZLWkFDQ25BRS9Cc1c1Tks2WTRhVTN3ckZ2Y3N1WUFBQUNIRENDQWhnR0NTcUdTSWIzRFFFSEJxQ0NBZ2t3Z2dJRkFnRUFNSUlCL2dZSktvWklodmNOQVFjQk1CNEdDV0NHU0FGbEF3UUJMakFSQkF4M21KMUYyWTVvMVBTM0x1WUNBUkNBZ2dIUE5rVTdldzdiNFZSVlB2SVkwYW9sTGxXVjJBakRQMlZ5MXBtdW9FQzF0V3F1Rk5YckE3bmcwMFF6bmtuS1JRakZwTnRDeWVkaDZsdFVTUmhZbTRwaXpVVjdPUERJT1RqNTZ6OHZGVzVRVVI5RTVxUGdta3hiNDdjUFJQMjVxdTVpM3RUMFVPaXNBREN1VHRXdTlZYXhqa1NuVzVDWnNoZFdFK3hMd2NBQ3JWa2dzQ2dBZ2hmZ2M1RVp5Mk5EVnB4SWJjc2ZVYW9va2UxWEcwcUt0cGhjN2gzQkVWaC81d1lXVU9iMzVjMEZ4d3lyVGxoRllvSjQzRzZWU1FBbW8rdU5oMEt6Wm9WY0NvZElqMjg0UXM4NXFPZFc4dDhIaHJ6SnJMN0orZ3oxUU95QU1XQ1JXa2V6c1ZkYVFpalRpV0JYaktQS3hUY1Z0NEk4R0F0R1Q4OWI0VnlDdmpnN05RVXZpSTFJWTN6bTl1Vm5ydXNBYXhzT0ZUUEZKZVBaRDVQM3BvWDBueFdxTkkvWDhQNkt6Q1ljdW1jVzUvTUlLYzJXVU9sTXdyRE5TK3F0Wis4cnZFVHBmaVhiQlZEcFYzdmlJdXlnQkwwaURXRE95MzdrTDVFc3A5V2R1TkRHTDZMRklZem43VFdOL0t2cTArQnpZT3RLTFQvakZPZXphcHdRcFM5WXM3a0h3cWVaWFVXRjNQWlZDT0JaQWlMaHI1c3R6RkhwRmtVYlNPYzU2M1g5Tm03dzNsM3JBMVgyUFRkamxFVFN5RjNCS2ltU1YzLzFzSmVhNm93ODNJeGdueG1qeWZqaE5ZeVUrZz09In0=") {
    items {
      id
    }
    nextToken
  }
}

console return

{
  "data": {
    "listUsers": null
  },
  "errors": [
    {
      "path": [
        "listUsers"
      ],
      "data": null,
      "errorType": "DynamoDB:UserIllegalArgumentException",
      "errorInfo": null,
      "locations": [
        {
          "line": 2,
          "column": 3,
          "sourceName": null
        }
      ],
      "message": "com.amazonaws.deepdish.common.pagination.InvalidPaginationTokenException: Invalid pagination token given."
    }
  ]
}

This is still happening for me on 4.13.4

I am also coming across this error when using next token for pagination. This seems to have began after I pushed to my development environment the same day this ticket was raised (The error wasn’t spotted until today). This push included a two field change to a table in my graphql schema, where said fields had their type changed from ‘[String]!’ to ‘AWSJSON!’. Interestingly, the now failing query that uses next token for pagination reads from a different table than that which was changed.

I am using Cognito user groups for authentication, and I can provide full schema if necessary.

Here is the error that I have captured in the AppSync console:

{
  "data": {
    "articlesByDate": null
  },
  "errors": [
    {
      "path": [
        "articlesByDate"
      ],
      "data": null,
      "errorType": "DynamoDB:UserIllegalArgumentException",
      "errorInfo": null,
      "locations": [
        {
          "line": 2,
          "column": 3,
          "sourceName": null
        }
      ],
      "message": "com.amazonaws.deepdish.common.pagination.InvalidPaginationTokenException: Invalid pagination token given."
    }
  ]

And here is the query req vtl in question (not quite as tidy as the gentleman above):

## [Start] Set query expression for @key **
#set( $modelQueryExpression = {} )
## [Start] Validate key arguments. **
#if( !$util.isNull($ctx.args.datePublished) && $util.isNull($ctx.args.dataType) )
  $util.error("When providing argument 'datePublished' you must also provide arguments dataType", "InvalidArgumentsError")
#end
## [End] Validate key arguments. **
#if( !$util.isNull($ctx.args.dataType) )
  #set( $modelQueryExpression.expression = "#dataType = :dataType" )
  #set( $modelQueryExpression.expressionNames = {
  "#dataType": "dataType"
} )
  #set( $modelQueryExpression.expressionValues = {
  ":dataType": {
      "S": "$ctx.args.dataType"
  }
} )
#end
## [Start] Applying Key Condition **
#if( !$util.isNull($ctx.args.datePublished) && !$util.isNull($ctx.args.datePublished.beginsWith) )
  #set( $modelQueryExpression.expression = "$modelQueryExpression.expression AND begins_with(#sortKey, :sortKey)" )
  $util.qr($modelQueryExpression.expressionNames.put("#sortKey", "datePublished"))
  $util.qr($modelQueryExpression.expressionValues.put(":sortKey", { "S": "$ctx.args.datePublished.beginsWith" }))
#end
#if( !$util.isNull($ctx.args.datePublished) && !$util.isNull($ctx.args.datePublished.between) )
  #set( $modelQueryExpression.expression = "$modelQueryExpression.expression AND #sortKey BETWEEN :sortKey0 AND :sortKey1" )
  $util.qr($modelQueryExpression.expressionNames.put("#sortKey", "datePublished"))
  $util.qr($modelQueryExpression.expressionValues.put(":sortKey0", { "S": "$ctx.args.datePublished.between[0]" }))
  $util.qr($modelQueryExpression.expressionValues.put(":sortKey1", { "S": "$ctx.args.datePublished.between[1]" }))
#end
#if( !$util.isNull($ctx.args.datePublished) && !$util.isNull($ctx.args.datePublished.eq) )
  #set( $modelQueryExpression.expression = "$modelQueryExpression.expression AND #sortKey = :sortKey" )
  $util.qr($modelQueryExpression.expressionNames.put("#sortKey", "datePublished"))
  $util.qr($modelQueryExpression.expressionValues.put(":sortKey", { "S": "$ctx.args.datePublished.eq" }))
#end
#if( !$util.isNull($ctx.args.datePublished) && !$util.isNull($ctx.args.datePublished.lt) )
  #set( $modelQueryExpression.expression = "$modelQueryExpression.expression AND #sortKey < :sortKey" )
  $util.qr($modelQueryExpression.expressionNames.put("#sortKey", "datePublished"))
  $util.qr($modelQueryExpression.expressionValues.put(":sortKey", { "S": "$ctx.args.datePublished.lt" }))
#end
#if( !$util.isNull($ctx.args.datePublished) && !$util.isNull($ctx.args.datePublished.le) )
  #set( $modelQueryExpression.expression = "$modelQueryExpression.expression AND #sortKey <= :sortKey" )
  $util.qr($modelQueryExpression.expressionNames.put("#sortKey", "datePublished"))
  $util.qr($modelQueryExpression.expressionValues.put(":sortKey", { "S": "$ctx.args.datePublished.le" }))
#end
#if( !$util.isNull($ctx.args.datePublished) && !$util.isNull($ctx.args.datePublished.gt) )
  #set( $modelQueryExpression.expression = "$modelQueryExpression.expression AND #sortKey > :sortKey" )
  $util.qr($modelQueryExpression.expressionNames.put("#sortKey", "datePublished"))
  $util.qr($modelQueryExpression.expressionValues.put(":sortKey", { "S": "$ctx.args.datePublished.gt" }))
#end
#if( !$util.isNull($ctx.args.datePublished) && !$util.isNull($ctx.args.datePublished.ge) )
  #set( $modelQueryExpression.expression = "$modelQueryExpression.expression AND #sortKey >= :sortKey" )
  $util.qr($modelQueryExpression.expressionNames.put("#sortKey", "datePublished"))
  $util.qr($modelQueryExpression.expressionValues.put(":sortKey", { "S": "$ctx.args.datePublished.ge" }))
#end
## [End] Applying Key Condition **
## [End] Set query expression for @key **
#set( $limit = $util.defaultIfNull($context.args.limit, 10) )
#set( $QueryRequest = {
  "version": "2017-02-28",
  "operation": "Query",
  "limit": $limit,
  "query": $modelQueryExpression,
  "index": "byDate"
} )
#if( !$util.isNull($ctx.args.sortDirection)
                    && $ctx.args.sortDirection == "DESC" )
  #set( $QueryRequest.scanIndexForward = false )
#else
  #set( $QueryRequest.scanIndexForward = true )
#end
#if( $context.args.nextToken ) #set( $QueryRequest.nextToken = "$context.args.nextToken" ) #end
#if( $context.args.filter ) #set( $QueryRequest.filter = $util.parseJson("$util.transform.toDynamoDBFilterExpression($ctx.args.filter)") ) #end
$util.toJson($QueryRequest)

And the res vtl: $util.toJson($ctx.result)

Has anyone rebuilt their environment from scratch to see if this fixes it? I would prefer not to have to do this, but when in doubt…

Just a bit of info, I was running into this as well after I upgraded from amplify-cli 4.13.1 to 4.13.2. Once I downgraded to 4.13.1 and did another amplify push life was better.

We’ve root-caused the issue with the latest release and working on a fix for this which will be released soon. cc: @yuth

In an earlier version of the Amplify CLI a bug was introduced which double encoded the payload toJSON. What I suspect is you might have overridden the template when the CLI had the bug, which got you the buggy resolver. This was fixed in the CLI later, but since the template was overriden, that fix not propagate to that template.

Hello, I’m having this issue with amplify 4.25.0.

Hello, This issue is back with the latest (4.18.1) Amplify CLI Version. Overriding the version with 4.16.1 (which is the one I happen to know is working) as described at https://docs.aws.amazon.com/amplify/latest/userguide/custom-build-image.html#setup-live-updates no longer throws this error.

I just wanted to confirm that I had this issue with 4.13.3 but upgrading to 4.13.4 resolve it for me. The steps I took were:

  • npm install -g @aws-amplify/cli@latest
  • make a simple change to the schema (I added a field)
  • amplify api gql-compile
  • amplify push That was it… works fine now… thanks guys.

Upgraded to 4.13.4 . The issue is resolved .Thanks

On Fri, 21 Feb 2020, 03:03 Andrew Panagos, notifications@github.com wrote:

We tested it today and I can confirm that the fix does work. Thank you @kaustavghosh06 https://github.com/kaustavghosh06 , @SwaySway https://github.com/SwaySway , @AaronHarris https://github.com/AaronHarris , @sammartinez https://github.com/sammartinez , @yuth https://github.com/yuth , @attilah https://github.com/attilah for addressing the issue so quickly. Now I have made some changes to our process. I changed “AWS Amplify” to not use the latest version of Amplify but to use a versioned (4.13.4) so it is in-sync with what we are developing with. As we upgrade and test our version of Amplify then we will change the “AWS Amplify” version so we are not surprised. Thanks Again.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/aws-amplify/amplify-cli/issues/3411?email_source=notifications&email_token=AE5UPKLAYER57S6PXF6AYT3RD3ZMJA5CNFSM4KT7CL52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMQHIIY#issuecomment-589329443, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE5UPKMMSR7TTIIIRJG3L4TRD3ZMJANCNFSM4KT7CL5Q .

I just upgraded to version 4.13.4 and things appear to be working for me as well. I’ll keep folks posted if otherwise. Thanks!

@kaustavghosh06 Please note we (i.e., myself and @AshwiniRaysecur )have tested publishing our app with version 4.13.4 and the issue seems to be resolved. Smoke tests show our app once again behaves as expected.

I attempted to build the application again from scratch, and still getting the pagination error for Amplify version 4.13.3

The resolver generated ‘Query.listSecurSamples.req.vtl’ file:

#set( $limit = $util.defaultIfNull($context.args.limit, 10) )
#set( $ListRequest = {
  "version": "2017-02-28",
  "limit": $limit
} )
#if( $context.args.nextToken )
  #set( $ListRequest.nextToken = "$context.args.nextToken" )
#end
#if( $context.args.filter )
  #set( $ListRequest.filter = $util.parseJson("$util.transform.toDynamoDBFilterExpression($ctx.args.filter)") )
#end
#if( !$util.isNull($modelQueryExpression)
                        && !$util.isNullOrEmpty($modelQueryExpression.expression) )
  $util.qr($ListRequest.put("operation", "Query"))
  $util.qr($ListRequest.put("query", $modelQueryExpression))
  #if( !$util.isNull($ctx.args.sortDirection) && $ctx.args.sortDirection == "DESC" )
    #set( $ListRequest.scanIndexForward = false )
  #else
    #set( $ListRequest.scanIndexForward = true )
  #end
#else
  $util.qr($ListRequest.put("operation", "Scan"))
#end
$util.toJson($ListRequest)

Worked correctly after downgrading to Amplify version 4.13.0

As per @a-j-olly’s comment above, can confirm it works when I downgraded the cli from 4.13.2 to 4.13.0 and do amplify push.

@malcomm For me, this issue was occurring on CLI version 4.13.1, however once I downgraded to 4.13.0 it cleared up. I did try and upgrade to 4.13.2 to see if that might be a fix, but I could still reproduce the error.

@olivoil I’d recommend taking the CLI down to 4.13.0 and then doing a amplify push and see if that works.

Hopefully the Amplify team can get to the bottom of this, but I have a work around for now.

Having the same issue with the latest CLI version (4.13.3) on any @model list query. Forcing version 4.13.1 of the CLI in build settings -> Live package updates does not seem to solve it for me, and I could not test 4.13.0 as there’s another issue with this version that was fixed in 4.13.1

It happens with any @model, and is reproducible directly from the AppSync Console.

Generate a simple @model

type Todo @model @auth(rules: [{ allow: owner }]) {
  id: ID!
  owner: ID!
  text: String
  checked: Boolean
}

Create a few objects, and query them, copying the value of nextToken verbatim from a previous query directly in AppSync Console

query ListTodos {
  listTodos(limit: 2, nextToken: "ey...SRiJ9") {
    items {
      id
      text
      checked
    }
    nextToken
  }
}

The error shows up consistently

{
  "data": {
    "listTodos": null
  },
  "errors": [
    {
      "path": [
        "listTodos"
      ],
      "data": null,
      "errorType": "DynamoDB:UserIllegalArgumentException",
      "errorInfo": null,
      "locations": [
        {
          "line": 2,
          "column": 3,
          "sourceName": null
        }
      ],
      "message": "com.amazonaws.deepdish.common.pagination.InvalidPaginationTokenException: Invalid pagination token given."
    }
  ]
}