aws-sdk-js: DynamoDB: 500 InternalServerError transactWrite on specific table

Hi,

Description We are trying to execute the following TransactItems request

documentClient.transactWrite({ TransactItems: [{ Put: { TableName, Item } }] }).promise()

About 50% of the time we get the following error. Before throwing the error, the request hangs for about 20s.

Error [InternalServerError]: Internal server error
    at Request.extractError (/Users/code/node_modules/aws-sdk/lib/protocol/json.js:52:27)
    at Request.callListeners (/Users/code/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/Users/code/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/Users/code/node_modules/aws-sdk/lib/request.js:688:14)
    at Request.transition (/Users/code/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/code/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/code/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/code/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/code/node_modules/aws-sdk/lib/request.js:690:12)
    at Request.callListeners (/Users/code/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'InternalServerError',
  time: 2021-09-16T17:10:16.935Z,
  requestId: 'FM8089LSJTQNBP762J7V06NJLVVV4KQNSO5AEMVJF66Q9ASUAAJG',
  statusCode: 500,
  retryable: true
}

Things we have checked

• The error disappears when we use the aws-cli like so aws dynamodb transact-write-items --transact-items '[{"Put":{"TableName":"fake_table_name","Item":{"fake_pk":{"S":"2e0fe3a7-2609-4a45-8183-0ce6ade9870e"}}}}]' . This is the primary reason I have listed it here as a js cli bug.
• The error also disappears when we use aws-sdk-v3.
• All other tables execute fine with this exact same operation.
• We get the error regardless of the value of the item Key / payload.
• We have tried deleting and recreating the table multiple times to no avail. Note we have recreated the table many times in the past in the same way and we’ve never had this issue.
• Other operations on this table succeed with no errors - we’ve tested PutItem, GetItem and DeleteItem.
• We are not being throttled. We have chosen ON_DEMAND / PAY_PER_REQUEST billing model. Read and write capacity remain below 1. Cloudwatch insights and the dynamo dashboard on the table show we have not been throttled. Just in case, we tried manually upgrading the read and write capacity units to a minimum of 5 RCU and 5 WCU but regardless the issue persists. Also the error does not indicate throttling issues.
• We tried populating the table with 10,000 rows incase that could fix things but the error persisted after the insertion.
• The error disappears if we recreate the table with all the same config except with a different table name.
• The error disappears if we recreate the table with all the same config except with a different partition key name.

At this point we’re out of ideas. The only thing we haven’t explored is deleting backups but I don’t see why this would affect anything.

😱😢

C’mon AWS! Show me the light!

Thanks

Is the issue in the browser/Node.js? Node.js

If on Node.js, are you running this on AWS Lambda? Yes but also fails locally.

Details of the browser/Node.js version v14.16.1

SDK version number 2.989.0

  ├─┬ aws-cdk@1.102.0
│ ├── aws-sdk@2.866.0
│ └─┬ cdk-assets@1.102.0
│   └── aws-sdk@2.866.0 deduped
├── aws-sdk@2.989.0
├─┬ serverless-domain-manager@3.3.2
│ └── aws-sdk@2.989.0 deduped
├─┬ serverless-offline@6.9.0
│ └── aws-sdk@2.989.0 deduped
└─┬ serverless@2.51.0
  └── aws-sdk@2.989.0 deduped