aws-sdk-js-v3: SignatureDoesNotMatch when migrating from AWS SDK v2 to v3

Checkboxes for prior research

Describe the bug

I’m trying to migrate my codebase from v2 to v3 (like the title suggests) and I’m getting the following error:

{"name":"SignatureDoesNotMatch","$fault":"client","$metadata":{"httpStatusCode":403,"requestId":"76a6508d-7495-4152-8915-3b082272f2dc","attempts":1,"totalRetryDelay":0},"Type":"Sender","Code":"SignatureDoesNotMatch","message":"Signature expired: 20230907T155358Z is now earlier than 20230907T155403Z (20230907T155903Z - 5 min.)"}

This is my code I’m trying to replicate:

const client = new CloudWatchClient({
  region: 'us-east-1',
  credentials: fromEnv(),
})

export class CloudWatchUtils {
  async logMySqlResponseTimes(millis: number, operation: string) {
    const params = this.getMillisParams(operation, 'MySql', millis)

    try {
      return await client.send(new PutMetricDataCommand(params))
    } catch (e) {
      logger.error(`Could not log mysql response time to cloudwatch: ${JSON.stringify(e)}`)
    }
  }
}

And my code used to be like this (with v2):

const cloudwatch = new CloudWatch()

export class CloudWatchUtils {
  async logMySqlResponseTimes(millis: number, operation: string) {
    const params = this.getMillisParams(operation, 'MySql', millis)

    try {
      await cloudwatch.putMetricData(params).promise()
    } catch (error) {
      logger.error(`Could not log mysql response time ${error}`)
    }
  }
}

There’s only one thing to consider: The code you are seeing is executed but we are not awaiting for the response when we call logMySqlResponseTimes since we are not interested in getting a result back.

With that being said, the sdk v3 throws the SignatureDoesNotMatch exception and the one with the v2 does not.

Also, tried using requestHandler with a maxSockets number (like 60/70) but it still didn’t work and I want my code to be agnostic of any infraestructure or traffic that may be going on.

Please note that this is happening on a very busy lambda with around 1.7k invocations per minute (around 8-25 concurrent executions)

SDK version number

@aws-sdk/client-cloudwatch@3.391.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v18.17.0

Reproduction Steps

This is where we comunicate with cloudwatch

import { CloudWatchClient, PutMetricDataCommand } from '@aws-sdk/client-cloudwatch'
import { fromEnv } from '@aws-sdk/credential-providers'

const client = new CloudWatchClient({
  region: 'us-east-1',
  credentials: fromEnv(),
})

export class CloudWatchUtils {
  async logMySqlResponseTimes(millis: number, operation: string) {
    const params = this.getMillisParams(operation, 'MySql', millis)

    try {
      return await client.send(new PutMetricDataCommand(params))
    } catch (e) {
      logger.error(`Could not log mysql response time to cloudwatch: ${JSON.stringify(e)}`)
    }
  }


private getMillisParams(operation: string, service: string, millis: number) {
    return {
      Namespace: this.getNameSpace(),
      MetricData: [
        {
          MetricName: operation,
          // This might seem counterintuitive, but this way we have all
          // metrics grouped by their respective service
          Dimensions: [{ Name: service, Value: 'Milliseconds' }],
          Timestamp: new Date(),
          Value: millis,
          Unit: 'Milliseconds',
        },
      ],
    }
  }
}

And here where we use it:

import { CloudWatchUtils } from '../utils/CloudWatchUtils'

const NS_PER_SEC = 1e9
const MS_PER_NS = 1e6

type Function = () => Promise<any>
type Chronometer = (millis: number, operation: string) => Promise<void>

export function getProbabilisticChronometer(p: number, chron: Chronometer): Chronometer {
  if (p < 0 || p > 1) throw new Exception("'p' must be in the [0,1] range")
  return async (millis: number, operation: string) => {
    if (Math.random() < p) await chron(millis, operation)
  }
}

export async function timeIt(func: Function, chron: Chronometer, operation: string) {
  const time = process.hrtime()
  const ans = await func()
  const millis = hrtimeToMillis(process.hrtime(time))
  // Not awaiting, since we are not interested in getting a result back
  chron(millis, operation)
  return ans
}


const mysql = require('serverless-mysql')()

mysql.config({
  host: process.env.MYSQL_HOST,
  database: process.env.MYSQL_DATABASE,
  user: process.env.MYSQL_USER,
  password: process.env.MYSQL_PASSWORD,
})

export const enum MySQLTables {
  Devices = 'devices',
}

const CHRON_PROBABILITY = parseFloat(0.2)

const cloudWatch = new CloudWatchUtils()

const chron = getProbabilisticChronometer(
  CHRON_PROBABILITY,
  cloudWatch.logMySqlResponseTimes.bind(cloudWatch)
)

export class MySQLStorage {
  async select(table: MySQLTables, columnValues: object) {
    const valuesCount = Object.keys(columnValues).length

    const equalConditions = new Array(valuesCount).fill('?? = ?').join(' AND ')

    const query = `SELECT * FROM ?? WHERE ${equalConditions}`

    const queryValues = [table].concat(...Object.entries(columnValues))

    return await timeIt(() => mysql.query(query, queryValues), chron, 'QUERY')
  }
}

Since this may be a little complicated, the only thing important is that we do not await for the cloudwatch method. And for some context, this is not done all the time, only 20% of the time.

Observed Behavior

It seems that it throws error:

{"name":"SignatureDoesNotMatch","$fault":"client","$metadata":{"httpStatusCode":403,"requestId":"76a6508d-7495-4152-8915-3b082272f2dc","attempts":1,"totalRetryDelay":0},"Type":"Sender","Code":"SignatureDoesNotMatch","message":"Signature expired: 20230907T155358Z is now earlier than 20230907T155403Z (20230907T155903Z - 5 min.)"}

Expected Behavior

It should not throw that error and log the metric to cloudwatch

Possible Solution

No response

Additional Information/Context

No response

About this issue

  • Original URL
  • State: open
  • Created 10 months ago
  • Reactions: 3
  • Comments: 31 (13 by maintainers)

Most upvoted comments

We are also experiencing this issue, although in our case it’s occurring sporadically when calling out to SSM from a lambda.

We did not see this behaviour previously on the V2 SDK, only on the V3 SDK.

Hi there, seeing a similar thing since migrating from v2 to v3 where we are quite regularly getting errors like this:

Signature expired: 20231020T104900Z is now earlier than 20231020T105332Z (20231020T105832Z - 5 min.)

I have also tried upping the max connections to 100 but still seeing this issue that never occured on v2 on exactly the same system with exactly the same load.

It is possible we are calling out too much to cloudwatch from our lambdas but it is just odd that we never saw this error on v2. Was something changed around how the sdk request queue is handled?

For a workaround, you can move the api call inside the Lambda handler as follows:

Before

// ...
const client = new SSM();
const secret = await client.getSecretValue(params);

const handler = async (event) => {
  // Use the secret.
}

After

// ...
const client = new SSM();
let secret;

const handler = async (event) => {
  if (secret === undefined) {
    secret = await client.getSecretValue(params);
  }
  // Use the secret.
}

This workaround ensures that SDK API calls will not face Node.js Async Init Problem, as the API call is made inside the handler. The API call will also be made just once per function init.

Hey @trivikr @RanVaknin - any further updates regarding this? We had a production incident due to this issue (again).

Hey @RanVaknin - Are there any updates regarding this issue? Is this issue prioritised? Thank you kindly