aws-sdk-js: InvalidSignatureException: Signature expired

First, I know there have been several similar issues in the past. I’ve tried to read through them all but they all seem to be either for issues that are closed or for different problems.

I have a bunch of apps that run on AWS + Docker and at some point will randomly start throwing these errors. I think the error usually comes up when I haven’t worked on the app for a couple days, so the “expiration” is pretty extreme. For example, the one I just got is “expired” by 2 days.

Signature expired: 20150307T194740Z is now earlier than 20150309T203545Z

Some details that are probably relevant:

  • These apps are using static API keys in ~/.aws/
  • The docker container links the .aws directory along the lines of docker run --volume "${HOME}/.aws":/home/app/.aws whatever-image-name
  • Eventually through some random combination of re-installing dependencies, running the app outside of docker, and reading chicken entrails, things do start working again — unfortunately, I can never figure out exactly why

I’ve dug through the internals of the SDK a bit, but I don’t quite know what I’m looking for and hoping you can point me in a more productive direction.

Currently I’m using v2.1.14 (I know there are a couple newer patch releases in the last couple days). Here is a recent stack track.

InvalidSignatureException: Signature expired: 20150307T194740Z is now earlier than 20150309T203545Z (20150309T204045Z - 5 min.)
  at Request.extractError (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/protocol/json.js:43:27)
  at Request.callListeners (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/sequential_executor.js:100:18)
  at Request.emit (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
  at Request.emit (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/request.js:604:14)
  at Request.transition (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/request.js:21:12)
  at AcceptorStateMachine.runTo (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/state_machine.js:14:12)
  at /home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/state_machine.js:26:10
  at Request.<anonymous> (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/request.js:22:9)
  at Request.<anonymous> (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/request.js:606:12)
  at Request.callListeners (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/sequential_executor.js:104:18)
  at Request.emit (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
  at Request.emit (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/request.js:604:14)
  at Request.transition (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/request.js:21:12)
  at AcceptorStateMachine.runTo (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/state_machine.js:14:12)
  at /home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/state_machine.js:26:10
  at Request.<anonymous> (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/request.js:22:9)
  at Request.<anonymous> (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/request.js:606:12)
  at Request.callListeners (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/sequential_executor.js:104:18)
  at callNextListener (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/sequential_executor.js:90:14)
  at IncomingMessage.onEnd (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/event_listeners.js:183:11)
  at IncomingMessage.emit (events.js:117:20)
  at _stream_readable.js:943:16
  at process._tickDomainCallback (node.js:463:13)

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Reactions: 2
  • Comments: 26 (10 by maintainers)

Most upvoted comments

I encountered this issue locally, and restarting my docker daemon fixed it.

+25
Clee681 on Aug 9, 2016

Turns out mine was a clock syncing issue on my VM. Running a sync between the VM and the host cleared it up.

+19
kyriesent on Jun 12, 2015

hi all,

same issue ‘Signature expired’ on aws command line resolved by ‘calling ntpd’ (The Network Time Protocol (NTP) is used to synchronize the time of a computer).

hope it helps regards

+5
lraulier on Feb 8, 2016

Run the command to sync clock ntpdate pool.ntp.org

+4
zeeshanjamal16 on Jan 14, 2019

None of the above mentions workarounds worked for me, even though the environment was similar.

One difference may be that I was uploading via the --zip-file method of the aws-cli over a slow network. The upload was actually taking more than 5 minutes and consequently giving the SignatureDoesNotMatch error.

I was able to work around this slow network by uploading first to s3, and then instead of the zip-file command line option using the --s3-bucket and --s3-key options. Perhaps this is an unrelated issue but it took me long enough to solve that I thought I’d document it here in case.

+4
mankins on Mar 7, 2017

@evansolomon I’ve just pushed a patch that offsets the SDK clock when a clock skew error is detected. Simply set the correctClockSkew option when constructing a service client.

Here’s an example:

var dynamodb = new AWS.DynamoDB({correctClockSkew: true});

Let me know if this works for you. Thanks for your patience.

+3
AdityaManohar on Sep 15, 2015

it looks like the skew is actually 5 min and not 48 hours

I assume you mean this line

20150307T194740Z is now earlier than 20150309T203545Z (20150309T204045Z - 5 min.)

I parsed that as March 7, 2015 19:47 is earlier than March 9, 2015 20:40. I don’t really know what the 5 minute thing on the end means, I was thinking maybe that was the threshold.

Would you be able to re-run the date command on your Docker image and let me know what you find?

Just tried again on an image I haven’t used since yesterday and got “Tue Mar 10 18:22:48 UTC 2015” (which is correct at the time I’m writing this).

Do you also encounter the error when you run the same code outside your Docker image?

I’m not sure. Our workflow is pretty heavily Docker-centric, so this isn’t much of a use case for me.

would you be able to share your Docker image?

It is based on https://github.com/phusion/passenger-docker. The only differences are some nginx configs, the shared ~/.aws directory, and my apps.

+3
evansolomon on Mar 10, 2015

I’ve looked into the clock thing before but don’t think that’s it. In the stack trace above, the date is off by like 48 hours, so that would be a lot of clock skew.

+3
evansolomon on Mar 10, 2015

Just curious are you able to synchronize your NTP server/clock with AWS’ ? Several months ago, I encountered this error a couple of times, when creating signatures on my local machine. Hopefully this doc will be relevant: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html

+3
odeke-em on Mar 10, 2015

a restart of the VM helped me

+1
Kovaloff on Sep 1, 2018
Read more comments on GitHub
← aws-sdk-js: Intermittent "EC2 Metadata roleName request returned error" (EINVAL) on ECS Fargate
aws-sdk-js: 'JavaScript heap out of memory' when adding a dependency on the SDK to a small react-native app →