amplify-js: Clock Drift/Skew code does not handle all skew error scenarios
Describe the bug See my comments https://github.com/aws-amplify/amplify-js/issues/2014#issuecomment-668914147 onwards. DateUtils’ isClockSkewError only checks for one kind of error type related to bad clock, but there’s actually many possible error types. See https://github.com/aws/aws-sdk-js-v3/blob/47770fa493c3405f193069cd18319882529ff484/packages/service-error-classification/src/constants.ts#L8
To Reproduce Steps to reproduce the behavior:
- Setup API Gateway via amplify cli
- Set your computer time 10 minutes early or more
- Have your code sign into cognito, then make a REST API call to API Gateway
- You’ll get error back with ‘x-amzn-errortype’ of “InvalidSignatureException” and now retry will occur with proper clock offset.
Expected behavior Retry would occur with correct clock offset. For some reason your code expects “BadRequestException” error to be returned but in my case “InvalidSignatureException” is returned.
Screenshots See https://github.com/aws-amplify/amplify-js/issues/2014#issuecomment-668914147 for example error. In this case error came from cloudfront in an edge-optimized API Gateway, but same error comes back from regional API Gateway too.
What is Configured? If applicable, please provide what is configured for Amplify CLI:
- Which resources do you have configured?
- If applicable, please provide your
aws-exportsfile:
"aws_project_region": "us-east-1", "aws_cloud_logic_custom": [ { "name": "EssRestAPI", "endpoint": "https://....execute-api.us-east-1.amazonaws.com/...", "region": "us-east-1" } ], "aws_appsync_graphqlEndpoint": "https://....appsync-api.us-east-1.amazonaws.com/graphql", "aws_appsync_region": "us-east-1", "aws_appsync_authenticationType": "AMAZON_COGNITO_USER_POOLS", "aws_cognito_identity_pool_id": "us-east-1:8...", "aws_cognito_region": "us-east-1", "aws_user_pools_id": "us-east-1_g0...", "aws_user_pools_web_client_id": "1e1rhg332...", "oauth": {} - If applicable, please provide your
};```
- If applicable, please provide your manual configuration example:
Auth.configure({
authenticationFlowType: 'USER_PASSWORD_AUTH'
});```
System: OS: macOS 10.15.6 CPU: (12) x64 Intel® Core™ i7-8850H CPU @ 2.60GHz Memory: 104.32 MB / 32.00 GB Shell: 5.7.1 - /bin/zsh Binaries: Node: 13.14.0 - ~/.nvm/versions/node/v13.14.0/bin/node Yarn: 1.22.0 - /usr/local/bin/yarn npm: 6.14.4 - ~/.nvm/versions/node/v13.14.0/bin/npm Watchman: 4.9.0 - /usr/local/bin/watchman Browsers: Chrome: 84.0.4147.105 Firefox: 78.0.2 Safari: 13.1.2 npmPackages: @angular-builders/custom-webpack: ^10.0.0 => 10.0.0 @angular-devkit/build-angular: ^0.1000.4 => 0.1000.4 @angular/animations: ~10.0.5 => 10.0.5 @angular/cdk: ^10.1.0 => 10.1.0 @angular/cli: ~10.0.4 => 10.0.4 @angular/common: ~10.0.5 => 10.0.5 @angular/compiler: ~10.0.5 => 10.0.5 @angular/compiler-cli: ^10.0.5 => 10.0.5 @angular/core: ~10.0.5 => 10.0.5 @angular/elements: ~10.0.5 => 10.0.5 @angular/forms: ~10.0.5 => 10.0.5 @angular/material: ^10.1.0 => 10.1.0 @angular/platform-browser: ~10.0.5 => 10.0.5 @angular/platform-browser-dynamic: ~10.0.5 => 10.0.5 @angular/router: ~10.0.5 => 10.0.5 @aws-amplify/api: ^3.1.20 => 3.1.20 @aws-amplify/api-rest: ^1.1.3 => 1.1.3 @aws-amplify/auth: ^3.3.2 => 3.3.2 @ng-select/ng-option-highlight: 0.0.5 => 0.0.5 @ng-select/ng-select: ^4.0.0 => 4.0.4 @ngneat/until-destroy: ^7.1.6 => 7.3.2 @stripe/stripe-js: ^1.8.0 => 1.8.0 @types/jasmine: ~3.5.0 => 3.5.11 @types/jasminewd2: ~2.0.3 => 2.0.8 @types/node: 14.0.4 => 14.0.4 @types/node-zendesk: ^1.4.0 => 1.4.0 array-flat-polyfill: ^1.0.1 => 1.0.1 axios: ^0.19.2 => 0.19.2 body-parser: ^1.19.0 => 1.19.0 codelyzer: ^6.0.0 => 6.0.0 document-register-element: ^1.7.2 => 1.14.3 folder-hash: ^3.3.0 => 3.3.2 fs-extra: ^9.0.0 => 9.0.1 jasmine-core: ~3.5.0 => 3.5.0 jasmine-spec-reporter: ~5.0.0 => 5.0.2 karma: ~5.0.0 => 5.0.9 karma-chrome-launcher: ~3.1.0 => 3.1.0 karma-coverage-istanbul-reporter: ~3.0.2 => 3.0.3 karma-jasmine: ~3.3.0 => 3.3.1 karma-jasmine-html-reporter: ^1.5.0 => 1.5.4 libphonenumber-js: ^1.7.51 => 1.7.54 lodash-es: ^4.17.15 => 4.17.15 lodash.merge: ^4.6.2 => 4.6.2 lodash.snakecase: ^4.1.1 => 4.1.1 ng2-pdf-viewer: ^6.3.2 => 6.3.2 ngx-build-plus: ^10.1.1 => 10.1.1 ngx-stripe: ^10.1.1 => 10.1.1 node-sass: ^4.13.1 => 4.14.1 object.fromentries: ^2.0.2 => 2.0.2 postcode-validator: ^3.1.0 => 3.1.0 prettier: ^1.19.1 => 1.19.1 prettier-tslint: ^0.4.2 => 0.4.2 prompt-sync: ^4.2.0 => 4.2.0 protractor: ~7.0.0 => 7.0.0 rxjs: ~6.6.0 => 6.6.0 stripe: ^8.25.0 => 8.72.0 tmp: ^0.1.0 => 0.1.0 ts-node: ~8.3.0 => 8.3.0 tslib: ^1.13.0 => 1.13.0 tslint: ~6.1.0 => 6.1.2 typescript: ~3.9.6 => 3.9.6 webpack-bundle-analyzer: ^3.8.0 => 3.8.0 which: ^2.0.2 => 2.0.2 zone.js: ~0.10.3 => 0.10.3 npmGlobalPackages: @angular/cli: 10.0.3 @aws-amplify/cli: 4.26.0 local-web-server: 4.2.1 node-gyp: 7.0.0 npm: 6.14.4```
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 2
- Comments: 29 (10 by maintainers)
Hi all, I spent a good chunk of today trying to debug this issue. So, It looks like this is not a library issue but is instead an API Gateway / CLI issue.
The way we correct any clock skew error, as pointed out by @hisham is as follow
Dateheader from the error response and callDateUtils.setClockOffsetby calculating the difference between the request time & the server response timesee: https://github.com/aws-amplify/amplify-js/blob/8448b4fe1f995f0d6056bcb46dd0fb2023fb197d/packages/api-rest/src/RestClient.ts#L194-L225
This part actually works as intended, the problem is, if we look at the screenshot provided, the CORS headers are missing (which is also the problem I encountered when I repro-ed the problem). Javascript won’t be able to access the error response, therefore the library couldn’t identify it as a clock skew error. Therefore the solution is to actually make API gateway respond with the CORS header for 4XX errors.
You can follow the walkthrough below for a temporary workaround:
Dateandx-amzn-errortypeheaders Here’s an example, (change ‘my-domain.com’ to a value that fits your use case)Demo:
https://user-images.githubusercontent.com/40404256/143974566-429a7c05-d989-485e-8ad1-8a2e41e1981f.mov You can see the first request fails, then it was able to automatically retry and succeed on the second call.
I will look to get a PR on the CLI side to add this setting by default.
https://github.com/aws-amplify/amplify-cli/pull/9158 I have submitted a PR on the CLI side to add this behavior as a default 4xx / 5xx gateway response
@hisham Thanks. I will look into the cognito/rest issue.
The solution in https://github.com/aws-amplify/amplify-js/issues/6494#issuecomment-669019819 does not seem to work for S3 PUT uploads (
Storage.put(`${cred.identityId}/${randomFilename}`, file, {...).Looks like
systemClockOffsetneeds to be set but there doesn’t seem to be any way to do this from Amplify.https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Config.html#systemClockOffset-property
Parsing the XML response and patching
Date.nowworks but is extremely hackySeems like the clock offset should be set automatically in this case by Amplify and this is a bug.
So it seems like the issue is that there is a middleware for the s3client that can automatically adjust
systemClockoffset(see: awsAuthMiddleware)if the server time and client time mismatches. However it’s not invoked as intended (it’s patched in a very recent release, but our team is not upgrading to that atm). I wrote a custom middleware that should solve this issue.I was able to repro the problem. I will track down the root cause and give an update here when I do 😄
Ours customers are facing this issue too. We have used one workaround to set clockskew on API calls. Is there any similar workaround for storage.get() calls?
Multiple customers of ours are hitting this issue on production. And there is still no patch released for this bug even after one year! Amazing and unbelievable.