aws-vault: `aws-vault` suddenly started to report "UnauthorizedException: Session token not found or invalid" every morning

Since a few days I receive every morning a message from aws-vault that looks like the following:

aws-vault: error: exec: Failed to get credentials for xyz-sso: operation error SSO: GetRoleCredentials,
https response error StatusCode: 401, RequestID: , UnauthorizedException: Session token not found or invalid

I frequently use the command aws-vault exec xyz-sso --json to generate some new temporary credentials. Previously this worked flawlessly and opened the SSO page, if my session had expired, but now I only receive the above mentioned error.

To fix it I need to run aws-vault clear manually.

My aws-vault configuration:

AWS_VAULT_BACKEND=file
AWS_VAULT_FILE_PASSPHRASE=*some-secret-string*

I am using the latest release of AWS Vault

aws-vault --debug
2022/02/01 09:12:01 aws-vault 6.4.0

I have provided my .aws/config (redacted if necessary)

[profile xyz-sso]
sso_start_url = https://xyz.awsapps.com/start
sso_region = eu-central-1
sso_account_id = 1234567890
sso_role_name = AdministratorAccess
region = eu-central-1

I have provided the debug output using aws-vault --debug (redacted if necessary)

➜ aws-vault exec xyz-sso --json --debug
2022/02/01 09:13:05 aws-vault 6.4.0
2022/02/01 09:13:05 Loading config file /home/xyz/.aws/config
2022/02/01 09:13:05 Parsing config file /home/xyz/.aws/config
2022/02/01 09:13:05 [keyring] Considering backends: [file]
2022/02/01 09:13:05 Profile 'default' missing in config file
2022/02/01 09:13:05 [keyring] Expanded file dir to /home/xyz/.awsvault/keys/
2022/02/01 09:13:05 [keyring] Expanded file dir to /home/xyz/.awsvault/keys/
2022/02/01 09:13:05 [keyring] Expanded file dir to /home/xyz/.awsvault/keys/
2022/02/01 09:13:05 [keyring] Expanded file dir to /home/xyz/.awsvault/keys/
2022/02/01 09:13:05 [keyring] Expanded file dir to /home/xyz/.awsvault/keys/
2022/02/01 09:13:05 [keyring] Expanded file dir to /home/xyz/.awsvault/keys/
aws-vault: error: exec: Failed to get credentials for xyz-sso: operation error SSO: GetRoleCredentials, https response error StatusCode: 401, RequestID: , UnauthorizedException: Session token not found or invalid

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 18
Comments: 17 (9 by maintainers)

Most upvoted comments

Quick update here - Our AWS support case has confirmed that our ExpiresIn was returning 28800 (8 hours) and is now returning 604800 (7 days). This apparently is an internal configuration in the AWS service itself, which cannot be edited by customers. They’re investigating the issue, but I would recommend reaching out to AWS support about this issue.

endzyme on Feb 7, 2022

Quick update here - Our AWS support case has confirmed that our ExpiresIn was returning 28800 (8 hours) and is now returning 604800 (7 days). This apparently is an internal configuration in the AWS service itself, which cannot be edited by customers. They’re investigating the issue, but I would recommend reaching out to AWS support about this issue.

Hey thanks for digging into this. I started seeing this issue recently, and had only started to poke around to see what might have changed. I had the same suspicion as you - glancing at the token details they all claimed to expire in 7 days but were throwing errors sooner.

This morning I generated fresh SSO creds with aws-vault, but separately went through a manual aws sso-oidc create-token dance to get a test outside the aws-vault world. I’m curious to see if one or both tokens are still valid later tonight or tomorrow.

ajkerrigan on Feb 7, 2022

According to our AWS rep, this has been fixed on the AWS side. Anyone still experiencing the issue?

Same. AWS has confirmed that SSO was broken for a period and the rolled back some change set on their side. We’re also getting the correct values again from the API.

endzyme on Feb 10, 2022

Quick update here - Our AWS support case has confirmed that our ExpiresIn was returning 28800 (8 hours) and is now returning 604800 (7 days). This apparently is an internal configuration in the AWS service itself, which cannot be edited by customers. They’re investigating the issue, but I would recommend reaching out to AWS support about this issue.

Hey thanks for digging into this. I started seeing this issue recently, and had only started to poke around to see what might have changed. I had the same suspicion as you - glancing at the token details they all claimed to expire in 7 days but were throwing errors sooner.

This morning I generated fresh SSO creds with aws-vault, but separately went through a manual aws sso-oidc create-token dance to get a test outside the aws-vault world. I’m curious to see if one or both tokens are still valid later tonight or tomorrow.

To follow up on this, the response from my manual aws sso-oidc create-token with the AWS CLI this morning had the 604800 expiration:

{
    "accessToken": "eyJ...",
    "tokenType": "Bearer",
    "expiresIn": 604800
}

I was able to use that token successfully in a aws sso list-accounts call ~8 hours ago, and it’s failing now. Both tests were not using aws-vault. This is mostly a big +1 to what @endzyme said with it seeming like a backend issue on the AWS side, misreporting the expiration. I was wondering if this is even something aws-vault could gracefully handle, but I don’t think it is since the API error response is vague and would often suggest a non-retryable problem with the credentials.

ajkerrigan on Feb 13, 2022