terraform-provider-aws: Doesn't ask MFA token code when using assume_role with MFA required

Hi all!

In general, Terraform doesn’t currently support this sort of dynamic prompting for providers, and indeed the existing prompting features in Terraform are there primarily to help new users get started and we expect users to switch pretty early to using Terraform as a non-interactive, scripted tool, ideally running in a centrally-administered environment to avoid the need to sprawl various secrets across multiple user workstations.

Some users have reported success using a wrapper/helper script that calls sts:GetSessionToken to obtain temporary credentials in return for an MFA token, and then passes those temporary credentials to Terraform via environment variables, thus allowing that wrapper program to use whatever interactive means is appropriate (CLI prompt, web UI, etc) to collect that MFA token. This is, indeed, the methodology used by Terraform Enterprise for its AWS MFA support.

Managing authentication use-cases in wrapper scripts allows for other more advanced use-cases too, such as SAML authentication.

At this time we do not have plans to support interactive authentication to providers since it would require some significant changes to the provider model. We will probably look at this again in future in the context of some other breaking change to the provider model so that we can bundle multiple changes together, though as noted above we generally expect Terraform to be used non-interactively in some sort of automation once users pass the experimental phase, so any features we build in this area will need to also solve for a non-interactive workflow for that common case.

I would also like to see this work when simply using AWS_PROFILE env with the target profile configured to use a cross account role_arn and mfa_serial. In fact this would be the preferred way to use terraform for us.

$HOME/.aws/credentials:

[default]
aws_access_key_id = ***
aws_secret_access_key = ***

[subaccount]
role_arn = arn:aws:iam::***:role/OrganizationAccountAccessRole
source_profile = default
mfa_serial = arn:aws:iam::***:mfa/xxx@yyy.com

Terraform invocation:

AWS_PROFILE=subaccount terraform init

We can work around this using aws-vault or something similar, but I’d love to see the idea of interactive MFA authentication revisited (both for AWS and for other cloud providers where applicable).

I strongly disagree with @apparentlymart’s rationale from 2 years ago. While iterating on a TF configuration, I might terraform apply it against our dev account dozens of times over the course of a day. Doing that through our build/deploy pipeline is an obvious non-starter.

Of course we use that pipeline for deployment to environments after dev. So it’s not a matter of using TF interactively to manage our infrastructure. But we’re contractually obligated to enforce MFA even in our dev account. (And we use account federation with centralized IAM management so it’s not really possible to selectively enforce it anyway.) So as is, it makes our dev workflow more complex.

I can’t imagine our workflow or our requirements are particularly unusual, so it seems like a high value add for TF to support it.

I ended up getting this to work by requesting temporary credentials using MFA and saving them to ~/.aws/credentials

I suggest aws-vault for this purpose as it is much more convenient. aws-vault stores your long-term access credentials encrypted and will deal with exposing temporary AWS access credentials to your shell or application so you don’t have to. As a matter of fact, you can delete your credentials file and manage AWS API access entirely w/ aws-vault and named profiles defined in ~/.aws/config.

I’ll show an example that uses cross account IAM role delegation for executing terraform commands. OrganizationAccountAccessRole is the IAM role that authorized IAM users from an AWS master account can assume. The trust policy of the IAM role enforces MFA (set aws:MultiFactorAuthPresent boolean flag as described here)

The terraform AWS provider config:

provider "aws" {
  profile = "${var.aws_vault_profile}"
  region  = "${var.aws_region}"
  version = "~> 1.30.0"

  assume_role {
    role_arn = "arn:aws:iam::${var.aws_app_account_id}:role/OrganizationAccountAccessRole"
  }
}

Here is the corresponding ~/.aws/config file. It defines a terraform profile which uses MFA (the [profile ...] block is automatically created by aws-vault when adding a new profile - you just have to fill in the arn of the MFA device).

[default]
aws_region=eu-central-1

[profile terraform]
mfa_serial=arn:aws:iam::123456789012:mfa/Andreas

Then simply use aws-vault exec to wrap the terraform command - or set up an alias like so (note to change the backend depending on your system environment - this example uses the secret service API w/ keyring which works well for me on Ubuntu):

# always use aws-vault to wrap terraform command
alias terraform='aws-vault exec terraform --backend=secret-service -- terraform'

When you run terraform you will be automatically prompted for the access token:

➜  terraform-setup (master) ✗ terraform workspace list                                                          
Enter token for arn:aws:iam::123456789012:mfa/Andreas: 

Of course you can also use multiple profiles w/ aws-vault. There are more advanced configuration examples in the aws-vault docs.

My 2 cents:

• My workflow requires MFA even for CLI actions.
• I want to be able to use multiple providers with assume_role to a different role/account to manage different resources. It is especially important for account/organization management where multiple accounts are unavoidable.
• While I understand the point of non-interactive flow. I’d really like to see the possibility of using environment variable for specifying MFA value, something like this:

AWS_MFA_TOTP=123456 terraform apply

so if assumed role requires MFA we can work around it by using variable and if that role is not protected with MFA, it will not be used.

Thanks

Thank you, @dac73, I was not looking for the workaround.

I’m just adding one more voice to implement fully STS AssumeRole API call in Terraform. Not in the way it is done right now.

Currently TF supports only three input parameters:

    role_arn     = "arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME"
    session_name = "SESSION_NAME"
    external_id  = "EXTERNAL_ID"

while API call accepts more.:

• DurationSeconds
• Policy
• PolicyArns.member.N
• SerialNumber
• Tags.member.N
• TokenCode
• TransitiveTagKeys.member.N

Probably it should be a separate issue to add the rest of the parameters, but I would love to see there

• DurationSeconds
• SerialNumber
• TokenCode

Let the users deal with how they want to provide those values to TF, just add support for it.

I support requesting for MFA input. Profiles, etc are good for single provider, but thats not always the case. For example I have a need to build VPC peering between two accounts. Each of them requires MFA. Unless I setup some cross account trust in AWS - there is no way to achieve that using environment variables as all of them - will be for single account only.

There is no need to prompt for Terraform to support asking for a MFA value. If you run AWS_PROFILE=profile_that_requires_mfa aws iam list-users from the AWS CLI tool, and enter your token into the CLI tool, you will then find your temporary credentials cached in ~/.aws/cli/cache, so Terraform could just use that, and keep around the requirement that terraform is non-interactive.

This feels like a glaring omission. While being a “non-interactive, scripted tool” is a great primary goal, expecting everything to run in an automated pipeline is unrealistic, especially when terraform has commands such as destroy, force-unlock, import, refresh, state, taint, untaint, which only make sense to be invoked manually. Not to mention that the CLI already prompts for user input if a variable isn’t set (for example).

Occasionally you need an escape hatch to deal with edge cases, and how many people don’t have MFA enabled for live environments?

I’m forced to choose between turning off MFA (non-option), or adding third-party tooling/custom scripts (which increases maintenance, cognitive overhead, and learning curves for new developers). I’m trying to reduce the complexity of infrastructure deployments, not increase it.

Thankfully I’m thinking of this in advance and making sure there’s a solution in place for my client. I feel sorry for anyone that relies on their tf pipeline, and then ends up screwed because something unexpected happened in live, and they can’t fix it just because terraform can’t ask them for an MFA code.

https://awsu.me/

provider "aws" {
  profile = "prod"
  region  = "eu-west-1"
  version = "3.4.0"
}
terraform {
  required_version = "0.13.2"
  backend "s3" {
    profile        = "prod"
    region         = "eu-west-1"
    dynamodb_table = "terraform-locks"
    bucket         = "prod-terraform-states"
  }
}

Also you could check with

export AWS_SDK_LOAD_CONFIG=1
aws --profile prod sts get-caller-identity

it should use credentials from aws-vault and ask for mfa only once

or run aws-vault directly:

env AWS_SDK_LOAD_CONFIG=0 aws-vault --some-backend-options --prompt=osascript exec --region eu-west-1 --duration=1h root --json

If you want to run a terraform project that requires MFA, this is possible. Before running terraform, you can get a session token with MFA and then use its credentials to run terraform. This is explained here: https://aws.amazon.com/premiumsupport/knowledge-center/authenticate-mfa-cli/

The pseudo code is:

# Get an AWS session
# NOTE: MFA_CODE is a fresh CODE given by your MFA virtual device, like google authenticator. This code change so you can't automate this command
aws sts get-session-token --serial-number "${MFA_DEVICE_SERIAL_NUMBER}" --token-code "${MFA_CODE}"

# The credentials of the session are in the output of the previous command
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_SESSION_TOKEN=

# Run terraform
terraform apply

The following script allows you to get the MFA_DEVICE_SERIAL_NUMBER:

CURRENT_USER_ARN="$(aws sts get-caller-identity --query Arn --output text)
MFA_DEVICE_SERIAL_NUMBER="$(aws iam list-virtual-mfa-devices --query "VirtualMFADevices[?User.Arn=='${CURRENT_USER_ARN}'] | [0].SerialNumber" --output text)"

As said, you have to get the MFA_CODE manually.

For anyone who’s just looking for a simple escape hatch: There are plenty of programs that manage this problem better, as written above, but honestly, I’ve used this incredibly simple bash script for years, although I call it exec-in-aws-context instead of aws-assume-role. Basically, if you are using an AWS CLI profile that has a source profile configured, it sets up the AssumeRole-driven session using the AWS CLI, then grabs the temporary credentials and shoves them into the “basic” envvars (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY). In other words, the invoked program (e.g. Terraform) doesn’t need to support the full suite of AWS CLI profile features. From the program’s perspective, it’s just regular AWS credentials.

For example:

Place the script somewhere on your path, e.g. ~/bin/exec-in-aws-context. Make sure you’re using an AWS_PROFILE that has a source_profile in ~/.aws/config. Invoke Terraform like

exec-in-aws-context terraform apply

The script will call the AWS CLI and set up the temporary session (which will prompt you for MFA), then invoke terraform apply with the “simple” envvars instead of AWS_PROFILE.

I’ve actually found that little bash script very useful over the years. I prefer it over some I’ve seen because it just goes through the regular AWS CLI flow instead of re-implementing part(s) of it in bash. Additionally, it doesn’t mess with your existing shell session envvars.

Different accounts can be used with aws-vault as a credential_process for aws sdk, ie

~/.aws/config

[profile root]
mfa_serial = arn:aws:iam::XXX:mfa/olfway
credential_process = env AWS_SDK_LOAD_CONFIG=0 aws-vault --backend=pass --pass-prefix=aws-vault --prompt=osascript exec --region eu-west-1 --duration=1h root --json

[profile prod]
source_profile = root
parent_profile = root
role_arn = arn:aws:iam::YYY:role/Prod

[profile dev]
source_profile = root
parent_profile = root
role_arn = arn:aws:iam::ZZZ:role/Dev

and then run terraform as

export AWS_SDK_LOAD_CONFIG=1
terraform plan

@apparentlymart should I create separate issue for what I’m asking for, since this particular one is about dynamic prompting for MFA value?

For my purposes, I needed this feature for AWS org resource management across multiple accounts, so I can easily bootstrap new accounts. Another use case is for DevOps to manage resources across accounts originating from a “security” account (as described here). In all scenarios, MFA should be enabled for cross account access (internal requirement).

I ended up getting this to work by requesting temporary credentials using MFA and saving them to ~/.aws/credentials. With these credentials, the “MFA required” condition has been met and you’re free to use the assume_role block of the aws provider in Terraform:

provider "aws" {
  alias  = "dev_account"
  region = "${var.aws_region}"

  assume_role {
    role_arn = "${var.dev_account_target_role_arn}"
  }
}

I created a simple python cli to generate the creds: https://github.com/phil-hachey/aws-mfa-tool

Also here to support implementing MFA input. When implementing AWS CLI accounts, MFA is a must here.

@yn-academia If you don’t need this specifc feature, it doesn’t mean other people don’t need it as well.

This multiline script was made for single switch between main role and assumed role. one way, no way back. In terraform I want to work with multiple accounts easily. Which I do right now, but without MFA. This weakens our security policy. I want to make my work more secure and do not overcomplicate it at the same time.

By the way despite seatmates to the contrary in other tickets the AWS CLI does cache session tokens for assume role operations see https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html#cli-configure-role-mfa for details so an other option would be to allow terraform to use the cached credentials. if it used the cache then users could just do a simple aws sts get-caller-identity against the profile and they would be prompted for their MFA then terraform could used the cached credentials

I have been reading this ticket and would like to clarify if I understand it correctly.

Using this:

provider "aws" {
  region = "eu-west-2"
  access_key = var.aws_key
  secret_key = var.aws_secret
}

Terraform prompts for the key and secret which come straight away from a password management tool with encrypted storage.

In case the user has MFA enabled would it be possible that terraform also prompts for the TOTP code? Is there any intention to support this on terraform or is this not considered an interesting scenario?

Thanks! 😃

@tamsky I tried to add the MFA functionality into #1608 (we ideally wanted to be using MFA) but you are correct that the lack of an STS credentials cache essentially makes this unusable for the end user (repeated prompts for MFA token, and as tokens can only be used once, you have to wait until a new token is generated each time).

Since the inability to assume roles defined in the profile file was hurting us more, I elected to pursue that fix for now (still hasn’t quite been released, see https://github.com/hashicorp/terraform/pull/16661), though I might return to the MFA issue when I next have some spare capacity (if someone else hasn’t stepped in to solve it).

(the original PR https://github.com/hashicorp/terraform/pull/11734 that I resurrected to form #1608, had some discussion on MFA which led to me experimenting and finding the limitations)