terraform-provider-aws: Unable to invoke Lambda with environment variables due to KMS AccessDeniedException

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave “+1” or “me too” comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

  • Terraform v0.11.10
  • provider.aws v1.42.0

Affected Resource(s)

  • aws_lambda_function

Terraform Configuration Files

resource "aws_iam_role" "myrole" {
  name = "terraform-kms-test"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "basic_exec" {
  role       = "${aws_iam_role.myrole.name}"
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

resource "aws_lambda_function" "myfunction" {
  filename         = "build.zip"
  function_name    = "terraform-kms-test"
  role             = "${aws_iam_role.myrole.arn}"
  handler          = "index.handler"
  source_code_hash = "${base64sha256(file("build.zip"))}"
  runtime          = "nodejs8.10"

  environment {
    variables {
      MY_CONFIG = "config value"
    }
  }
}

Expected Behavior

Function using environment variables should be invocable after it’s role name is changed.

Actual Behavior

  • On initial deployment, the function is able to be invoked without any errors
  • But if you change the IAM role name and rerun terraform apply, invoking the function returns the following error:

Calling the invoke API action failed with this message: Lambda was unable to decrypt the environment variables because KMS access was denied. Please check the function’s KMS key settings. KMS Exception: AccessDeniedExceptionKMS Message: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.

Steps to Reproduce

  1. terraform apply
  2. Change aws_iam_role name to something different
  3. terraform apply

References

  • Seems to be a related issue #4633

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 46
  • Comments: 16 (5 by maintainers)

Most upvoted comments

It has been over a year since anyone commented on this issue. I will be working to repro this and closing it if AWS and/or AWS provider changes have fixed the problem in the interim. Please let us know if you continue to face problems with this!

This is a great explanation from Paul Allen on the problem:

When you provide environment variables to a Lambda function, they’re encrypted using a KMS key. Either a customer-managed key that you provide or an AWS managed default key (with the alias aws/lambda). When environment variables are first defined, if the default key is used then Lambda creates a grant on that key letting the execution role use it for decrypting the environment variables.

But, if that role is deleted and then re-created, the grant is no longer valid! This is the same as other resource-based policies when the principal is removed but this is special because we never actually explicitly created that grant ourselves. This means the function will start failing for no apparent reason.

Would this not be as simple as adding the role for the lambda to the depends_on attribute in the Lambda, so you make sure that the Role is created before the lambda?

Thank you for taking the time to consider this.

I think updating the documentation to explicitly mention this issue might be good. Would it also be possible to recommend including the IAM role name in the source_code_hash? I’m not sure if a function update is enough to fix the issue though, but something like this. Is that what you mean by publish?

source_code_hash = "${base64sha256(file("build.zip"))}-${aws_iam_role.myrole.name}"