aws-sam-cli: `sam build --parameter-overrides` fails to override

Description

sam build --parameter-overrides fails to override SAM template yaml Parameters

Setup

Ubuntu 18.04.3 x86_64
Python 3.7.3
aws-cli/1.16.265 Python/3.6.8 Linux/5.0.0-32-generic botocore/1.13.1
SAM CLI, version 0.23.0
empty requirements.txt
arbitrary app.py
below parampoof.yaml

parampoof.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: |
  Repo of param override failing to override

Parameters:
  GitTag:
    Description: description field on lambda version
    Type: String
    Default: defaultgittag

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .
      AutoPublishAlias: test
      VersionDescription: !Ref GitTag
      Handler: app.lambda_handler
      Runtime: python3.7
      Timeout: 10

Steps to reproduce

sam build \
    --template ./parampoof.yaml \
    --manifest requirements.txt \
    --parameter-overrides "ParameterKey=GitTag,ParameterValue=aabbccdd"
grep aabbccdd .aws-sam/build/template.yaml

Observed result

First…

Building resource 'MyFunction'
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>

Then…

Nothing. grep finds no value of aabbccdd in the file

Expected result

grep should find the value aabbccdd in the file. Likely in Parameters: section by overriding/replacing the default value defaultgittag. This is needed because the next steps sam package and sam deploy in the workflow depend on this .aws-sam/build/template.yaml transformed output of sam build.

Debug output of `sam build`

Using SAM Template at **redacted**/parampoof.yaml
Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
'build' command is called
Collected default values for parameters: {'GitTag': 'defaultgittag'}
1 resources found in the template
Found Serverless function with name='MyFunction' and CodeUri='.'
Building resource 'MyFunction'
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r **redacted**/requirements.txt --dest /tmp/tmpk1rvadh4
Full dependency closure: set()
initial compatible: set()
initial incompatible: set()
Downloading missing wheels: set()
compatible wheels after second download pass: set()
Build missing wheels from sdists (C compiling True): set()
compatible after building wheels (no C compiling): set()
Build missing wheels from sdists (C compiling False): set()
compatible after building wheels (C compiling): set()
Final compatible: set()
Final incompatible: set()
Final missing wheels: set()
PythonPipBuilder:ResolveDependencies succeeded
Running PythonPipBuilder:CopySource
PythonPipBuilder:CopySource succeeded

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>
    
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 523, 'exitReason': 'success', 'exitCode': 0, 'requestId': '9f3a7fc1-3397-48a2-afe0-22848b8743b4', 'installationId': 'c648e689-e57c-4a3c-a26f-ca6d1fd13041', 'sessionId': 'd5aced2b-51ec-4288-aab8-a47317da031d', 'executionEnvironment': 'CLI', 'pyversion': '3.7.3', 'samcliVersion': '0.23.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

Workaround

Don’t use sam build --parameter-overrides. Instead use sed to search/replace the base template.yaml for a known value to replace/override.

https://github.com/awslabs/aws-sam-cli/issues/572

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 15 (5 by maintainers)

Most upvoted comments

This feature is extremely confusing, my expectations were that sam build is using the parameter overrides flag to create an output artifact that I can use for execution or deployment purposes.

+11

rpstreef on May 1, 2020

@diablodale You have a really good point. sam build should capture the flags provided earlier and pass it along to the next commands. Your mental model makes sense and in fact is more powerful IMO.

I think #1503 is going to enable this capability. Please take a look at the design and provide comments. You have an interesting perspective and I think we can all benefit from it.

Closing this Issue because sam is working as expected

sanathkr on Nov 11, 2019

Ooo, that’s super-subtle. What just now clicked for me is ‘in memory’. Personally, I think the ‘in-memory’ of sam build should include mutation of Parameters:. Just like it does for the CodeUri, and other params (I’ve already see sam dramatically reformats the top-level Description).

Production CloudFormation mutates things ‘on-disk’. Yes, it can receive a param that overrides the CloudFormation template default, but when it writes ‘on-disk’ to an EC2 instance, CloudWatch rule, or any other created resource that references that overridden param…that’s on-disk. It’s not just in-memory. And those on-disk mutations persist and may further propagate if the created resources do creation of their own.

Since the sam workflow is a series of steps (at minimum package and deploy are required because of the S3 package location mutation, build for safer dependency management), I suggest that the mutations of sam includes param-overrides of the top-level Parameters:. I acknowledge that it is a bit odd to mutate the default value. So perhaps an approach like…

Spec new param Parameters.MyParam.RuntimeValue
Update sam code so that if that RuntimeValue is present in the template, that it uses that value instead of the Parameters.MyParam.Default value.
Update sam code to add Parameters.MyParam.RuntimeValue for any --parameter-overrides provided on the cli.
The above allows sam build, sam local invoke, or sam deploy with --parameter-overrides "ParameterKey=MyParam,ParameterValue=value123". The first two commands are the updated sam code that writes the new RuntimeValue param and the 3rd you tell me already works with its alias to aws cloudformation deploy.

That brings the behavior of --parameter-overrides more in alignment and behavior for local and AWS deployments. This allows for scenarios like:

Environment variables of a lambda function based on Parameters: will now work both locally and on AWS. Without mutating the Parameters: with a value passed with --parameter-overrides, the local testing docker container won’t receive the overridden value -> can’t adequately test.
Dynamic git commit tagging of a build->package->deploy using the Serverless::Function.Prop.VersionDescription as a reference to an overriden param. It is not uncommon for a running service to know its own git commit.

diablodale on Oct 30, 2019

I’m having the same problem as @Manouchehri with specifying the architecture to be built with sam build. My goal is to test my lambda locally using sam build and sam local invoke, but I don’t want people on my team to have to manually modify the Architectures property of the lambda depending on their machine. Is there any way to do this?

anhvungoc21 on Sep 21, 2023

Note to those who do use CodePipeline’s built in CloudFormation deploy option who were tripped up by this: Look in the advanced options in the Deploy stage. There is a JSON input field for parameter overrides that does what you want it to (if you don’t resort to scripting the deploy).

shane-js on May 17, 2020

I don’t think mutating and writing it to disk is the right thing to do for build. --parameter-overrides is more of an escape hatch to get the template into a state that the CLI can understand than something you should be using regularly. Ideally, build doesn’t need for customers to use --parameter-overrides but right now there are cases for it.

Doing what you are suggesting, isn’t ideal for how many customers setup pipelines. The recommended/best practice is to build -> package to produce a single template with all artifacts, then deploy that to regions/accounts. In these cases, you do not want build to set defaults because your template becomes very narrow. Yes, you could then do many builds but the CloudFormation way to do this is through Parameters and then overriding them at deploy time.

The path forward here would be to only use --parameter-overrides if you need to on build. For invoke, pass --parameter-overrides to get the template to a state you want to test. This is the current pattern which is all modeled on how CloudFormation APIs work.

jfuss on Oct 30, 2019