aws-cdk: ‼️ NOTICE: last modified date does not match with the last modified date of the retrieved parameters

Please add your +1 👍 to let us know you have encountered this

Status: INVESTIGATING

Overview:

Cloudformation recently deployed a change to allow dynamic references to ssm parameters within templates without needing to provide a version. This allows always referencing the latest value of a parameter within your template. This change had a bug that caused change set creation to fail when the parameter value was updated.

This breakage was caused by the references resolving to a version of the parameter that include a timestamp. Upon changeset creation, if the parameter has been updated, cloudformation sees the parameter name and the previously resolved timestamped version as different values and fails to update. They have since changed logic so that the strings resolved do not include timestamps, however if a template was successfully deployed during the time that the timestamp was being appended, a stack cannot be updated via changeset while referencing a dynamic parameter.

UPDATE:

It is clear that removing all dynamic ssm references doesn’t cause users to get unstuck. Users have to manually deploy their stacks without a changeset to get past this. That being the case, for users with lots of stacks updating each manually is a pain. We are considering adding an option to the CLI to skip a changeset if this error occurs during deploy.

Complete Error Message:

Parameters: [PARAMETER_NAME] last modified date does not match with the last modified date of the retrieved parameters.

Workaround:

Attempt the following steps in order as a workaround:

Upgrade both your CLI and all your @aws-cdk/xyz modules to v1.103.0 - This removes dynamic SSM references and goes back to referencing them via CFN parameters.
Delete your stack and redeploy - If you are unable to do so because this is a production stack, and upgrading to v1.103 didn’t fix the issue, please comment here with the following information and try number 3.
- The name of the parameter causing the failure
- An example of how you are referencing that parameter in code, IE through an enum value, manually specifying the param name etc.
Run cdk synth and then manually deploy this stack using the the aws cli or cfn console - This is not an ideal workaround for users who may have a lot of different stacks that need updating but in the meantime, this can unblock you if you need a stack deployed. We are working on figuring out another option that will allow skipping changeset creation via the aws cli.

Solution:

This issue only shows up for users who deployed stacks during a short period of time before cloudformation fixed some of these bugs. Because of that, it is hard for us to test ourselves if removal of dynamic param references (https://github.com/aws/aws-cdk/pull/14527) solves this. If it doesn’t, another solution mentioned here is to update your stack without creating a changeset. We may be able to provide an option in the CLI to skip a changeset if this error is encountered. Since we don’t have broken stacks to test this with, please let us know if this is or an additional fix is needed.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 102
Comments: 40 (19 by maintainers)

Most upvoted comments

I managed to find a fix for this with @Shogan . It is a “surgical” workaround , but will allow you to get the stack in an “updateable” state using CDK without recreating it.

get the new ssm parameter last modified date time stamp from aws console or using ssm cli get-parameter
convert the time into Unix format including milliseconds . I used https://time.lol for this. You should end up with something that looks like “1616080117000”
In AWS Console cloudformation screen , open the stack , click on “edit in designer”
the lower pan of the designed contains the template in YAML format
find and replace all instances of “”{{resolve:ssm:/ssm_parameter_name:1:OLD_TIME_STAMP}}" with “”{{resolve:ssm:/ssm_parameter_name:1:NEW UNIX_TIME_STAMP}}"
Save the template to a local YAML file to your computer
Close the Designer
From CloudFormation Console again , Click on Update Stack , Replace Existing stack , upload your template file that you downloaded from the designer.
Follow the wizard until the stack update is completed.

If all goes well , you should be able to deploy the stack using cdk going forward.

+19

AbdoNile on Jun 28, 2021

Also getting this issue for obtainDefaultFluentBitECRImage from @aws-cdk/aws-ecs… This is really frustrating.

FlorinAsavoaie on May 27, 2021

Thank you @andreialecu for investigating!

Anyone else that is still blocked and having trouble with the workarounds please keep letting us know.

MrArnoldPalmer on May 14, 2021

Updated the issue to include manually deploying your stack without a changeset as a workaround while we figure out if/how to allow users to do this via CDK. For users who have CI/CD deploying a high volume of stacks this obviously isn’t ideal and we are figuring out if we want to add a way for the CDK CLI to deploy without a changeset and if that is good enough. If people are using the codepipeline stack deploy actions with a changeset this won’t cover them and I’m not sure if there is another way to let them easily workaround this.

MrArnoldPalmer on May 13, 2021

We are running into this after upgrading to 1.103.0 today, seemingly caused by usage of EcsOptimizedImage.amazonLinux2. We’ve run several deploys over the last few days with no errors until we upgraded to 1.103.0.

Error is FAILED, Parameters: [ssm:/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id:59:1619744777312] last modified date does not match with the last modified date of the retrieved parameters.

Updating the stack with a 1.103.0 generated template fixed the issue for us.

sundeqvist on May 14, 2021

I did something similar to @AbdoNile but simply replaced the parameters with the actual values. After that succeeded, I deployed again through CDK and it worked.

FlorinAsavoaie on Jun 28, 2021

@clemans Thanks! Your reply made me realize I hadn’t tried uploading via the AWS Management Console, only command line. The upload via the console worked! Saves me from deleting / recreating stacks. Appreciate the help!

kastningj on May 18, 2021

I managed to go long time without this issue but it came back today. I upgraded from 1.101.0 to 1.104.0 but it did not fix it. This is for an ECS cluster and the parameter is implicit in the construct as opposed to anything I create in my own code: Failed to create ChangeSet cdk-deploy-change-set on alfresco-app-dev: FAILED, Parameters: [ssm:/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id:60:1620247830216] last modified date does not match with the last modified date of the retrieved parameters.

I used the workaround of cdk synth and manually uploaded the template via the console and that is working.

cseickel on May 17, 2021

@MrArnoldPalmer I can confirm that manually updating the stack using the AWS console did work.

andreialecu on May 14, 2021

Oops. My screenshot does show that it was actually previously deployed on the 5th of May, probably using 1.100.0. NOT in March. Sorry about that. Had a mental hiccup, I thought 05.05 was today 🤦🏻‍♂️

andreialecu on May 14, 2021

Also ran into this with the ec2.BastionHostLinux construct. Since we only use this for connecting to resources within our vpc and it’s not actively receiving usage all the time, I was able to comment it out, deploy, add it back, deploy and the issue went away.

TLadd on May 13, 2021

As I’ve mentioned in https://github.com/aws/aws-cdk/issues/14467#issuecomment-833323448, I encounter this in the ec2.BastionHostLinux construct. Still present after an update to 1.103.0. Stack is in eu-west-1 region.

The reference in the stack is simple:

this.bastion = new ec2.BastionHostLinux(this, `Bastion`, {
      vpc: props.vpc,
      instanceName: `${props.name}-bastion`,
      securityGroup: bastionSg,
      subnetSelection: {
        subnetType: ec2.SubnetType.PUBLIC,
      },
    });

Diff for the stack:

cdk diff vpc

Stack vpc (vpc)
Parameters
[+] Parameter SsmParameterValue:--aws--service--ami-amazon-linux-latest--amzn2-ami-hvm-x86_64-gp2:C96584B6-F00A-464E-AD19-53AFF4B05118.Parameter SsmParameterValueawsserviceamiamazonlinuxlatestamzn2amihvmx8664gp2C96584B6F00A464EAD1953AFF4B05118Parameter: {"Type":"AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>","Default":"/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2"}

Resources
[~] AWS::EC2::Instance Bastion/Bastion/Resource BastionBastion34649AD2 replace
 └─ [~] ImageId (requires replacement)
     └─ @@ -1,1 +1,3 @@
        [-] "{{resolve:ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2}}"
        [+] {
        [+]   "Ref": "SsmParameterValueawsserviceamiamazonlinuxlatestamzn2amihvmx8664gp2C96584B6F00A464EAD1953AFF4B05118Parameter"
        [+] }
[~] AWS::EC2::EIP Bastion/BastionEIP BastionBastionEIP54D072EC may be replaced
 └─ [~] InstanceId (may cause replacement)
     └─ [~] .Ref:
         ├─ [-] BastionBastion34649AD2
         └─ [+] BastionBastion34649AD2 (replaced)

Fails with:

vpc (vpc): deploying...
vpc: creating CloudFormation changeset...

 ❌  vpc (vpc) failed: Error: Failed to create ChangeSet cdk-deploy-change-set on -vpc: FAILED, Parameters: [ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2:45:1619654085749] last modified date does not match with the last modified date of the retrieved parameters.
    at /home/malanius/.nvm/versions/node/v16.0.0/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:227:11
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at waitFor (/home/malanius/.nvm/versions/node/v16.0.0/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:188:20)
    at Object.waitForChangeSet (/home/malanius/.nvm/versions/node/v16.0.0/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:213:15)
    at Object.deployStack (/home/malanius/.nvm/versions/node/v16.0.0/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:260:32)
    at CdkToolkit.deploy (/home/malanius/.nvm/versions/node/v16.0.0/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:180:24)
    at initCommandLine (/home/malanius/.nvm/versions/node/v16.0.0/lib/node_modules/aws-cdk/bin/cdk.ts:209:9)
Failed to create ChangeSet cdk-deploy-change-set on -vpc: FAILED, Parameters: [ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2:45:1619654085749] last modified date does not match with the last modified date of the retrieved parameters.

The workaround with manually updating stack from generated template by 1.103.0 worked for me. Probably just temporarily deleting the offending bastion resource could work as well, but this way I at least avoided updating all the stacks where Bastion is mentioned.

Malanius on May 12, 2021

Mentioned this in the other issue but just a friendly reminder so it doesn’t fall through the cracks: I’m happy to try this on the 2.0 line (I recently ported to from 1.x to 2 RC1) but I’m waiting for the 2.0 release to come through (last one was ~12 days ago).

blaenk on May 11, 2021

We have this issue after upgrading to v1.103, but haven’t encountered it before. We like to keep our dependencies up to date, so we have updated to every minor release in recent history.

The change set that failed has this parameter:

SsmParameterValueawsserviceecsoptimizedamiamazonlinux2recommendedimageidC96584B6F00A464EAD1953AFF4B05118Parameter:
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: /aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id

That parameter is referenced as the ImageId of autoscaling group launch configuration. In the template currently deployed with CDK v1.102, ImageId is defined as "{{resolve:ssm:/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id}}".

Urokhtor on May 11, 2021