pulumi-aws: Inconsistent behavior and unable to update `aws.ssm.Document`

What happened?

(context from customer)

I have a simple Pulumi app that deploys an AWS SSM Document (Automation Runbook). When the Runbook doesn’t exist, it is created as expected ✅.

However, if the YAML data is changed, then Pulumi fails to update the document ❌. The AWS API returns the following message.

error: 1 error occurred:
        * updating urn:pulumi:case-3012::zendesk::aws:ssm/document:Document::nodeBuildRunbook-doc: 1 error occurred:
        * updating SSM Document (nodeBuildRunbook): ValidationException: 1 validation error detected: Value at 'documentVersion' failed to satisfy constraint: Member must satisfy regular expression pattern: ([$]LATEST|[$]DEFAULT|[$]APPROVED|[$]PENDING_REVIEW|^[1-9][0-9]*$)
        status code: 400, request id: bad1c566-66aa-40cd-b474-17076ccff4f4

Furthermore, running the Pulumi app a second time, Pulumi returns no error. However, the SSM Document isn’t updated either ‼ (I checked in the AWS Console). This makes things really complicated.

I’ve attached the verbose logs (-v=9) in case there’s any relevant information.

logs-error.txt logs-no-error.txt

Finally, to remove some possible confusion. The YAML data may contain a documentVersion field (under inputs) but setting or removing this field doesn’t change anything.

Expected Behavior

1. The Automation Runbook Document should be updated based on the updated YAML data.
2. If the update fails, then Pulumi should consistently return an error.

Steps to reproduce

1. Create a new Pulumi app (files provided below).
2. Run the pulumi app pulumi up. The deployment should finish as expected. A new SSM document is visible in the AWS Console.
3. Update the runbook.yaml and change a description field.
4. Run pulumi up. The command ends with the error shown above ❌.
5. Run pulumi up a second time (no changes). The command completes and it seems the Document was updated but the console shows the same previous content ‼.

Output of pulumi about

CLI          
Version      3.69.0
Go Version   go1.20.4
Go Compiler  gc

Plugins
NAME    VERSION
aws     5.41.0
awsx    1.0.2
docker  3.6.1
nodejs  unknown

Host     
OS       debian
Version  11.7
Arch     x86_64

This project is written in nodejs: executable='/usr/local/bin/node' version='v18.16.0'

Current Stack: menfin/zendesk/case-3012

TYPE                           URN
pulumi:pulumi:Stack            urn:pulumi:case-3012::zendesk::pulumi:pulumi:Stack::zendesk-case-3012
pulumi:providers:aws           urn:pulumi:case-3012::zendesk::pulumi:providers:aws::default_5_41_0
aws:ssm/parameter:Parameter    urn:pulumi:case-3012::zendesk::aws:ssm/parameter:Parameter::newRelicWinServiceConfig
aws:iam/role:Role              urn:pulumi:case-3012::zendesk::aws:iam/role:Role::ssmAutomation-role
aws:ssm/parameter:Parameter    urn:pulumi:case-3012::zendesk::aws:ssm/parameter:Parameter::newRelicWinEventConfig
aws:ssm/document:Document      urn:pulumi:case-3012::zendesk::aws:ssm/document:Document::nodeBuildRunbook-doc
aws:iam/rolePolicy:RolePolicy  urn:pulumi:case-3012::zendesk::aws:iam/rolePolicy:RolePolicy::eventBridge-nodeBuildRunbook-policy


Found no pending operations associated with case-3012

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/aureq
User           aureq
Organizations  aureq, team-ce, menfin, menfin-team, demo, pulumi

Dependencies:
NAME            VERSION
@pulumi/pulumi  3.69.0
@types/node     16.18.34
handlebars      4.7.7
@pulumi/aws     5.41.0
@pulumi/awsx    1.0.2

Pulumi locates its logs in /tmp by default

Additional context

package.json

{
    "name": "l",
    "main": "index.ts",
    "devDependencies": {
        "@types/node": "^16"
    },
    "dependencies": {
        "@pulumi/aws": "^5.0.0",
        "@pulumi/awsx": "^1.0.0",
        "@pulumi/pulumi": "^3.0.0",
        "handlebars": "^4.7.7"
    }
}

index.ts

import * as aws from "@pulumi/aws";
import * as handlebars from 'handlebars';
import * as fs from 'fs';
import * as path from 'path';
import * as pulumi from '@pulumi/pulumi';

/**
 * Function to Handle All the resources need for instances entering Warm Pool
 * we well as Launching directly into the Active Pool.
 *
 * @param instanceLaunchLch
 * @param ssmAutomationRoleArn
 * @param eventBridgeRole
 * @param opts
 */

const ssmAutomationRole = new aws.iam.Role('ssmAutomation-role', {
    name: 'ssmAutomation-role',
    assumeRolePolicy: JSON.stringify({
        Version: '2012-10-17',
        Statement: [{
            Action: 'sts:AssumeRole',
            Effect: 'Allow',
            Sid: '',
            Principal: {
                Service: 'ssm.amazonaws.com',
            },
        }],
    }),
});


export const runbook = (ssmAutomationRole: aws.iam.Role) => {
    // Create File Object
    const runbookFile = handlebars.compile(
        fs.readFileSync(path.join(__dirname, './runbook.yml'), 'utf8'),
    );

    // Create SSM Parameter(s) for NewRelic Config
    const newRelicWinEventConfigParam = new aws.ssm.Parameter('newRelicWinEventConfig', {
        name: '/platform/newRelicWinEventConfig',
        type: 'String',
        value: 'test new relic event config value'
    });

    const newRelicWinServiceConfigParam = new aws.ssm.Parameter('newRelicWinServiceConfig', {
      name: '/platform/newRelicWinServiceConfig',
      type: 'String',
      value: 'test new relic service config value'
    });

  // Inject Parameters into Template
    const nodeBuildRunbookTemplate = pulumi.all([ssmAutomationRole.arn, newRelicWinEventConfigParam.name, newRelicWinServiceConfigParam.name,]).apply(([ssmRoleArn, newRelicWinEventConfigParamName, newRelicWinServiceConfigParamName,]) => {
        return runbookFile({
            assumeRoleArn: ssmRoleArn,
            newRelicWinEventConfigParamName,
            newRelicWinServiceConfigParamName,
            newRelicLicenseKey: '1234567890123456789',
            newRelicQueueDepth: '1000'
        });
    });

    // Create the SSM Automation Runbook
    const nodeBuildRunbookDoc = new aws.ssm.Document('nodeBuildRunbook-doc', {
        name: 'nodeBuildRunbook',
        content: nodeBuildRunbookTemplate,
        documentType: 'Automation',
        documentFormat: 'YAML',
    });

    const policyStatementResource = pulumi.all([nodeBuildRunbookDoc.arn]).apply(([arn]) => {
        return `${arn.replace('document', 'automation-definition',)}:${'$DEFAULT'}`
    });

    /*
    * This Role Policy grants permission to the Event Bridge Role to invoke
    * the Runbook for Building Instances.
    */
    new aws.iam.RolePolicy('eventBridge-nodeBuildRunbook-policy',{
        name: 'eventBridge-nodeBuildRunbook-policy',
        role: ssmAutomationRole.id,
        policy: {
            Version: '2012-10-17',
            Statement: [{
                Action: ['ssm:StartAutomationExecution'],
                Effect: 'Allow',
                Resource: policyStatementResource,
            }],
        },
    });

    return nodeBuildRunbookDoc;
};


// Emulating external call to function from parent typescript file in full project
runbook(ssmAutomationRole)

runbook.yaml

description: |
  *Some description*

schemaVersion: '0.3'
assumeRole: '\{{ AutomationAssumeRole }}'
parameters:
  AutomationAssumeRole:
    type: String
    default: '{{ assumeRoleArn }}'
    description: (Required) The ARN of the role that allows automation to perform the actions on your behalf.
  InstanceId:
    type: String
    description: (Required) AMI Source EC2 instance ID
  Region:
    type: String
    description: AWS Region
    default: 'us-west-2'
mainSteps:
  ##############################################################################
  - name: 'Wait_for_SSM_Agent'
    description: SSM Agent Needs to be Ready
    action: aws:waitForAwsResourceProperty
    timeoutSeconds: 3600
    inputs:
      Service: ssm
      Api: DescribeInstanceInformation
      InstanceInformationFilterList:
        - key: InstanceIds
          valueSet: ['\{{ InstanceId }}']
      PropertySelector: '$..PingStatus'
      DesiredValues:
        - Online
    isCritical: 'true'

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you’ve opened one already).