aws-cdk: (pipelines): Malformed YAML during stage deployment

Hi everyone in this thread.

Good news, CodePipeline team have been able to diagnose and fix this issue!

There are now no more problems deploying CloudFormation templates that contain unicode characters from CodePipeline. Closing this issue. Thank you all for sticking it out.

We were also experiencing the issue. In our case it was a combination of size and non-ascii characters. After increasing the size of the Stack the Prepare Step failed with “Template format error: JSON not well-formed”. After removing non-ascii characters (ü in our case) the stack could be deployed via the pipeline normally. 24KB with non-ascii characters -> Working 98KB with non-ascii characters -> Not Working (JSON not well-formed Error) 98KB without non-ascii characters -> Working

I created an internal ticket to the CodePipeline team for this, but that’s unfortunately the best we can do.

Internal reference: D68266996

Yup, that was my thought. Feel free to resolve this , if you think it’s technically not a CDK issue, even though the CFN limit isn’t clear.

But a more descriptive error would be amazing in this case! 😃 And if possible, using a TemplateURL for CloudFormation calls to allow for bigger templates (although I can see how this could be more complicated, due to S3 cross account permissions).

I am in contact next week with the CodePipeline team to discuss this. This is ridiculous in my opinion.

Experienced the same issue. I did some tests and it seems a combination of both size and special characters in the template.

A template of around 285KB deploys ok without any special characters. After adding unicode characters to an SES template it started failing with JSON (or sometimes YAML) not well-formed. Downloading the template from the synth output in S3 and deploying it directly with Cloudformation works.

To test a bit more I removed a lot of the resources from the project and just deployed the templates with unicode characters using the pipeline which worked correctly.

So from what I can make of it the issue seems to be triggered by a combination of a large template like @ghdoergeloh also mentioned, in combination with unicode characters (in my case in an SES template).

@hoegertn Hi, we have same problem when using AWS Pipelines. If I do directly from my computer using cdk deploy no problems occured and everything is fine.

Template format error: JSON not well-formed. (line 2454, column 1) (Service: AmazonCloudFormation; Status Code: 400; Error Code: ValidationError; Request ID: c08894e6-4398-4441-b83f-c3b60ed5c4f1; Proxy: null)

Template info:

• Size is 85kb
• Lines 2454
• Region: eu-west-1

I changed resources to minimize a template size. After change Codepipeline success if I decrease template size under 51kb. Solved this size issue (ValidationError) using nested stacks and keeping all of those templates size under 51kb.

Hopefully they are making better error messages or changing those limits much bigger.

Can you send the account id and executions ids via dm? I will forward it to the respective teams for inspection.

@hoegertn we have just hit this problem. We are deploying via CdkPipelines. The prepare step fails:

This is curious, since during prepare the pipeline just creates a CFN Changeset. I can download the artifact and create the changeset manually however:

As you can see the template is 56,1 kB. This is the location corresponding to the Error-Message:

Seems to me the issue might be lying in the underlying Codepipeline Action for Cloudformation. 🤔

Is there something in @aws-cdk/assert I can use to recursively cover nested stacks in my snapshots?

Not out of the box I don’t think. You can build something though.

May as well add my notes here.

We’ve been dealing with this for a few months, but I finally got a chance to dig into this today. The error always happens when trying to deploy email templates containing non-standard characters. As noted by everyone else, it works whenever CodePipeline isn’t involved.

Was able to gather enough information to make some assumptions about the nature of the issue by poking at CloudTrail:

• The error specifically happens during cloudformation:CreateChangeSet operations called by CodePipeline in the Prepare step.
• When the template file is small enough, CodePipeline embeds the template in the CreateChangeSet call itself. Others have noted that for small enough files it works fine, this is probably why. None of our templates containing special characters are small enough to be deployed this way, but I can see the difference in the requests being made for templates without special characters that are small enough so it make sense with what others have observed.
• When the template is too large to be included in the CreateChangeSet call, it is uploaded to S3 and CreateChangeSet is passed a pre-signed templateURL.
• The S3 bucket where the template is uploaded is not owned by the account doing the deployment. It is not the CodePipeline artifacts bucket, nor is it any bucket related to CDK. For us-east-1, the bucket it uses is called cloudformation-codepipeline-largetemplate-bucket-useast1. This appears to be an internal AWS bucket used by CodePipeline. I couldn’t find any mention or documentation of this bucket online.
• The pre-signed URL has the format: https://${INTERNAL_BUCKET}.s3.amazonaws.com/${DATE}/${HOUR}/${ACCOUNT_ID}/${UUID}
• My best guess is that something related to the upload to this bucket is the source of the problem. Possibly something in the bucket or built into the action pre-upload is set up to sanitize uploaded templates (given it seems to be a single bucket shared by all accounts deploying CloudFormation through CodePipeline). The sanitization is too aggressive and can mangle valid templates. This results in the JSON/YAML not well formed errors for templates that we know should be valid as they can be deployed by any other means.
• We can see the pre-signed URL’s for templates used to successfully create ChangeSets by pulling them out of CloudTrail (provided they aren’t expired). Unfortunately, if a cloudformation:CreateChangeSet operation fails with the errors reported here then the requestParameters for the call are not captured by CloudTrail and there is no way to retrieve the template to view what, if anything, may have been changed to produce the error.

Is anyone actively looking at this issue? It has blocked our deployment pipelines and has been really difficult to debug. The codepipeline build steps work but it fails when preparing in the deploy step despite the fact it can be deployed from command line ok. We are using cdk pipelines. After finding this thread I tried to search the cf template for non ascii characters and found that \ufeff is being inserted before the word schema in the appsync definition in the template and it is not yet clear if we put it there or aws processing did. I have tried to delete any character before the word schema in the file and redo the synth but it always appears. For what it’s worth as some mentioned size above, the template is too big to be evaluated on file and has to be uploaded to s3 to run validation tests. Any idea how we may remove this character? If this wont be fixed we would need to add tests somehow to check for such characters when using cdk pipelines/codepipeline but this is made difficult because the code compiles and can be deployed by cdk deploy outside of cdk pipelines/codepipeline. This seems like a significant problem. Is there a work around for CI/CD in AWS with cdk that wouldn’t experience this?

Update: after breaking up the stack into multiple stacks removing any infra not explicitly related to the AppSync API we were able to get it to deploy. But since then we have further developed our API adding more types, queries and mutations and seemingly hit the size limit and or character bug again. Not sure how we can further break up the API stack now so we are stuck. Any help would be appreciated it.

Update: We are currently working round this by putting the schema and api resource in one stack and all the resolvers in another stack and then importing the api resource by api_id so that the templates are kept small. Seems odd but it’s working at least for now.

We are currently experiencing the same issue with one of our stacks. We made a change in our code that increased the size of the respective template from 77472 to 78841 bytes. We’re able to deploy the stack locally ($ npx cdk deploy ...), but it fails on the Prepare step when trying to deploy it through CDK Pipelines.

I find it a bit surprising that the error message mentions YAML, while the TemplatePath is a JSON template. Is there some kind of preprocessing happening behind-the-scenes by CloudFormation? If so, that makes it even harder to debug wrt. the reported offending line and column number.

Full error message:

Template format error: YAML not well-formed. (line 2449, column 2) (Service: AmazonCloudFormation; Status Code: 400; Error Code: ValidationError; Request ID: <redacted>; Proxy: null)

EDIT: The change was isolated to using one or more of the characters æøå in the EmailMessage property of the AWS::Cognito::UserPoolInviteMessageTemplate CloudFormation resource, even though this is valid according to the pattern. Using HTML-encoded characters instead of these characters mitigates the error, so it seems like the error from CodePipeline is masking something happening downstream similar to what @hoegertn is mentioning rather than occurring due to template size issue.

Unfortunately not. I sent them another occurrence too. In this case I found out it was a resource type that was not supported in that region.

So currently I suspect that this error means “something went wrong” but we have no idea what.