copilot-cli: Bug with SSM secrets?

I ended up having this issue again, but determined it was a copy and paste error. When receiving the guidance from the CLI to update the manifest file with the secrets I was copying output like this:

MY_SECRET: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/MY_SECRET`

and leaving the character ` in the manifest file… removing that fixed the issue 🤦

Hey all! I did some digging and was able to reproduce the issue (thanks @aflansburg!) and find something that I think makes this a hard bug to track down:

To reproduce:

1. create secret TEST_SECRET_ONE (i copy-pasted the same value in for each of my env’s)
2. update manifest with secret correctly (no trailing `, key and value copy&pasted from terminal.app directly into vscode)
3. Run copilot svc deploy - it works fine
4. add ` to the end of secret value in manifest
5. copilot svc deploy, doesn’t work (relevant cfn in copilot svc package:

Secrets:
  - Name: TEST_SECRET_ONE
    ValueFrom: /copilot/playground/beta/secrets/TEST_SECRET_ONE`

Error message:

Resource handler returned message: "Invalid request provided: Create T                                       
    askDefinition: The Systems Manager parameter name specified for secret                                       
     TEST_SECRET_ONE is invalid. The parameter name can be up to 2048 char                                       
    acters and include the following letters and symbols: a-zA-Z0-9_.-, (S                                       
    ervice: AmazonECS; Status Code: 400; Error Code: ClientException; Requ                                       
    est ID: 44b16b74-452a-4e55-ae73-10c7ae5f611e; Proxy: null)" (RequestTo                                       
    ken: ba1b9a59-caf9-de68-b843-c28b531ccc8e, HandlerErrorCode: InvalidRe                                       
    quest)                                                                                                       
✘ deploy service iss-3446 to environment beta: deploy service: stack playground-beta-iss-3446 did not complete successfully and exited with status UPDATE_ROLLBACK_COMPLETE

1. Remove ` from end of secret in copilot manifest
2. copilot svc deploy, still fails (relevant cfn from copilot svc package:

Secrets:
  - Name: TEST_SECRET_ONE
    ValueFrom: /copilot/playground/beta/secrets/TEST_SECRET_ONE

(so the CFN template generated by Copilot appears to be getting updated - the ` is gone)

Error message:

✘ deploy service iss-3446 to environment beta: deploy service: change set with name copilot-14edaf98-e209-4ce3-913c-45b8c4fbcd2d for stack playground-beta-iss-3446 has no changes: Resource handler returned message: "Invalid request provided: Create TaskDefinition: The Systems Manager parameter name specified for secret TEST_SECRET_ONE is invalid. The parameter name can be up to 2048 characters and include the following letters and symbols: a-zA-Z0-9_.-, (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 44b16b74-452a-4e55-ae73-10c7ae5f611e; Proxy: null)" (RequestToken: ba1b9a59-caf9-de68-b843-c28b531ccc8e, HandlerErrorCode: InvalidRequest)

Based on this last error message, it appears that CloudFormation doesn’t generate a change set when ValueFrom changes in a Task Definition. If you make a different change in the manifest (like change count), CloudFormation will pick up the secret change as well and deploy a new Task Definition that works.

---

Next steps:

• Make the change suggested by @efekarakus, should reduce the potential of getting into this situtation
• Test changing ValueFrom outside of Copilot and follow up with relevant the internal teams to make sure changes to ValueFrom get detected

@dannyrandall Hi Danny, The old version I used: v1.8 Now, I updated it to v1.17

omg!

ok it sounds like if we print with code blocks ``` instead of just the ` character this bug scenario would go away.

We should replace https://github.com/aws/copilot-cli/blob/a28a4682b4bf53e616c7840077c30e6224170498/internal/pkg/cli/secret_init.go#L419 with color.HighlightCodeBlock

👋 I’m also experiencing this issue when setting an IP address as a secret with copilot secret init. Important to note, that this was working previously, but I needed to add a domain and HTTPS endpoint for our app and so after destroying and recreating the app and attempting to deploy a service I receive:

    Resource handler returned message: "Invalid request provided: Create T                                   
    askDefinition: The Systems Manager parameter name specified for secret                                   
     MY_SECRET_HERE is invalid. The parameter name can be up to 2048 character                                   
    s and include the following letters and symbols: a-zA-Z0-9_.-, (Servic                                   
    e: AmazonECS; Status Code: 400; Error Code: ClientException; Request I                                   
    D: 967b0aac-3f81-4910-8e05-52601c8897eb; Proxy: null)" (RequestToken:                                    
    5d80ebd9-b11a-02b2-27da-43040b7fc2a9, HandlerErrorCode: InvalidRequest                                   
    )

I have also tried deleting and manually recreating the secret in SSM parameter store, but no luck.

Sidenote: I just updated to 1.19 today from 1.18 and am running a “Load Balanced Web Service”

UPDATE: In a bizarre twist of events, I renamed my secrets, which all began with MONGO_ (for MongoDB) to begin with just DB_ instead, “hand typed” the secrets instead of pasting them from a .env in vsCode and it started working 🤔 Now, I’m wondering if something with copy/paste was inserting some incorrect characters into the cli command OR if renaming had any bearing.