copilot-cli: Worker Service getting permission errors when reading from its own queue

Hi! I used copilot to add a scheduled job and a worker service. The scheduled job manifest looks something like this:

name: scheduled-job
type: Scheduled Job

on:
  schedule: "@every 1m"

image:
  build: scheduler-service/Dockerfile

cpu: 256 
memory: 512 
platform: linux/x86_64

publish:
  topics:
    - name: jobs

The worker service manifest looks something like this:

name: worker-service
type: Worker Service

image:
  build: worker-service/Dockerfile

count: 1
exec: true

# storage:
# readonly_fs: true   

# The events can be received from an SQS queue via the env var $COPILOT_QUEUE_URI.
subscribe:
  topics:
    - name: jobs
      service: scheduled-job

In my worker-service/Dockerfile I run my node script. It uses the AWS SDK to interact with the SQS queue created for the worker service. Here is a simplified version of the server:

import { prisma } from "./prisma/prisma-client";
import AWS from "aws-sdk";

const queueUrl = process.env.COPILOT_QUEUE_URI;
const sqs = new AWS.SQS();

const receiveAndProcessMessage = async () => {
  const params = {
    MaxNumberOfMessages: 1,
    QueueUrl: queueUrl,
    VisibilityTimeout: 20,
    WaitTimeSeconds: 0,
  };

  try {
    const data = await sqs.receiveMessage(params).promise();
    if (data.Messages) {
      const message = data.Messages[0];
      await processMessage(message);
      await deleteMessage(message);
    } else {
      console.log("No messages in queue.");
    }
  } catch (err) {
    console.error(err);
  }
};

async function main() {
  await receiveAndProcessMessage();
}

main()

I am seeing a permission error saying:

AccessDenied: Access to the resource https://sqs.us-east-2.amazonaws.com/ is denied.

I checked and it appears that the SQS queue’s permissions (the default one that got created for this worker service) has the correct IAM role (the same one that the created ECS has)

I haven’t messed with any permissions outside of copilot. Is my setup incorrect, am I missing something? Seems the service should have access to this queue without any additional trouble.

Thanks!

About this issue

Original URL
State: closed
Created a year ago
Comments: 34 (17 by maintainers)

Most upvoted comments

Hi! Thanks so much for your help!!

Therefore, if there are other AWS credentials present in your container, it is possible that the SQS client is making calls from that credential (which doesn’t have access to the SQS queue), instead of from the TaskRole.

^ You were totally right about this, that is exactly what was going on!

Everything is working as expected now 🎉 Haha finally 😅

Thanks again. 🎉 It feels almost sad to close this issue, been going on for so long! I’ll miss you 😛 !

marytal on May 25, 2023

Yeah AWS support should be able to help! Probably not related but just in case 💭 In const sqs = new SQSClient({ region: "use-east-2" });, “us-east-2” was mistyped!

Lou1415926 on Apr 28, 2023

Oh… I think I see what’s going. When I created the deploy the “worker-service” queue got briefly created but then it got deleted because the deploy was rolled back. The other two queues I think just belong to a separate worker service (queue-worker) that someone else created.

So what I’m going to try to do is deploy without referencing the SQS queue. Then hopefully the deploy will succeed and I can copy the ARN of the created queue and add it to my addon. Then deploy again with the code accessing the queue.

🧐🧐🧐

marytal on Apr 20, 2023

Yes! You’d use an addon resource. You may have to hardcode the queue ARN in this policy template, but you should be able to look it up either in the console or via copilot svc show --resources.

Then you’d add the following template into copilot/svcname/addons/sqs-iam.yml.

Parameters:
  App:
    Type: String
    Description: Your application's name.
  Env:
    Type: String
    Description: The environment name your service, job, or workflow is being deployed to.
  Name:
    Type: String
    Description: The name of the service, job, or workflow being deployed.

Resources:
  SQSAccessPolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Sid: SQSActions
            Effect: Allow
            Action:
              - sqs:ReceiveMessage
              - sqs:DeleteMessage
            Resource: ${REPLACE_ME_YOUR_QUEUE_ARN}

Outputs:
  # 1. You need to output the IAM ManagedPolicy so that Copilot can add it as a managed policy to your ECS task role.
  SQSAccessPolicyArn:
    Description: "The ARN of the ManagedPolicy to attach to the task role."
    Value: !Ref SQSAccessPolicy

bvtujo on Apr 20, 2023

Hey @marytal, you’re right that this shouldn’t be an issue. Let me look into this and see if there’s a quick fix.

bvtujo on Apr 18, 2023