aws-cdk: Logs: Faulty Resource Policy generated

What is the problem?

I’m attempting to create a simple stack that houses a task definition for some fargate tasks.
When I add a logging configuration to a container definition inside, a resource policy is added in the generated cf template:

  "Resources": {
    "ecsstackloggroupPolicy0078989C": {
      "Type": "AWS::Logs::ResourcePolicy",
      "Properties": {
        "PolicyDocument": "{\"Statement\":[{\"Action\":[\"logs:CreateLogStream\",\"logs:PutLogEvents\"],\"Effect\":\"Allow\",\"Principal\":
        ...
        ...

Reproduction Steps

from aws_cdk import (
    core as cdk,
    aws_ecs as ecs,
    aws_iam as iam,
    aws_ecr as ecr,
    aws_logs as logs,
)
REGION="us-east-2"
FAMILY="example"
TASK_ARN="arn:aws:iam::<example existing iam role arn>"
EXEC_ARN="arn:aws:iam::<example existing iam role arn>"
ECR_PIPELINE_ARN="arn:aws:ecr:<example existing ECR arn>"
LOG_GROUP_ARN="arn:aws:logs:<example existing log group arn>"
class EcsStack(cdk.Stack):

    def __init__(self, scope: cdk.Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # reference execution role
        exec_role = iam.Role.from_role_arn(
            self,
            'fargate-exec-role',
            EXEC_ARN,
            mutable=False
        )

        # reference task role
        task_role = iam.Role.from_role_arn(
            self,
            'fargate-task-role',
            TASK_ARN,
            mutable=False
        )

        # reference pipeline ecr image
        repo = ecr.Repository.from_repository_arn(
            self,
            'repo',
            ECR_PIPELINE_ARN
        )

        image = ecs.ContainerImage.from_ecr_repository(
            repo,
            'image-name'
        )
        
        # logging
        log_group = logs.LogGroup.from_log_group_arn(
            self,
            'ecs-stack-log-group',
            log_group_arn=LOG_GROUP_ARN
        )
        log_driver = ecs.LogDriver.aws_logs(
            stream_prefix="test",
            log_group=log_group
        )

        # construct base task definition
        base_def = ecs.TaskDefinition(
            self,
            'base-definition',
            family=FAMILY,
            execution_role=exec_role,
            task_role=task_role,
            compatibility=ecs.Compatibility.FARGATE,
            cpu='512',
            memory_mib='1024'
        )
        base_def.add_container(
            id='pipeline',
            container_name='pipeline',
            image=image,
            cpu=512,
            memory_limit_mib=1024,
            memory_reservation_mib=512,
            essential=True,
            environment=[],
            secrets=[],
            logging=log_driver
        )

What did you expect to happen?

I was attempting to generate a template that behaved similarly to a CF template I’ve written below.

When deployed, a task definition with a single container definition is created, with references to the pre-existing resources I’ve specified using ARNs.

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Resources": {
        "TaskDefinition": {
            "Type": "AWS::ECS::TaskDefinition",
            "Properties": {
                "Family": "test-deploy-definition",
                "ExecutionRoleArn": "...",
                "TaskRoleArn": "...",
                "RequiresCompatibilities": [
                    "FARGATE"
                ],
                "Cpu": "512",
                "Memory": "1024",
                "NetworkMode": "awsvpc",
                "ContainerDefinitions": [
                    {
                        "Name": "...",
                        "Image": "...",
                        "Command": [],
                        "EntryPoint": [],
                        "MountPoints": [],
                        "Cpu": 512,
                        "Memory": 1024,
                        "MemoryReservation": 512,
                        "PortMappings": [],
                        "Essential": true,
                        "Environment": [
                        ],
                        "Secrets": [],
                        "LogConfiguration": {
                            "LogDriver": "awslogs",
                            "Options": {
                              "awslogs-group": "/test/log",
                              "awslogs-region": "us-east-2",
                              "awslogs-stream-prefix": "test"
                            }
                    
                        }
                    }
                ]
            }
        }
    }

What actually happened?

  1. The resource policy request generates an error:
11:06:15 AM | CREATE_FAILED        | AWS::Logs::ResourcePolicy | ecsstackloggroupPolicy0078989C
Resource handler returned message: "Invalid request provided: AWS::Logs::ResourcePolicy" (RequestToken: 24b0df2b-73fa-056d-4371-7b19f637cb1c, HandlerErrorCode: InvalidRequest)

        new ResourcePolicy (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/jsii-kernel-RXZcI5/node_modules/@aws-cdk/aws-logs/lib/policy.js:17:9)
        \_ Import.addToResourcePolicy (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/jsii-kernel-RXZcI5/node_modules/@aws-cdk/aws-logs/lib/log-group.js:125:27)
        \_ Function.addToPrincipalOrResource (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/jsii-kernel-RXZcI5/node_modules/@aws-cdk/aws-iam/lib/grant.js:70:49)
        \_ Import.grant (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/jsii-kernel-RXZcI5/node_modules/@aws-cdk/aws-logs/lib/log-group.js:97:26)
        \_ Import.grantWrite (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/jsii-kernel-RXZcI5/node_modules/@aws-cdk/aws-logs/lib/log-group.js:89:21)
        \_ AwsLogDriver.bind (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/jsii-kernel-RXZcI5/node_modules/@aws-cdk/aws-ecs/lib/log-drivers/aws-log-driver.js:48:23)
        \_ new ContainerDefinition (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/jsii-kernel-RXZcI5/node_modules/@aws-cdk/aws-ecs/lib/container-definition.js:114:50)
        \_ TaskDefinition.addContainer (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/jsii-kernel-RXZcI5/node_modules/@aws-cdk/aws-ecs/lib/base/task-definition.js:265:16)
        \_ /private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/tmpp_n4_w2q/lib/program.js:8248:134
        \_ Kernel._wrapSandboxCode (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/tmpp_n4_w2q/lib/program.js:8860:24)
        \_ /private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/tmpp_n4_w2q/lib/program.js:8248:107
        \_ Kernel._ensureSync (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/tmpp_n4_w2q/lib/program.js:8841:28)
        \_ Kernel.invoke (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/tmpp_n4_w2q/lib/program.js:8248:34)
        \_ KernelHost.processRequest (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/tmpp_n4_w2q/lib/program.js:9757:36)
        \_ KernelHost.run (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/tmpp_n4_w2q/lib/program.js:9720:22)
        \_ Immediate._onImmediate (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/tmpp_n4_w2q/lib/program.js:9721:46)
        \_ processImmediate (internal/timers.js:461:21)


 ❌  EcsStack failed: Error: The stack named EcsStack failed to deploy: UPDATE_ROLLBACK_COMPLETE
...
  1. It is unclear if this resource policy was even required, given that the defined task and execution roles have necessary log permissions. The documentation does not explain why this is generated in the first place when a logging configuration is attached to the container definition.

CDK CLI Version

1.132.0 (build 5c75891)

Framework Version

No response

Node.js Version

v12.21.0

OS

MacOS Catalina 10.15.7

Language

Python

Language Version

3.9.6

Other information

Our AWS account is controlled by a information security service provider - I thought that perhaps this was a personal account permissions issue. However, the error is InvalidRequest and not AccessDenied so I’m gut checking here first.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 6
  • Comments: 34 (9 by maintainers)

Commits related to this issue

Most upvoted comments

Any update on this? The 1 to 3 weeks are over since some time. Still not working on my side. Also on 1.147 the same issue happens. "Invalid request provided: AWS::Logs::ResourcePolicy" (RequestToken: ce0942e0-30c6-4221-bbac-88a8bbc09e25, HandlerErrorCode: InvalidRequest)

+4
simonscheurer on Mar 8, 2022

This is a possible workaround for logging issue

const logGroup = new LogGroup(this, `${API_PREFIX}LogGroup`, {
        ... some props
});

const taskDefinition = new FargateTaskDefinition(this, `${API_PREFIX}TaskDefinition`, {
        ... some props
});
       
taskDefinition.addContainer(`${API_PREFIX}Container`, {
        ... some props

        // -- Disabled logging in container, this is where the bug in CDK lives. 
        // logging: LogDrivers.awsLogs({
        //     streamPrefix: API_PREFIX,
        //     logGroup: logGroup
        // })
    }
);

// -- Logging is added with a cfnTaskDefinition override
const cfnTaskDefinition = taskDefinition.node.defaultChild as CfnTaskDefinition;
cfnTaskDefinition.addOverride("Properties.ContainerDefinitions.0.LogConfiguration", {
    "LogDriver": "awslogs",
    "Options": {
        "awslogs-group": (logGroup.node.defaultChild as CfnLogGroup).ref,
        "awslogs-stream-prefix": API_PREFIX,
        "awslogs-region": "*****" // change this to actual region
    }
});
+3
BuruY on Feb 10, 2022

@comcalvi I am not sure what you mean by “exported value”. I am using a CDK stack (in Typescript if that’s relevant). Today I upgraded from 1.141.0 to 2.10.0 and still get the same error when I try to deploy the stack.

9:58:33 AM | CREATE_FAILED        | AWS::Logs::ResourcePolicy | LogGroupPolicyResourcePolicy6FA18555
Resource handler returned message: "Invalid request provided: AWS::Logs::ResourcePolicy" (RequestToken: 469a133f-e000-4d55-e8f0-b33a77787491, HandlerErrorCode: InvalidRequest)

I am creating a log group manually and then attaching it to an ECS task container

const logGroup = new logs.LogGroup(this, 'LogGroup', {
  logGroupName: '/ecs/someName',
  retention: logs.RetentionDays.ONE_MONTH,
  removalPolicy: RemovalPolicy.DESTROY,
})

task.addContainer('Reporter', {
  image: ecs.RepositoryImage.fromEcrRepository(repository, imageTag.valueAsString),
  environment: {
    // ...
  },
  logging: ecs.AwsLogDriver.awsLogs({
    streamPrefix: '',
    logGroup: logGroup
  }),
  secrets: {
    // ...
  }
})

The task is allowed to create log streams via a shared execution role that I load via it’s ARN (but with mutable: false).

I initially only had the ECR repository in the stack and successfully deployed it, but now after adding the other resources deployments always fail.

Even with multiple -v I don’t get a message what actually is invalid in the request though.

+3
FrontierPsychiatrist on Jan 31, 2022

Thanks @BuruY ! This one was a real doozie. Here’s the override part of the code for the Python crowd:

cfn_taskdef: ecs.CfnTaskDefinition = fargate_taskdef.node.default_child
cfn_loggroup: logs.CfnLogGroup = log_group.node.default_child
cfn_taskdef.add_override(
  "Properties.ContainerDefinitions.0.LogConfiguration",
  {
    "LogDriver": "awslogs",
    "Options": {
        "awslogs-group": cfn_loggroup.ref,
        "awslogs-stream-prefix": "your-log-group-prefix",
        "awslogs-region": "us-east-1"
    }
  }
)
+2
ghost on Mar 1, 2022

Same problem here but for MSK Using CDK 2.0.0

+2
bjornhandersson on Dec 8, 2021

This is taking too long. Can someone provide a L1 example fix using Python code, please 😃

+1
hao-public on Mar 9, 2022

likely 1-3 weeks.

+1
comcalvi on Feb 8, 2022

I also have this issue when trying to create a log group and use it for an ECS Task container with version 1.139.0.

+1
FrontierPsychiatrist on Jan 18, 2022

@bjornhandersson thanks for the stack, I can confirm that it reproduces the issue. However the line that you rightfully pointed out causes the problem:

logGroup.grantWrite(new iam.ArnPrincipal(cluster.ref));

is working as intended. The cluster.ref returns a resource ARN, which is not a valid IAM principal (see docs).

This reproduces the error for the wrong reason (the error is desired here). Did this successfully deploy on a version prior to 1.132.0? Can someone provide a stack that successfully deployed on older versions but was failed deployment with that error message when you upgraded to 1.132.0?

+1
comcalvi on Dec 16, 2021

I have just ran into this issue. A CDK Stack that created a log group now fails after upgrade to 1.132.0. In my case, I am creating a CodeBuild project in the stack and the IAM role assigned already has the required permissions.

Looks like resource policy support was added and it is adding the policy by default (auto permissions I guess). Maybe an option is required when creating the LogGroup whether resource policy should be created automatically or not.

Reverting back to 1.130.0 works for my existing stack.

+1
vramanlal on Nov 19, 2021
Read more comments on GitHub
← aws-cdk: (lambda-python): arm64 architecture is not respected
aws-cdk: Make PublicAccessBlockConfiguration optional for cdk bootstrap →