aws-cdk: (@aws-cdk/aws-ecr-assets): DockerImageAsset - can't tell which images are outdated, and where they came from

Description

I got the results back from a 3rd part security test of our AWS account. The results included this finding:

Amazon Elastic Container Registry (ECR) repositories had vulnerabilities identified by the ECR scanning service.

And it listed the affected assets. They were all CDK assets.

aws-cdk/assets:157c44972d2cfea90aa4428e8b06b6527062d992e06eef1a9f12e2ec1c6d4821"
aws-cdk/assets:e887bc7ca1693059443036ccb17ccc10f1a203a1ecf4004dc137c111d3c8e919"
aws-cdk/assets:c91fc18db0adf214fc5ca60d9edd90d546b0328f7ebb3fc47251b6aed9eb6ab6"
aws-cdk/assets:a5d71a67cf09b808b3534a73780548dc0613a8a45925098fbc3ebdbb7f46cfab"
...

So I went to ECR to take a look.

But I can’t see a way to differentiate between the 8 DockerImageAssets for different projects that all deploy to the same AWS account using CDK, and I assume it will just keep all the old stuff lying around and growing.

So clearly this isn’t the way to do things:

        const dockerfile = path.join(__dirname, '../../')
        const dockerImage = new ecrAssets.DockerImageAsset(this, 'frontend', {
            directory: dockerfile,
            exclude: ['.git', 'cdk.out', 'node_modules'],
            buildArgs: {
                NEXT_PUBLIC_TAG_MANAGER_URL: props.tagManagerUrl,
            },
        })
        const image = ecs.ContainerImage.fromDockerImageAsset(dockerImage)

Perhaps DockerImageAsset needs to be more aggressive about cleaning up after itself, and also to provide information in ECR so that it’s possible to find out which CDK project it’s related to?

Use Case

Need to find the source of the security issues in the Docker containers without looking into each CDK project build history to find the hash in the CI logs.

Proposed Solution

Also tag the ECR instances with the CloudFormation ID.

Other information

No response

Acknowledge

I may be able to implement this feature request
This feature might incur a breaking change

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 8
Comments: 15 (5 by maintainers)

Most upvoted comments

I’ll +1 the suggestion that DockerImageAsset be retired. It makes absolutely zero sense that multiple Stacks can deploy to the same ECR repo with zero indication of where or what the images contained within are. It’s not even clear to me how you ensure that you get the right image for each task/service right now aside from hoping that they have independent tags on them.

I’d much rather have DockerImageAsset create a new (named) ECR repo, and then upload images to that specific repo. Then we could easily manage images through lifecycle policies on the named ECR repos.

+10

jgodwin on Mar 1, 2023

This is a massive problem for those using AWS Inspector.

Firstly, when vulnerabilities are reported, I have no simple way to attribute those vulnerabilities to the component that generated the image and needs updating.

Secondly, because there’s no cleanup, I’m going to be getting security reports for outdated images that haven’t actually been deployed for potentially years.

SamStephens on May 15, 2023

yes, I definitely agree with that. And what I suggested wouldn’t fix this issue. I should move that to another feature request 👍🏻

On the issue subject, I had a lambda that stopped working in the past because the image has been deleted by an ECR lifecycle. I got no notifications from AWS lambda - and it was not possible to update the CDK stack of that lambda, ultimately I had to delete the stack (I also had to delete some resources manually or the stack deletion would never finish…) and recreate it.

Ultimately, I don’t think DockerImageAsset is a good design, and its use should be discouraged in favour of a new design which uses explicit naming and tagging, immutable tags, and manages the deletion of unused images at the CDK layer.

I think the new construct should also enforce to use an explicit ECR repository, instead of publishing everything to the same cdk repo

rawpixel-vincent on Feb 21, 2023

This is still a problem. CDK’s DockerImageAsset construct still creates untraceable assets with no mechanism for cleaning them up, or identifying the source of the asset.

a-h on Feb 2, 2023

Thanks @madeline-k - I’ve put some detailed notes on how I think the overall problem could be resolved into https://github.com/aws/aws-cdk-rfcs/issues/64#issuecomment-1027740674

a-h on Feb 2, 2022

Since I’m only using ECS, I wrote a Go program to:

Search through the ECS tasks and find all the containers that were currently in use.
List all of the containers in ECR.
Do a diff between the in-use and not in-use containers.
Print out which ECS services are using which containers in ECR.
Print out a list of all of the unused containers.
Optionally delete all the unused containers.

I was able to clean up 1200+ unused images from my testing environment, and then focus on solving the security vulnerabilities in images that are actually in use.

https://github.com/a-h/cdk-ecr-asset-cleaner/

a-h on Jan 28, 2022