aws-cdk: aws-eks: addHelmChart() fails with public.ecr.aws

Describe the bug

When updating a stack containing an EKS cluster and adding an helm chart as following:

const chart = this.cluster.addHelmChart('karpenter', {
wait: true,
timeout: Duration.minutes(15),
chart: 'karpenter',
release: 'karpenter',
repository: 'oci://public.ecr.aws/karpenter/karpenter',
namespace: 'karpenter',
version: 'v0.19.2', 
createNamespace: false,
values: {
  serviceAccount: {
    create: false,
    name: serviceAccount.serviceAccountName,
    annotations: { 'eks.amazonaws.com/role-arn': serviceAccount.role.roleArn },
  },
  clusterName: this.cluster.clusterName,
  clusterEndpoint: this.cluster.clusterEndpoint,
  aws: { defaultInstanceProfile: instanceProfile.ref },
},
});
chart.node.addDependency(namespace);

The process timeout after the specified (maximum allowed) 15 minutes.

Looking at the lambda logs on CloudWatch, we get: [INFO] 2022-11-23T02:24:24.338Z ba8e8ffd-8069-4590-a492-9a58168180d0 b'\nCould not connect to the endpoint URL: "https://api.ecr-public.ap-southeast-2.amazonaws.com/"\nLogin Succeeded\nPulled: public.ecr.aws/karpenter/karpenter:v0.19.2\nDigest: sha256:4ee72250b55d7c45d72f4c8382ed9b2f0f63ec67def84cd63300667e58ff2167\n' 2022-11-23T02:39:13.812Z ba8e8ffd-8069-4590-a492-9a58168180d0 Task timed out after 900.10 seconds

Expected Behavior

The helm chart should be pulled and installed.

Current Behavior

The process timeout.

Reproduction Steps

This is a simplified version of our code

export class EksCluster extends Stack {
  cluster: Cluster;
  version = eks.KubernetesVersion.of('1.24');
  vpc: ec2.IVpc;

  constructor(scope: Construct, id: string, props: EksClusterProps) {
    super(scope, id);
    this.vpc = ec2.Vpc.fromLookup(this, 'Vpc', { tags: { BaseVPC: 'true' } });

    this.cluster = new eks.Cluster(this, 'EKSCluster', {
      clusterName: 'testCluster',
      version: this.version,
      vpc: this.vpc,
      defaultCapacity: 0,
      kubectlLayer: new KubectlV24Layer(this, 'KubeCtlLayer'),
    });

    this.cluster.addHelmChart('karpenter', {
      timeout: Duration.minutes(15),
      wait: true,
      chart: 'karpenter',
      release: 'karpenter',
      repository: 'oci://public.ecr.aws/karpenter/karpenter',
      namespace: 'karpenter',
      version: 'v0.19.2',
      createNamespace: false,
    });
  }
}

Possible Solution

No response

Additional Information/Context

https://api.ecr-public.ap-southeast-2.amazonaws.com/ seems wrong as it looks like only us-east-1 and us-west-2 are available regions. This URL is generated by cdk using the default region of our stack. I don’t know if there is a way to specify a region for this specific helm chart repo?

CDK CLI Version

2.50.0 (build 4c11af6)

Framework Version

2.50.0

Node.js Version

v16.17.0

OS

Ubuntu 22.04.1 LTS

Language

Typescript

Language Version

No response

Other information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 17 (8 by maintainers)

Most upvoted comments

Also… one thing that helm chart values changed between Karpenter v0.18.x and v0.19.x.

values: {
  serviceAccount: {
    create: false,
    name: serviceAccount.serviceAccountName,
    annotations: { 'eks.amazonaws.com/role-arn': serviceAccount.role.roleArn },
  },
  clusterName: this.cluster.clusterName,
  clusterEndpoint: this.cluster.clusterEndpoint,
  aws: { defaultInstanceProfile: instanceProfile.ref },
}

is now…

values: {
  serviceAccount: {
    create: false,
    name: serviceAccount.serviceAccountName,
    annotations: { 'eks.amazonaws.com/role-arn': serviceAccount.role.roleArn },
  },
  settings: {
    aws: {
      clusterName: this.cluster.clusterName,
      clusterEndpoint: this.cluster.clusterEndpoint,
      defaultInstanceProfile: instanceProfile.ref,
    },
  },
}

Hi @Wyfy0107, I was just going to test my change before pushing a PR!

I think it’s probably because of this line:

https://github.com/aws/aws-cdk/blob/6224b6d850ad1e019e60e905b1799baa071cf269/packages/%40aws-cdk/aws-eks/lib/kubectl-handler/helm/__init__.py#L113

When we authenticate with aws ECR public, I believe we should always use us-east-1 rather than the $AWS_REGION environ variable.

I am making this a P2 and any PR would be highly appreciated.