aws-node-termination-handler: Bad images on Public ECR?

Just started getting standard_init_linux.go:228: exec user process caused: exec format error on the v1.13.3 image from the public ECR repo, which looks to have just been pushed an hour ago. This is on a standard x86_64 host (Amazon Linux 2).

Bad image example: public.ecr.aws/aws-ec2/aws-node-termination-handler:v1.13.3 sha256:63faf9b7bd275d505d150a0510813a9cb0cab9ad599a2b29d7cd70bd97209007

From the ECR gallery page (https://gallery.ecr.aws/aws-ec2/aws-node-termination-handler), a load of images were just pushed but luckily NOT the v1.13.4 image (sha256:259100e8c555e97170f0cb83578f2e5302d062d29e58ea3976ce54efce62f35b) which looks OK so far, and hence I’ve switched to that as a workaround.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 18
Comments: 24 (9 by maintainers)

Most upvoted comments

The bad manifests in the ECR public repo have been patched. Apologies for the inconvenience this caused. Let us know if anyone is still experiencing issues.

bwagner5 on Nov 18, 2021

We are facing the same issue.

For now, we retrieved the image digest from a working node and we set it on the deployment directly (instead of using the image tag).

Example for the v1.12.2 version:

public.ecr.aws/aws-ec2/aws-node-termination-handler@sha256:0a322003234bad0d0139646388f1f1f41802cf04259bc41f032c9e32579a2a67

For helm, we can trick the release by using

image:
  repository: public.ecr.aws/aws-ec2/aws-node-termination-handler@sha256
  tag: 0a322003234bad0d0139646388f1f1f41802cf04259bc41f032c9e32579a2a67

AdrienKuhn on Nov 18, 2021

I also faced this issue. Upgraded to 1.14.0 and It is fine.

Jooyoung-dable on Nov 18, 2021

Any news? we are receiving the same error with v1.13.0

lobosg on Nov 18, 2021

Thanks, I see new pods are starting ok now.

dougbyrne on Nov 18, 2021

All manifests have been patched. Let us if you are still having problems. NOTE: you may need to change the image pull policy to Always since the default is IfNotPresent which would cache the bad image pulled. Or you can roll the node.

bwagner5 on Nov 18, 2021

@kanor1306 v1.14.0 is available: public.ecr.aws/aws-ec2/aws-node-termination-handler:v1.14.0

bwagner5 on Nov 18, 2021

So are we. v1.13.3

ToniRib on Nov 18, 2021