amazon-eks-pod-identity-webhook: ECR Published Image Cannot be Fetched for Custom Cluster

What happened: When trying to run make cluster-up deployments get stuck in an image pull backoff., this causes the latter parts of the make cluster-up to fail in that there are no CSRs. When trying to pull the image locally, the ECR repository is not able to be pulled from publicly.

e.g.

$ docker pull 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pod-identity-webhook:latest
Error response from daemon: Get https://602401143452.dkr.ecr.us-west-2.amazonaws.com/v2/eks/pod-identity-webhook/manifests/latest: no basic auth credentials

Relevant k8s events from the pod:

Events:
  Type     Reason     Age                  From                     Message
  ----     ------     ----                 ----                     -------
  Normal   Scheduled  2m45s                default-scheduler        Successfully assigned default/pod-identity-webhook-56c67dff46-c6h9h to docker-desktop
  Normal   Pulling    78s (x4 over 2m44s)  kubelet, docker-desktop  Pulling image "602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pod-identity-webhook"
  Warning  Failed     78s (x4 over 2m44s)  kubelet, docker-desktop  Failed to pull image "602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pod-identity-webhook": rpc error: code = Unknown desc = Error response from daemon: Get https://602401143452.dkr.ecr.us-west-2.amazonaws.com/v2/eks/pod-identity-webhook/manifests/latest: no basic auth credentials
  Warning  Failed     78s (x4 over 2m44s)  kubelet, docker-desktop  Error: ErrImagePull
  Warning  Failed     65s (x6 over 2m43s)  kubelet, docker-desktop  Error: ImagePullBackOff
  Normal   BackOff    51s (x7 over 2m43s)  kubelet, docker-desktop  Back-off pulling image "602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pod-identity-webhook"

What you expected to happen: The pod-identity-webhook image to be publicly pullable.

How to reproduce it (as minimally and precisely as possible):

  • Be on a machine with no authentication to the 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pod-identity-webhook repo and attempt to run:
docker pull 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pod-identity-webhook:latest

Anything else we need to know?: Obviously this can be fixed by pushing an image the cluster can be reached from and passing in the relevant env vars to make, just following the current README.md instructions does not produce an expected result.

Environment:

(N/A here since this is unrelated to k8s/EKS specific bits)

  • AWS Region: N/A
  • EKS Platform version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.platformVersion): N/A
  • Kubernetes version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.version): N/A
  • Webhook Version: N/A

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 11
  • Comments: 22 (4 by maintainers)

Most upvoted comments

@tabern any word? An issue to just publish a docker image has been open for over 6 months now…

Any update on official docker images being available?

Ahhh, it seems I have misunderstood what is required to use IAM roles with pods on EKS. We don’t need to have this docker image or run this ourselves, it’s built into the EKS service. We only need to create an OIDC provider, annotate the ServiceAccount and create an IAM role with the correct trust policy.

This will teach me for skim reading READMEs and AWS announcements 😅

What is the status? 😃

Hi @micahhausler Can this move forward?

I think @nckturner is going to fix this 🎉

I’m hesitant to actually use this as I’m afraid that it’s abandon-ware.

Why are no public images yet available? Is there a general refusal to make them public? But why then leave links to private images in the documentation? Is the current state too unstable to be used so AWS won’t make them public? Why does making an image public take so many months after being reported?

No releases, few commits … doesn’t look good to me.

Edit: I missed the releases somehow