pipelines: Pipeline steps that use alpine image can become stuck when DockerHub is overloaded: ImagePullBackOff: Back-off pulling image "alpine"

What steps did you take:

I’m running a pipeline I made where the first pipeline step loads from a cache. Approximately every other run, the step hangs.

What happened:

The step displays errors that oscillate between these three messages:

This step is in Pending state with this message: ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: error parsing HTTP 429 response body: invalid character 'T' looking for beginning of value: "Too Many Requests (HAP429).\n"

This step is in Pending state with this message: ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: unauthorized: authentication required

This step is in Pending state with this message: ImagePullBackOff: Back-off pulling image "alpine"

Environment:

Kubeflow pipelines standalone v0.5 on prem

KFP version: 0.5

KFP SDK version: 0.5.1

/area backend

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 18 (10 by maintainers)

Commits related to this issue

fix(cache): update to use google/cloud-sdk:alpine instead of alpine for mutating cached steps. Fixes #4099 (#5184) * set up cluster * allow to set with env variable * fix the market place * update... — committed to kubeflow/pipelines by NikeNano 3 years ago

Most upvoted comments

@Ark-kun I think this is of much higher priority because of docker hub rate limiting

Bobgy on Dec 22, 2020

I had the same problem. kubeflow’s cached steps are using official alpine image. so docker hub’s new rate limit policy make toomanyrequests error.

I solved it by

Signup docker hub and add imagePullSecret using this docker hub account (create-an-imagepullsecret)
option a) Patch default-editor serviceaccount of your working namespace - add imagePullSecret (add-image-pull-secret-to-service-account) option b) Or you set imagePullSecret in pipeline code(set_image_pull_secrets)

I think this is inevitable problem if kubeflow use alpine image and kubeflow admin don’t set default imagePullSecret

kim-sardine on Jan 18, 2021

Well for most things I definitely want to pull from the cache, since the step in question is lengthy. So I won’t want to disable the cache.

Is this issue related to the fact that anonymous dockerhub pulls are rate-limited?

https://docs.docker.com/docker-hub/download-rate-limit/

schmidt-jake on Jul 10, 2020