harbor: Docker Hub pull-through proxy stopped working

Expected behavior and actual behavior: Our docker pull-through proxy starting exhibiting more and more failures starting today. Now it only rarely succeeds in allowing a pull.

Steps to reproduce the problem:

$ docker pull my-harbor/dockerhub/library/alpine:3.2
Error response from daemon: unknown: artifact dockerhub/library/alpine:3.2 not found

Versions: Please specify the versions of following systems.

harbor version: 2.1.3
docker engine version: 18.09.1
docker-compose version: N/A

Additional context:

harbor.log

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 28
Comments: 19

Most upvoted comments

Why does a pull through cache fail anyway on images/tags that are present but when upstream is temporary down or not reachable.

+17

behrtam on May 5, 2021

Hey guys, we have opened a new issue so the team can mitigate this kind of problem in the future: #14822 Feel free to upvote if you also think this should be fixed and our proposal looks good to you. If not, please comment there to improve the issue

cbcos on May 7, 2021

Too bad that this issue has been closed because this comment from @behrtam is pertinent:

Why does a pull through cache fail anyway on images/tags that are present but when upstream is temporary down or not reachable.

Shouldn’t we improve the proxy cache feature in order to be fully functional even if upstream is failing ?

gregopenit on May 7, 2021

Update: Docker engineers are working on a solution (see incident URL above):

A highly probable root cause for the authentication and pull issues on the registry has been identified in part of the infrastructure involved in session creation. Engineers are working on a solution now. The incident may also affect Hub UI intermittently on authentication and logout.

cbcos on May 5, 2021