kubernetes: gcr.io/google-containers and gcr.io/k8s-artifacts-prod don't work anymore

What happened:

Can’t pull latest 1.18.8 container images from the two registries above. I can pull 1.18.6 though.

I can pull from k8s.gcr.io, though but I can’t find where this is documented.

What you expected to happen:

For it to work, since it was working for 1.18.6 or at least to know without doubt that the two registries would be terminated on a certain date.

How to reproduce it (as minimally and precisely as possible):

$ docker pull gcr.io/google-containers/kube-apiserver:v1.18.8
Error response from daemon: manifest for gcr.io/google-containers/kube-apiserver:v1.18.8 not found: manifest unknown: Failed to fetch "v1.18.8" from request "/v2/google-containers/kube-apiserver/manifests/v1.18.8".

$ docker pull gcr.io/google-containers/kube-apiserver:v1.18.6
v1.18.6: Pulling from google-containers/kube-apiserver
83b4483280e5: Pull complete
a26fab5270bd: Pull complete
Digest: sha256:67c8050918d28041163d973fa7cca9a84031a58e28e4ad3eb48a7a05a285520a
Status: Downloaded newer image for gcr.io/google-containers/kube-apiserver:v1.18.6
gcr.io/google-containers/kube-apiserver:v1.18.6

docker pull gcr.io/k8s-artifacts-prod/kube-apiserver:v1.18.8
Error response from daemon: manifest for gcr.io/k8s-artifacts-prod/kube-apiserver:v1.18.8 not found: manifest unknown: Failed to fetch "v1.18.8" from request "/v2/k8s-artifacts-prod/kube-apiserver/manifests/v1.18.8".

 docker pull gcr.io/k8s-artifacts-prod/kube-apiserver:v1.18.6
Error response from daemon: manifest for gcr.io/k8s-artifacts-prod/kube-apiserver:v1.18.6 not found: manifest unknown: Failed to fetch "v1.18.6" from request "/v2/k8s-artifacts-prod/kube-apiserver/manifests/v1.18.6".

 docker pull k8s.gcr.io/kube-apiserver:v1.18.8
v1.18.8: Pulling from kube-apiserver
4ba180b702c8: Already exists
bdc14197f36d: Pull complete
Digest: sha256:e9071531a6aa14fe50d882a68f10ee710d5203dd4bb07ff7a87d29cdc5a1fd5b
Status: Downloaded newer image for k8s.gcr.io/kube-apiserver:v1.18.8

Anything else we need to know?:

Approached @kubernetes/sig-release and @justaugustus linked a coupled gg discussions:

• https://groups.google.com/d/topic/kubernetes-dev/MkXnkTSJ_vs/discussion
• https://groups.google.com/d/topic/kubernetes-dev/Z-y1EK-XBQM/discussion

However, both mention moving from gcr.io/google-containers to gcr.io/k8s-artifacts-prod and none of them work as we speak.

The reason I ran into this is because I live in an air-gapped environment that requires private mirrors to these registries for things to work. Moving from one side is not trivial as at least two mirrors are required:

• one that mirrors the dying registry but keeps a cache of images we have deployed in production
• one that mirrors the new registry that containers more recent images

I really don’t know what’s the best medium for announcements like this to be made but my hunch is that this will affect a lot of people who are not prepared, eg users of kubeadm in air-gapped environments.