registry.k8s.io: 403 error when pulling images from Swiss cloud provider

For the past few weeks, we are unable to pull any images from registry.k8s.io or k8s.gcr.io. The problem started to occur randomly in december. Images would pull some days, and fail some other days. Now, for the past two to three weeks, we can’t pull any images form both registres.

Sadly, I have absolutly no idea who I should reach about that. So I’m trying my luck here.

Pull are made from AS29222, mainly 195.15.243.0/24.

Here are crane logs trying to pull metrics-server:v0.6.2:

ubuntu@k8s-worker-2:~$ ./crane pull --verbose k8s.gcr.io/metrics-server/metrics-server:v0.6.2 /dev/null
2023/03/13 17:43:44 --> GET https://k8s.gcr.io/v2/
2023/03/13 17:43:44 GET /v2/ HTTP/1.1
Host: k8s.gcr.io
User-Agent: crane/0.13.0 go-containerregistry/0.13.0
Accept-Encoding: gzip


2023/03/13 17:43:44 <-- 403 https://k8s.gcr.io/v2/ (167.668712ms)
2023/03/13 17:43:44 HTTP/2.0 403 Forbidden
Content-Length: 1582
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html; charset=UTF-8
Date: Mon, 13 Mar 2023 17:43:44 GMT
Referrer-Policy: no-referrer

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 403 (Forbidden)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>403.</b> <ins>That’s an error.</ins>
  <p>Your client does not have permission to get URL <code>/v2/</code> from this server.  <ins>That’s all we know.</ins>

Error: GET https://k8s.gcr.io/v2/: unexpected status code 403 Forbidden: <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 403 (Forbidden)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>403.</b> <ins>That’s an error.</ins>
  <p>Your client does not have permission to get URL <code>/v2/</code> from this server.  <ins>That’s all we know.</ins>

And trying registry.k8s.io:

ubuntu@k8s-worker-2:~$ ./crane pull --verbose registry.k8s.io/metrics-server/metrics-server:v0.6.2 /dev/null
2023/03/13 17:45:52 --> GET https://registry.k8s.io/v2/
2023/03/13 17:45:52 GET /v2/ HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/0.13.0 go-containerregistry/0.13.0
Accept-Encoding: gzip


2023/03/13 17:45:53 <-- 403 https://registry.k8s.io/v2/ (185.958816ms)
2023/03/13 17:45:53 HTTP/2.0 403 Forbidden
Content-Length: 298
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer


<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>403 Forbidden</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Forbidden</h1>
<h2>Your client does not have permission to get URL <code>/v2/</code> from this server.</h2>
<h2></h2>
</body></html>

Error: GET https://registry.k8s.io/v2/: unexpected status code 403 Forbidden:
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>403 Forbidden</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Forbidden</h1>
<h2>Your client does not have permission to get URL <code>/v2/</code> from this server.</h2>
<h2></h2>
</body></html>

The same pull works from other Swiss providers.

If this is not the right place, sorry for the noise and feel free to close this. But please, at least point me to somewhere I can get help with that.

Thanks!

About this issue

Original URL
State: closed
Created a year ago
Comments: 17 (12 by maintainers)

Most upvoted comments

Hi Dims!

Indeed, it’s been a long time. I’d love to discuss with you again.

I started a new thread here: https://issuetracker.google.com/issues/273978804

I’m not sure where this is going to lead us…

Cheers,

Thomas

thomasgoirand on Mar 17, 2023

Hi @ameukam . How can (we) the cloud provider handle the issue then? I’ve searched how to get de-listed from “cloud armor” but didn’t find any ways to have this happen.

thomasgoirand on Mar 17, 2023