release: Occasional failures with Google Cloud GPG keys and repos

(originally posted in Kubernetes Slack #kubernetes-users and #sig-release) Our testing infrastructure automatically sets up a Kubernetes cluster. Recently (starting approximately Sunday the 5th, plus or minus a day) we’ve started having errors fetching the GPG key and installing from the repos.

GPG key error example:

+ yum install -y 'kubectl-1.13*' 'kubelet-1.13*' 'kubeadm-1.13*'
...
Total download size: 52 M
Installed size: 236 M
Downloading packages:
Public key for 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm is not installed
warning: /var/cache/yum/x86_64/7Server/kubernetes/packages/14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY
--------------------------------------------------------------------------------
Total                                               20 MB/s |  52 MB  00:02     
Retrieving key from https://packages.cloud.google.com/yum/doc/yum-key.gpg
Importing GPG key 0xA7317B0F:
 Userid     : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>"
 Fingerprint: d0bc 747f d8ca f711 7500 d6fa 3746 c208 a731 7b0f
 From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Retrieving key from https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg


GPG key retrieval failed: [Errno 14] HTTPS Error 500 - Internal Server Error

And I can’t find it right now, but we’ve had errors curling that GPG key as well.

Repo pull error example:

+ apt-get install -y 'kubectl=1.15*' 'kubelet=1.15*' 'kubeadm=1.15*'
...
Get:6 http://newstack.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 conntrack amd64 1:1.4.3-3 [27.3 kB]
Get:7 http://newstack.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 ebtables amd64 2.0.10.4-3.4ubuntu2.16.04.2 [79.9 kB]
Get:8 http://newstack.clouds.archive.ubuntu.com/ubuntu xenial/universe amd64 socat amd64 1.7.3.1-1 [321 kB]
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.13.0-00 [8,776 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.7.5-00 [6,473 kB]
Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.15.7-00 [20.3 MB]
Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.15.7-00 [8,760 kB]
Err:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.15.7-00
  GnuTLS recv error (-110): The TLS connection was non-properly terminated.
Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.15.7-00 [8,253 kB]
Fetched 44.2 MB in 54s (809 kB/s)
E: Failed to fetch https://packages.cloud.google.com/apt/pool/kubectl_1.15.7-00_amd64_83a4bd7636fc1c94ddd08f2a0cd1afad814df716e084bfe9c4f1aaf003aec994.deb  GnuTLS recv error (-110): The TLS connection was non-properly terminated.

I also just remembered an issue we had going back a bit further, where we had intermittent pull errors from gcr.io for the kubeadm images.

Please let me know if any more information is requested, and I’ll update if/as it reoccurs.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 35 (17 by maintainers)

Most upvoted comments

2023 enters the chat. It looks like this is happening again. https://packages.cloud.google.com/apt/doc/apt-key.gpg

 curl https://packages.cloud.google.com/apt/doc/apt-key.gpg

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 500 (Internal Server Error)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/logos/errorpage/error_logo-150x54.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/logos/errorpage/error_logo-150x54-2x.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/logos/errorpage/error_logo-150x54-2x.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/logos/errorpage/error_logo-150x54-2x.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>500.</b> <ins>That’s an error.</ins>
  <p>  <ins>That’s all we know.</ins>

raykrueger on Feb 26, 2023

Key inaccessible neither from browser nor from curl

atanovv on Feb 27, 2023

This is being tracked at https://github.com/kubernetes/release/issues/2860 and https://github.com/kubernetes/release/issues/2862 apparently. Though, neither of those seem to mention the 500 error.

raykrueger on Feb 26, 2023

500 here also

What I find strange (EDIT: different DNS servers and different resolved IPs) that on MacOS zsh shell I get:

$ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg -v
<...>
< HTTP/2 200

From linux machine on the same LAN:

$ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg -v
<...>
< HTTP/2 500

administratorius on Feb 26, 2023

Thanks @listx ! We’ll keep an eye on our CI and report if we see any other errors.

Pure-AdamuKaapan on Jan 13, 2020

500 also here since morning

nivpenso on Feb 26, 2023

I’ve pinged the same Google folks from last time. Stay tuned.

listx on Jul 1, 2020

@listx – Can you take a look on the Google side?

justaugustus on Jun 30, 2020