cli: GPG key used to sign debian packages is expired

This change might end up being long term, as we have recently been discussing retiring our Linux packaging in favor of our releases

Please consider not doing this. The Linux packaging method is much more convenient and easier to work with.

Hello, I’m very sorry for this confusion and frustration. We dropped the ball here due to a confluence of life factors and coincidences.

Due to those same factors it’s possible this key will not be fixed in a short time frame. I would recommend that people automating the installation of gh switch to wgeting a binary from https://github.com/cli/cli/releases/latest . These releases are accessed over https and have the same level of authenticity as our signed Debian packages.

This change might end up being long term, as we have recently been discussing retiring our Linux packaging in favor of our releases (see #5941 ), but once the team is able we will attempt to fix the GPG key.

The GPG key issue should now be fixed.

Please note that if you’ve downloaded https://cli.github.com/packages/githubcli-archive-keyring.gpg previously, you will need to re-download it.

As per our Linux installation instructions, Debian/Ubuntu installation hasn’t changed:

curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg \
&& sudo chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg \
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null \
&& sudo apt update \
&& sudo apt install gh -y

For those using keyserver, the ID of the new key is 23F3D4EA75716059. Due to the number of gotchas involved in fetching keys from a keyserver, we chose to not include this in the official installation instructions, advising users to fetch the key from our website instead.

Let us know if there are any further issues!

I think if you can’t properly manage your GPG keys you should avoid signing packages with GPG.

@aucampia None of us on the team particularly enjoy the developer experience of signing anything with GPG. We didn’t have any experience with this prior to publishing GitHub CLI and, if we had a choice, we wouldn’t be doing it. However, signing package repositories seems to be mandatory, at least for Debian/Ubuntu.

Hi all, thanks for reporting the problem and for sharing your workarounds!

We’re pushing a new release of GitHub CLI soon that will be signed with an updated key served from the same location as before: https://cli.github.com/packages/githubcli-archive-keyring.gpg. Please stay tuned and don’t rewrite your setups if you can wait an hour or two more. ❤️

Unfortunately, we weren’t able to access the original signing key to bump up the expiration date, so the new key has a different ID and belongs to GitHub CLI rather than Nate Smith. I apologize for the disruption, especially because this broke so many people’s container setups and apt update upgrades over the long weekend for US folks. This caught us at the worst possible time too, since all members of our team were unavailable to deal with the incident in a timely manner. We have already taken precaution that our new key is shared within the organization so that there isn’t a single point of ownership anymore.

This change might end up being long term, as we have recently been discussing retiring our Linux packaging in favor of our releases

Please consider not doing this. The Linux packaging method is much more convenient and easier to work with.

For now we are focused on fixing and resuming our publishing of Debian packages to work as they did before. Thanks for all the feedback 🙇

This change might end up being long term, as we have recently been discussing retiring our Linux packaging in favor of our releases (see #5941 ), but once the team is able we will attempt to fix the GPG key.

I really hope this does not happen, and it is not just because of a personal preference but because of a concern for the security of the developer ecosystem.

Like everyone else, developers like to get their work done with as few context switches and interrupts as possible. The more “snowflake” updates we have to perform in order to keep software up-to-date, the less likely we are to do so. That’s harsh, but we all know that it’s the truth. Even if an app is kind enough to tell us it needs to be updated, it typically tells us this at the worst possible time - namely, right when we need to use it. This makes it likely that we might just click the option to “remind me later” … again.

On the other hand, when app updates are part of routine app maintenance, they are almost never singled out for exclusion.

Please consider sticking with de facto standards on each platform: yum, apt, snap, winget/choco, and the like.

Thank you.

Update: this is now fixed: https://github.com/cli/cli/issues/6175#issuecomment-1238477714

🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨

We are working on an update for Codespaces. (Update: https://github.com/cli/cli/issues/6175#issuecomment-1235998656)

If you are seeing issues in Actions workflows, please share more details in this issue.

Manually, you are able to reset your system with sudo apt-key del C99B11DEB97541F0 && sudo rm /etc/apt/sources.list.d/github-cli.list

Also see https://github.com/cli/cli/issues/6175#issuecomment-1235984381

🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨

And well it’s weird GH went with GPG-signing its CLI tool using someone’s personal GPG key and didn’t make it to switch to Org’s key until it was too late 😞

Had to do this on fedora:

curl -fsSL -o /var/tmp/githubcli-archive-keyring.gpg \
    https://cli.github.com/packages/githubcli-archive-keyring.gpg
gpg --keyring /var/tmp/githubcli-archive-keyring.gpg \
    --no-default-keyring --export --armor > /var/tmp/githubcli-archive-keyring.asc
rpm --import /var/tmp/githubcli-archive-keyring.asc

but it is an important safety measure.

Not if the keys are mismanaged.

The benefit of apt approach is auto or semi-auto upgrades of the package(s) w/o a need to watch for new releases on e.g. GH pages. So the issue is not affecting GH Workflows only but other apt-based installations too.

👋 For anyone using Codespaces, Remote-Containers, or the dev container CLI, we have updated the dev container feature to prefer pulling from the latest GitHub Release instead of adding the apt repo. (see: https://github.com/devcontainers/features/pull/133)

Example devcontainer.json

{
   "image": "mcr.microsoft.com/devcontainers/base:ubuntu", // Any debian-based image
   "features": {
      "ghcr.io/devcontainers/features/github-cli:1": {}
   }
}

For future reference, supplying the option installDirectlyFromGitHubRelease: false will attempt the traditional approach that adds and fetches the github-cli via apt.

{
   "image": "mcr.microsoft.com/devcontainers/base:ubuntu", // Any debian-based image
   "features": {
      "ghcr.io/devcontainers/features/github-cli:1": {
        "installDirectlyFromGitHubRelease": false
      }
   }
}

curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg \
&& sudo chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg \
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null \
&& sudo apt update \
&& sudo apt install gh -y

Note: We were recently forced to change our GPG signing key. If you’ve previously downloaded the githubcli-archive-keyring.gpg file, you should re-download it again per above instructions. If you are using a keyserver to download the key, the ID of the new key is 23F3D4EA75716059.

If you decide to ditch the packaging, could you at least make the CLI self-updatable? In other words, if I install using some mechanism you show on cli.github.com, then I can do something like gh self-update to get the latest version, without having to figure out how I installed in the first place, or how to update it without ending up in a broken state with multiple competing versions.

But much better would be to just fix the packaging.

The importance of the issue seems to be critical since it breaks all workflows/… containing any related update inside. Probably we can mention @vilmibm here for visibility 😃

With Debian, just follow the installation instructions and you’re done. (You don’t need to delete the previous key)

how fix this …?

Well, the cli still works for now. The only “fix” is to wait for Mr. Nate Smith to publish their updated key.

Re the suggestions of installing the latest package, here’s a, um, one line that does it (sprinkle sudo to taste):

t="/tmp/gh-$$.deb"; curl -sSLo "$t" "https://github.com$(curl -sSL "https://github.com/cli/cli/releases/latest" | grep -Po "(?<=href=\")/cli/cli/releases/download/[^\"]*$(dpkg --print-architecture)[.]deb(?=\")")"; apt -y install "$t"; rm "$t"

This should be fine for short-term installation (= docker images).

For longer term setups (eg, your work machine):

• Put this in a script
• Add it to some APT::Update::Post-Invoke-Success hook
• Optimize to avoid a download of an existing version
• Bandage your forehead, drink some soup
• Tell your next-door neighbor that you’re sorry about the wailing and the wall-thumping noises.

(And just because I suspect that this will be taken seriously, given the lack of a roaring laughter soundtrack when reading the above suggestions: it’s not. This comment said it nicely.)

Good work everyone. I’m running Ubuntu 22.04 LTS and had this issue as well. After doing sudo apt-key del C99B11DEB97541F0 && sudo rm /etc/apt/sources.list.d/github-cli.list and reinstalling through the installation instructions, everything worked perfectly.

@sebma , the only command you miss is sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 23F3D4EA75716059 It is the simplest way to import the new key from trusted root server.

I also received this error on my nvidia jetson xavier when running sudo apt update. To resolve this error, I manually downloaded the gpg key into the directory listed in the signed-by attribute within the /etc/apt/sources.list.d/github-cli.list

Command I ran:

sudo curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg

@aucampia Thanks for sharing the solution for Fedora!

Going forward, we’ll try to manage the new key in a better way— starting with it being owned by the organization, and updating its expiry date when it’s time. I am also learning about the approach of distributing public keys as packages as well, and considering it as public key upgrade mechanism for the future when this one expires.

@elibarzilay thanks for the very handy script! For those of us less fluent in Bash, here’s the same one-liner adapted into a more verbose docker command.

RUN GH_CLI_INSTALL_FILE="/tmp/gh-$$.deb" && \
    DPKG_ARCH="$(dpkg --print-architecture)" && \
    URL_REGEX="(?<=href=\")/cli/cli/releases/download/[^\"]*amd64[.]deb(?=\")" && \
    GH_CLI_INSTALL_FILE_URL=https://github.com$(curl -sSL https://github.com/cli/cli/releases/latest | grep -Po "$URL_REGEX") && \
    curl -sSLo "$GH_CLI_INSTALL_FILE" "$GH_CLI_INSTALL_FILE_URL" && \
    apt install -y "$GH_CLI_INSTALL_FILE" && \
    rm "$GH_CLI_INSTALL_FILE"

Manually, you are able to reset your system with sudo apt-key del C99B11DEB97541F0 && sudo rm /etc/apt/sources.list.d/github-cli.list

It is worth pointing out to any relatively inexperienced users that this is not a “fix” if you want to install using apt. This will remove gh from the apt repository resulting in the following:

• apt will still be able to see any pre-existing install of gh that it did for you
• apt will be able to uninstall any such pre-existing apt-based install
• if no such install exists, or if one exists but apt uninstalls it, gh will no longer be listed in your apt repositories

TL;DR - this is a great way to keep apt from throwing error messages because of the invalid key. If you do this you may want to uninstall any apt-based installations of gh and start maintaining gh by direct download or by using snap.

Many of us in that boat amigo.

Let’s hope that gpgkey exchange/update will be done successfully by @vilmibm @mislav before Sep, 6th 2024 😉

Another note: On Fedora 35 onwards we don’t need the extra repo gh-cli anymore. I found that package gh is in official Fedora repos now.

@dmitsf What image are you referring to? I have an example workflow here that runs apt-get update on a hosted and new self-hosted runner: https://github.com/robherley/tmp-apt-update/actions/runs/2982419946

@robherley I’m using Ubuntu-based image with GH CLI installed via apt-get, and do apt-get update in a workflow. But I see if I’m switching to a binary downloading instead - everything works nice. That resolves the problem!

You can use the --allow-unauthenticated to apt to install the package in the meantime without verification. Of course, this is somewhat of a security concern. You could also download the package directly from the releases page.

Good work everyone. I’m running Ubuntu 22.04 LTS and had this issue as well. After doing sudo apt-key del C99B11DEB97541F0 && sudo rm /etc/apt/sources.list.d/github-cli.list and reinstalling through the installation instructions, everything worked perfectly.

but for me can not again install bcuz not connection to server cli.gh 😭 and i live in iran and can not connect to server for download

@iakov I found it : Just removed the old signed-by instruction which was pointing to the previous path /usr/share/keyrings/githubcli-archive-keyring.gpg :

$ grep github /etc/apt/sources.list.d/*
/etc/apt/sources.list.d/github-cli.list:deb [arch=amd64] https://cli.github.com/packages stable main # gh
/etc/apt/sources.list.d/github-cli.list.save:deb [arch=amd64 signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main # gh
$ sudo apt update -qq
68 packages can be upgraded. Run 'apt list --upgradable' to see them.
$

@iakov Hi,

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 23F3D4EA75716059

I just did what you proposed and the sudo apt update does not work either, same result :

$ sudo apt-key del 23F3D4EA75716059
OK
$ apt-key list 23F3D4EA75716059
$ sudo rm /etc/apt/trusted.gpg.d/githubcli-archive-keyring.gpg
rm: cannot remove '/etc/apt/trusted.gpg.d/githubcli-archive-keyring.gpg': No such file or directory
$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 23F3D4EA75716059
Executing: /tmp/apt-key-gpghome.CYXt2ov1py/gpg.1.sh --keyserver keyserver.ubuntu.com --recv-keys 23F3D4EA75716059
gpg: key 23F3D4EA75716059: public key "GitHub CLI <opensource+cli@github.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1
$ apt-key list github
pub   rsa4096 2022-09-06 [SC] [expires: 2024-09-06]
      2C61 0620 1985 B60E 6C7A  C873 23F3 D4EA 7571 6059
uid           [ unknown] GitHub CLI <opensource+cli@github.com>
sub   rsa4096 2022-09-06 [E] [expires: 2024-09-06]

$ sudo apt update -qq
67 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://cli.github.com/packages stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 23F3D4EA75716059
W: Failed to fetch https://cli.github.com/packages/dists/stable/InRelease  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 23F3D4EA75716059
W: Some index files failed to download. They have been ignored, or old ones used instead.

I think this issue can be closed? Seems it’s resolved for everyone involved

https://cli.github.com/packages/rpm/gh-cli.repo is still not updated to point to the new keyserver entry, so the dnf installation instructions are still broken.

This is ridiculous, signing using someone’s personal GPG key. How can I remove it from apt list? I don’t find it anymore.

See the comments above, @SofianeB : 1, 2. Which depends on how you added the repository and the repository name may be slightly different due to which (Debian based) distribution you are using.

Or, alternatively, you could liekly disable the Repository in your your GUI Package Manager; how you do this depends on which you have installed.

This is ridiculous, signing using someone’s personal GPG key. How can I remove it from apt list? I don’t find it anymore.

I think that much of the confusion is that with apt v2.4 the apt approved way of doing this changed. From “apt changelog apt”:

apt (2.4.0) unstable; urgency=medium
      ...       ...      ...
  [ Julian Andres Klode ]
  * Install an empty /etc/apt/keyrings directory.
    This directory is intended to provide an alternative to
    /usr/share/keyrings for placing keys used with signed-by.
  * Warn if the legacy trusted.gpg keyring is used for verification
      ...       ...      ...
 -- Julian Andres Klode <jak@debian.org>  Tue, 22 Feb 2022 20:00:46 +0100

From “man apt-key”:

DEPRECATION
       Except for using apt-key del in maintainer scripts, the use of apt-key
       is deprecated. This section shows how to replace existing use of
       apt-key.

       If your existing use of apt-key add looks like this:

       wget -qO- https://myrepo.example/myrepo.asc | sudo apt-key add -

       Then you can directly replace this with (though note the recommendation
       below):

       wget -qO- https://myrepo.example/myrepo.asc | sudo tee /etc/apt/trusted.gpg.d/myrepo.asc

       Make sure to use the "asc" extension for ASCII armored keys and the
       "gpg" extension for the binary OpenPGP format (also known as "GPG key
       public ring"). The binary OpenPGP format works for all apt versions,
       while the ASCII armored format works for apt version >= 1.4.

       Recommended: Instead of placing keys into the /etc/apt/trusted.gpg.d
       directory, you can place them anywhere on your filesystem by using the
       Signed-By option in your sources.list and pointing to the filename of
       the key. See sources.list(5) for details. Since APT 2.4,
       /etc/apt/keyrings is provided as the recommended location for keys not
       managed by packages. When using a deb822-style sources.list, and with
       apt version >= 2.4, the Signed-By option can also be used to include
       the full ASCII armored keyring directly in the sources.list without an
       additional file.

FILES
       /etc/apt/trusted.gpg
           Keyring of local trusted keys, new keys will be added here.
           Configuration Item: Dir::Etc::Trusted.

       /etc/apt/trusted.gpg.d/
           File fragments for the trusted keys, additional keyrings can be
           stored here (by other packages or the administrator). Configuration
           Item Dir::Etc::TrustedParts.

       /etc/apt/keyrings/
           Place to store additional keyrings to be used with Signed-By.

APT 2.4.7                      22 February 2022                     APT-KEY(8)

erAck is suggesting that people use the new apt policy, which does require the signed-by directive.

Isu

@mislav Hi, on my Ubuntu 20.04 LTS, I typed this :

$ \curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/etc/apt/trusted.gpg.d/githubcli-archive-keyring.gpg
4+1 records in
4+1 records out
2270 bytes (2.3 kB, 2.2 KiB) copied, 0.0447046 s, 50.8 kB/s
$ ls -l /etc/apt/trusted.gpg.d/githubcli-archive-keyring.gpg
-rw-r--r-- 1 root root 2270 Sep  8 21:54 /etc/apt/trusted.gpg.d/githubcli-archive-keyring.gpg
$ file /etc/apt/trusted.gpg.d/githubcli-archive-keyring.gpg
/etc/apt/trusted.gpg.d/githubcli-archive-keyring.gpg: PGP/GPG key public ring (v4) created Tue Sep  6 13:27:06 2022 RSA (Encrypt or Sign) 4096 bits MPI=0xcf981f067ddf642e...
$ apt-key list | grep /etc/apt/trusted.gpg.d/githubcli-archive-keyring.gpg -A5
Warning: apt-key output should not be parsed (stdout is not a terminal)
/etc/apt/trusted.gpg.d/githubcli-archive-keyring.gpg
----------------------------------------------------
pub   rsa4096 2022-09-06 [SC] [expires: 2024-09-06]
      2C61 0620 1985 B60E 6C7A  C873 23F3 D4EA 7571 6059
uid           [ unknown] GitHub CLI <opensource+cli@github.com>
sub   rsa4096 2022-09-06 [E] [expires: 2024-09-06]
$ sudo apt update -qq
66 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://cli.github.com/packages stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 23F3D4EA75716059
W: Failed to fetch https://cli.github.com/packages/dists/stable/InRelease  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 23F3D4EA75716059
W: Some index files failed to download. They have been ignored, or old ones used instead.

I see the expiration date is 2024-09-06. I don’t understand what is wrong here. Any idea ?

@DogaCUlupinar Strange! The key that apparently wasn’t found looks correct, and I can verify that it’s present on at least one of the keyservers that the github-cli feature uses. Could you report this to the https://github.com/devcontainers/features project?

Yep, I can confirm it looks like it’s working now, once you update the locally-stored public key by following the instructions. Thanks for fixing this - the GPG signing of debian repos is a bit of a pain, yes, but it is an important safety measure. (Also it is not very much like using GPG for any other purpose, at least for automated publication like this.)

it works 🚀 🚀 🚀

...
Unpacking gh (2.15.0) over (2.14.7) ...
Setting up gh (2.15.0)
...
$ gh --version
gh version 2.15.0 (2022-09-06)
https://github.com/cli/cli/releases/tag/v2.15.0

If you’re not using gh (like I just wanted to learn about it) you can remove it from your package list:

1. remove from list of installed packages using synaptic (find gh and select remove);
2. comment out line in /etc/apt/sources.list for “##deb https://cli.github.com/packages bullseye main”

Hello @robherley ! Thank you! I’m not pulling GitHub CLI in a workflow (it’s included to the basic image, self-hosted runner), I mean, if there is, e.g. something like apt-get update in a workflow, this will be affected (the examples that I have and that OP mentioned).