dist: Errors when installing using the install guide

Workaround; please use this until the docs update is pushed live:

curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg

I found a solution.

chmod 664 /usr/share/keyrings/caddy-stable-archive-keyring.gpg

The problem was: executing the install script as root -> sets restrictive file permissions, which apt can’t use

I can confirm that “Some people have stated that running the commands as root breaks the key, and a chmod 664 is required on the key.” fixed it for me (was running the script as root).

I did get the same output:

ubuntu@ip-10-0-48-165:~$ sudo gpg --show-keys --fingerprint --with-subkey-fingerprint /usr/share/keyrings/caddy-stable-archive-keyring.gpg
pub   rsa4096 2016-04-01 [SC]
      6576 0C51 EDEA 2017 CEA2  CA15 155B 6D79 CA56 EA34
uid                      Caddy Web Server <contact@caddyserver.com>
sub   rsa4096 2020-12-29 [S] [expires: 2025-12-28]
      2F5C 3BE9 886A CD29 1329  9EFB ABA1 F9B8 875A 6661

Spending too much time on this…I just decided to download the binary from https://caddyserver.com/download and it seemed to work. 🤷‍♀️ Thanks for trying to help.

ubuntu@ip-10-0-48-165:~$ ./caddy
Caddy is an extensible server platform.

usage:
  caddy <command> [<args...>]

commands:
  adapt           Adapts a configuration to Caddy's native JSON
  add-package     Adds Caddy packages (EXPERIMENTAL)
  build-info      Prints information about this build
  environ         Prints the environment
  file-server     Spins up a production-ready file server
  fmt             Formats a Caddyfile
  hash-password   Hashes a password and writes base64
  help            Shows help for a Caddy subcommand
  list-modules    Lists the installed Caddy modules
  reload          Changes the config of the running Caddy instance
  remove-package  Removes Caddy packages (EXPERIMENTAL)
  reverse-proxy   A quick and production-ready reverse proxy
  run             Starts the Caddy process and blocks indefinitely
  start           Starts the Caddy process in the background and then returns
  stop            Gracefully stops a started Caddy process
  trust           Installs a CA certificate into local trust stores
  untrust         Untrusts a locally-trusted CA certificate
  upgrade         Upgrade Caddy (EXPERIMENTAL)
  validate        Tests whether a configuration file is valid
  version         Prints the version

Use 'caddy help <command>' for more information about a command.

Full documentation is available at:
https://caddyserver.com/docs/command-line

Hey @lskillen Thanks so much. (Sorry I was out sick for a few days.) We’re very happy with Cloudsmith and would be able to help promote it more.

Hey folks, Lee from @cloudsmith-io here. First of all, sorry for the breakage; that’s never a fun thing to hear about. 😦

Talking to the team here and this change was made based on the previous apt-key usage being deprecated due to security issues (and then no longer supported on newer distributions going forward). It was likely a bit of a kneejerk reaction because of the impending impact, as there’s a small blog/product update on the way that hasn’t been made live yet to explain it.

Part of the issue here is the compatibility to smooth this over was put into the automated script only, which Caddy isn’t using. That’s fine, of course, but it also means users with manual instructions need to update too. The blog will cover this, but the new instructions can also be found in the contextual setup within the repositories.

E.g. For our “demo” public repository, you can see the new instructions here: https://cloudsmith.io/~cloudsmith/repos/examples/setup/#formats-deb

In short (again, for the examples repository on Cloudsmith so don’t use this for Caddy anyone else):

apt-get install -y debian-keyring  # debian only
apt-get install -y debian-archive-keyring  # debian only
apt-get install -y apt-transport-https
# For Debian Stretch, Ubuntu 16.04 and later
keyring_location=/usr/share/keyrings/cloudsmith-examples-archive-keyring.gpg
# For Debian Jessie, Ubuntu 15.10 and earlier
keyring_location=/etc/apt/trusted.gpg.d/cloudsmith-examples.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/cloudsmith/examples/gpg.0365B910DDF4E7A4.key' |  gpg --dearmor > ${keyring_location}
curl -1sLf 'https://dl.cloudsmith.io/public/cloudsmith/examples/config.deb.txt?distro=ubuntu&codename=xenial' > /etc/apt/sources.list.d/cloudsmith-examples.list
apt-get update

I think you’ve pretty much got that as your current workaround, though!

Again, sorry for any inconvenience, pain and/or annoyance caused. 😃

/cc @francislavoie @mholt

Oh… looks like CloudSmith changed the format of the debian.deb.txt file 😬

It used to look like this:

# Source: Caddy
# Site: https://github.com/caddyserver/caddy
# Repository: Caddy / stable
# Description: Fast, multi-platform web server with automatic HTTPS


deb https://dl.cloudsmith.io/public/caddy/stable/deb/debian any-version main

deb-src https://dl.cloudsmith.io/public/caddy/stable/deb/debian any-version main

Now it looks like this:

# Source: Caddy
# Site: https://github.com/caddyserver/caddy
# Repository: Caddy / stable
# Description: Fast, multi-platform web server with automatic HTTPS


deb [signed-by=/usr/share/keyrings/caddy-stable-archive-keyring.gpg] https://dl.cloudsmith.io/public/caddy/stable/deb/debian any-version main

deb-src [signed-by=/usr/share/keyrings/caddy-stable-archive-keyring.gpg] https://dl.cloudsmith.io/public/caddy/stable/deb/debian any-version main

So I think we need to adjust our install script to write the GPG key elsewhere.

/cc @lskillen

(I moved the issue to the dist repo, which is the right place for this)

Note for people that still have problems with this:

Keep in mind that the keyfile must have the “read” permission for “others” (e.g. by doing chmod o+r) (as implicitly mentioned above with the chmod 664 suggestion), but also that all parent folders of the keyfile must to have the “execute” permission for “others” (e.g. by doing chmod o+x). So, for example, if you put the keyfile in a folder you created yourself using the root account, you likely need to assign the missing permission as well.

👀 Going to ask the team to have a look. I tried the standard route and it was fine (Ubuntu 20.10).

Then I decided I would be a bit more manual and use gpg directly to see what it reports.

Verifying without the public key:

gpg --verify <(curl -1sLf https://dl.cloudsmith.io/public/caddy/stable/deb/debian/dists/any-version/InRelease)
gpg: Signature made Tue 12 Jul 2022 07:55:46 PM BST
gpg:                using RSA key ABA1F9B8875A6661
gpg: Can't check signature: No public key

Importing the key:

gpg --import <(curl -1sLf https://dl.cloudsmith.io/public/caddy/stable/gpg.key) 
gpg: key 155B6D79CA56EA34: public key "Caddy Web Server <contact@caddyserver.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1

Verifying with the public key:

gpg --verify <(curl -1sLf https://dl.cloudsmith.io/public/caddy/stable/deb/debian/dists/any-version/InRelease)
gpg: Signature made Tue 12 Jul 2022 07:55:46 PM BST
gpg:                using RSA key ABA1F9B8875A6661
gpg: Good signature from "Caddy Web Server <contact@caddyserver.com>" [unknown]
Primary key fingerprint: 6576 0C51 EDEA 2017 CEA2  CA15 155B 6D79 CA56 EA34
     Subkey fingerprint: 2F5C 3BE9 886A CD29 1329  9EFB ABA1 F9B8 875A 6661

• Primary key: 155B6D79CA56EA34 which matches 155B 6D79 CA56 EA34 above.
• Sub key: ABA1F9B8875A6661 which matches ABA1 F9B8 875A 6661 above.

So it all checks out, however, noting a couple of observations:

• It doesn’t seem to fail everywhere (and I can’t personally replicate it, which is annoying).
• We haven’t heard of any other users report of NO_PUBKEY but that might be somehow related to the method.
• Some people have stated that running the commands as root breaks the key, and a chmod 664 is required on the key.

…

I did go off and check what our automated script is doing, and although I don’t think it is related, it’s worth saying that we vary the key functionality depending on the version of apt installed. If the user has a version of apt that is at least 1.1, then we use the standard signed-by, otherwise it gets copied to /etc/apt/trusted.gpg.d. However, since apt 1.1 is from, what, 2016 or around there, it just seems unlikely this is why.

Example:

function import_gpg_key {
    local text="Importing 'caddy/stable' repository GPG key ..."
    echo_running "$text"

    local gpg_keyring_path="/usr/share/keyrings/caddy-stable-archive-keyring.gpg"
    curl -1sLf "https://dl.cloudsmith.io/public/caddy/stable/gpg.155B6D79CA56EA34.key" | gpg --dearmor > $gpg_keyring_path

    local signed_by_version="1.1"
    local detected_version=$(dpkg -s apt | grep Version | cut -d' ' -f2)

    [ "$(printf "%s\n" $detected_version $signed_by_version | sort -V | head -n 1)" == "$signed_by_version" ]

    echo_okfail "Checking for apt signed-by key support ..." || {
        mv ${gpg_keyring_path} /etc/apt/trusted.gpg.d/caddy-stable.gpg
    }

    echo_okfail "$text" || die "Could not import the GPG key for this repository"
}

That code is a bit clunky, but it is sorting the apt version and the expected version, and see if the expected version is older than 1.1 (because it would appear first if 1.0 or something).

Docs update is live: https://caddyserver.com/docs/install#debian-ubuntu-raspbian

Just would’ve been nice to have gotten an alert/email ahead of time to let us prepare a fix, but it’s okay, the change makes sense 👍

💯 That’s an oversight on our part, and definitely not intentional. Normally we would communicate any change that may be breaking (it turns out that in the artifact/package management world, almost everything is a breaking change!) In this case, I think it was because of the belief that it wasn’t a breaking change due to the script, so that’s a learning lesson. Sorry again. Good excuse to signup for notifications on the changelog, and for us to be more vigilant. 😃

The fix works perfectly for us, thanks for the prompt reply and have a fantastic day!

Alright, I merged an update to the docs site in https://github.com/caddyserver/website/pull/219, just need @mholt to push it live as soon as he can.