flexo: signature from is invalid
Hey,
Sometimes when using flexo, I’ve noticed errors when my machine is validating packages in the form of:
error: <package>: signature from "<person>" is invalid
:: File /var/cache/pacman/pkg/<package>.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n]
Neither Y nor n fixes the problem. Clearing out the flexo cache does.
Re-downloading the archlinux-keyring also does not fix this issue.
Could this be a stale package cached in flexo somehow? Is there any mitigation for this instead of clearing out the flexo cache?
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 23 (11 by maintainers)
Commits related to this issue
- Increase logging verbosity To troubleshoot #93 — committed to nroi/flexo by nroi 2 years ago
Thanks for providing the logs, @exalented!
actually, 206 is the expected response when the client requests just part of the file. For example, when pacman initiates a download from flexo, then aborts it, and later on downloads the same file again, the expected behavior is that flexo resumes the download, so it sends a range request to the remote mirror and the remote mirror responds with 206. 416 is the response when the client for some reason sends a wrong range request that cannot be fulfilled by the server.
I wasn’t able to completely figure out what exactly is happening for what reasons from the logs, but something that definitely looks wrong is this:
The header we have received (1st line in the logs) is this one:
Notice it actually says “try again” in the log, so the re-try logic is already in place. And yet, in almost the exact same time, we have this in the logs:
So, even though we haven’t received the required payload from the server, we just serve whatever trash to the client (probably an empty file or an HTML file saying “Not found”), and only after we served this trash to the client, flexo fetches the actual content from the remote mirror.
This does not explain everything, but it’s a start.
Thanks again for providing the logs, I will look into it.
This showed up again for me. I still haven’t figured out how to reliably reproduce it, but it does seem to be failing with an HTTP 416. Thanks for looking into this!
flexo-verify-package output
Note: the modification time here is correct. I dumped the logs a while back and forgot about them until now…
docker logs output
Here are the differing .pkg.tar.zst files: package_tars.zip
One of flexo’s goal is to minimize the time users need to invest for mirrorlist maintenance, and not everyone uses reflector. But of course, it would be possible to implement this as an additional option.
Thanks for the journalctl output.
Keep in mind that flexo ignores
/etc/pacman.d/mirrorlist, it fetches a list of mirrors from the official JSON API and then runs latency tests to select fast mirrors. So, if you want to avoid flexo choosing bad mirrors, you need to use themirrors_blacklistoption, as explained in my previous post.So it just happened again. This happens on the client machine, during the
Checking package Integritystep:The
sha256sumbefore clearing out the package cache for all matching files is:I cleared out the pkg cache and restarted the flexo service, ran
pacman -Scon the client, and re-ran the update on the client. Running thesha26sumcommand on the flexo server again, I see:Edit: This is either caused by misbehaving mirrors or a bug in flexo, but I think a bug in flexo is more likely. I’m looking into this issue, but I still haven’t found a way to reproduce it. If anyone stumbles on this issue, you can help me troubleshoot this issue as follows:
docker logsif you use docker) from the time frame when the download has happened, e.g.journalctl --unit=flexo --since today.The more output I receive, the better!
Original answer:
I guess so. Probably either the package itself, or the signature (the corresponding file ending with
.sig) has changed, but flexo serves the stale version from cache.I’d like to find out if the package file (the file ending with
.zst), or the signature file (ending with.sig), or both files are stale. Next time when this issue happens, can you please check which files change after you clear out the flexo cache? For example, you can runsha256sum <package>.zstandsha256sum <package>.zst.sigbefore clearing the flexo cache (replace<package>by any package from the error message shown by pacman), and then run the same two commands again after clearing the flexo cache to compare the checksums and see which files have changed.Once I understand the problem, I can think of a solution.