git-secret: GnuPG2 2.2 vs 2.1 conflicts in keybox format

For anyone else arriving here, I created a Docker image that contains git-secret and gpg 2.2 to solve this problem for me. The repo (with usage documentation) is here:

https://github.com/ncpierson/git-secret

I get why the external format of a key is stable (people exchange keys) yet the key ring format may not be forwards compatible (people upgrade, but dont downgrade, and it’s unusual to be sharing key rings rather than keys).

How about we switch to storing keys in the public key format and don’t commit the keyring that can “cache them” for repeated use:

1. git secret tell exports the key into a .gitsecret folder in the standard export format
2. git secret hide tries to encrypt using a keyring marked as .gitignore so as not to be committed:

• If there is isn’t yet a keyring at the ignored location initialise it
• Speculatively try to encrypt with every identity. If there is a “key not found error” load the corresponding public key into the ignored keyring and retry

The advantages of this approach should be:

• if the user upgrades to some future gpg then that is still likely the able to load the keys in their local ignored keyring. The upgrade doesn’t pollute any users ignored keyrings.
• If a user downgrades gpg and can no longer open the ignored keyring they simply delete it. We could provide a git secret clean command to do that. Meanwhile it is a problem for only for that one user.
• teams can all use different distributions of gpg and pgp as long as all versions understand the public key export format which is typically stable and new software will be tested against old key formats

That’s just a rough sketch I may have missed something but fundermentally using a shared key ring rather than exported public keys is an “optimisation” over the approach above. Our current “optimisation” of a shared public key ring is causing teams who use different distributions or operating systems a compatibility challenge. So we should “downgrade” our approach to have logic that uses a local ignored keyring as a “cache”.

@joshrabinowitz @sobolevn @notjames From my perspective, there’s not an ideal solution to this from the git-secret side of things: upstream broke something they shouldn’t have. The least bad solution, imo, is to ignore the most specific error code you can ignore so long as the command seems to have worked, and then maybe issue a warning on STDERR. It’s never fun to have workarounds in your code for weird stuff like this, but I’d say that was the best bet at working for something that’s likely to be a fairly common failure mode without accidentally ignoring real problems.

@simbo1905 @sobolevn I like the RFC, I think it’s a big step forward. Would love to see 0.2.x released before this development starts though!

Because we do not want to pollute user’s main keyring with our keys. That was the main point for me in the past.

That’s good input about documenting how to export and import keys, I’ll add a ticket about that @emacdona

For bonus marks I think we can make the new approach forwards compatible with the current approach. Let’s say we version the new approach 0.3.0. We want it to be forward compatible with the current 0.2.x. It’s likely that any team that stated on 0.2.x will have one new team member join who first installs 0.3.0:

1. The new version will look for .gitsecret/cache and if it doesn’t exist tell the user “This repo was initialised by a previous version of git-secret, please run git secret migrate to upgrade. The migration will keep backwards compatibility with older versions.”
2. migrate can export out all the keys from the old keyring into the new format described above. It can the create the cache folder, add it to .gitignore, and create the ignored personal keyring “cache” within that folder, and load in all the exported keys into it.
3. The new version of will now test for the existence of the old public shared keyring when running commands. If it finds one it wil run the old logic to keep it updated.

This would mean:

• Each user of an existing project that upgrades to 0.3.x will get a more stable experience
• Users still on 0.2.x will work as now. They can still hit the keyring compatibility issue. The first thing they are likely to do is upgrade to the latest 0.3.x which will solve their problem.
• Teams that all start out on 0.3.x will just execute a single file exists check on the old keyring location and skip the backwards compatibility logic.

We will have to maintain the old logic in the code base for a while. We can give a deprecated warning and at some future 0.4 (or 1.0!) drop that logic. I think backwards compatibility is worth the effort though as the new logic simply needs to runs the old logic and have a migrate step. We can keep an eye on how much maintenance the old logic needs and make a call to ditch it sooner rather than later if the burden is high.

So how do we make the decision to adopt the proposed approach? One idea is I could make an “RFC001.md” containing the described approach and send a PR. We can then do a code review of it and update it and use the usual “LGTM” approach. Once we are happy to merge to master its the new design. Then we can open a fresh ticket to implement it. That would avoid these long and heavy comments in the future!

I think what @simbo1905 is suggesting is that .gitsecret/keys contain one or more files with text representations of the public keys of the users that can access the repo, and that these text file(s) would get checked in to git.

The file(s) of public keys in text representation would be portable across gpg versions. (You can see a public key in text representation with: gpg --export --armor email@id.)

The actual gpg keychains used on each given system (for each repo using git-secret) would be created, cached, and re-created from the public keys (and at least one of the needed private keys, which are probably in the user’s keychain in ~/.gnupg) as needed.

The idea is that keychain files, which differ across gpg versions, would not be checked into the git repo.

In this way git-secret would avoid issues with different systems’ versions of gpg reading and writing files in .gitsecret/keys differently, as these files would not get checked into the git repo or shared between systems.

@notjames as I understood @simbo1905’s suggestion, we are not going to use the default keyring in $HOME. We are going to still use the same keyring location as we use now.

But, this keyring will be ignored by the git. This way we can recreate a keyring on every machine with just importing the public keys.

I like the idea of moving away from managing keyrings by ourself. Currently I can not even see any disadvantages in this approach.

@joshrabinowitz I’m actually fairly booked up for the next couple of days. Can you give me a 50,000 ft overview of what import / export actually involves?

@joshrabinowitz @sobolevn I like that idea. Depending on whether gpg -n --list-keys returns a unique error code for this specific case, it may be possible to just check for that instead of parsing stdout/stderr (since parsing that kind of thing is always a hassle), but I have a suspicion it will need to be the stdout/stderr approach.

@gthank @sobolevn

So, is this the desired logic?

• capture the exit code, stderr and stdout of the gpg -n --list-keys command,
• if captured output consists just of lines like gpg: skipped packet of type 12 in keybox
• then output warning about the issue (without exiting, regardless of exit code)
• check that it looks like we got a list of keys as expected
• if we did not, issue error and exit

Perhaps related: https://github.com/StackExchange/blackbox/issues/222 https://lists.gnupg.org/pipermail/gnupg-devel/2017-May/032846.html