go: proposal: cmd/go: use the `go` version declared in the `go.mod` file to determine module boundaries and checksums

@marwan-at-work, proxies should forever use go mod download. The reason it exists is precisely so all proxies and other downloaders agree on the bits and don’t have to reimplement all the different version control mechanisms themselves, or even have stale libraries linked in.

@kardianos Part of the point of this proposal is that older clients won’t need to upgrade, as long as they’re getting their modules from an up-to-date module proxy. If the hash function hasn’t changed and the zipfile format hasn’t changed, why should the client consuming that zipfile and hashing it with that function need to change?

But perhaps we could adjust the behavior a bit. Perhaps we should only fail to record checksums for newer versions, but still verify them: we could emit the “you need to upgrade” warning instead of “checksum mismatch” only if the checksum fails and the required version is newer (as in #28221).

Then, the cases would be:

• If the proxy returns a zipfile extracted using the required go version, verify that the zipfile matches the checksum and record the checksum, even if the client does not support that go version.
• If the client supports the required go version but the proxy does not, fail with an explicit upgrade warning (or fall back to another proxy or fetch from the origin if so configured).
• If neither the client nor the proxy supports the required go version, but the go.sum file has a checksum entry for the given module, fetch the module (from the proxy, if so configured) using the closest supported semantics.
  • If the checksum matches, use the module: any changes to the extraction algorithm must not affect that module.
  • If the checksum does not match, do not use the module, and fail with an upgrade warning.

The interesting case is:

• If neither the client nor the proxy supports the required go version, and the client does not have a checksum entry for the module, what should we do?
  • Fail open: download the module (from the proxy or origin) using the closest supported semantics, but do not record the checksum, and replace the downloaded copy of the module as soon as the go command is upgraded (or when go mod verify is run).
  • Fail closed: emit an upgrade warning and refuse to download the module.
  • We should not do this, but for completeness, the third option is to “fail corrupted”: download and use the module using the closest supported semantics, and record the resulting checksum in the go.sum file.

On the other hand, it might be nice for older-version clients to be able to record the checksums for probably-incorrectly-extracted versions as well. (If we know that “the copy of module m when extracted with Go 1.12” has checksum X, then it should continue to have checksum X when extracted with Go 1.12, even if the copy extracted with Go 1.13 has a different checksum.)

However, I would argue that that should be part of the path information stored in the go.sum file, not the hash algorithm: any client that understands the algorithm, regardless of version, ought to be able to compute and verify the checksum for a given zipfile.

So perhaps we could add some sort of path suffix (like we do today for the go.sum file) to allow for multiple different extraction algorithms. I’m just not sure that extension would be worth the extra complexity, given that at least the public module proxies should always have the latest release available to use.

@kardianos

The hash version should reflect the input, not just the algorithm.

Note that the “input” in this case may be a .zip file provided by a proxy.

If a user with an older go toolchain obtains a zipfile from a proxy, and the proxy indicates that the file was extracted using the version from its module’s go.sum file, then the older go command ought to be able to verify that the contents match the checksum, even though the older toolchain cannot produce that zipfile itself.

If we change the name of the hash function, then that property will not hold: the older go toolchain would not know whether it has the correct hash function, so it could not flag a checksum error if the zipfile doesn’t match. That would allow a compromised proxy to serve arbitrary contents to older clients.

But the problem is even worse than that. Since the extraction algorithm changes the contents of the zipfile itself, a zipfile extracted using a newer Go version wouldn’t necessarily match the checksum computed by an older version, and the files needed to recompute the correct checksum wouldn’t necessarily even be present in the file (the change to the algorithm may have pruned them out). So we would still need some mechanism for older clients to determine whether to record their own checksum: changing the name of the hash function is still only a partial solution.

@bcmills I think your alternative proposal is spot on:

• Alternately, we could change the checksum prefix from h1: to h2: — even though the checksum algorithm itself doesn’t need to change — to indicate the reliability of that checksum.

I think the key insight is a hash is only valid if you both define the hash algorithm and what the input is defined to me. In simple situations, such as a hash of a file or chunk, this is easy, you define the hash to be the sequence of bytes of the file.

But in Go, this hash is used over many file over many folders. The hash version should reflect the input, not just the algorithm.

I could easily see people changing an embedded go version in tools or manually for one reason or another and mostly of the time nothing would break. But then in a corner case (hash change), doing so would break go.sum. Don’t make this a hidden dependency, tie directly to the hash version.