go: proposal: cmd/go: ignore +incompatible versions as of Go 1.14

Abstract

This is a proposal to ignore +incompatible versions found in the module graph starting with Go version 1.14.

Background

The go command requires that the import path of a module (or package within a module) match its semantically-versioned API. In particular, starting with major version 2 — the first breaking change from the API stabilized in major version 1 — the module path must end with a /vN suffix indicating the major version of its API.

However, prior to the introduction of modules, many Go package maintainers had already reused existing import paths (such as github.com/google/go-github) across multiple major versions. To accommodate the migration to modules for users of those packages, the vgo prototype — and the initial module support released in the go command in Go 1.11 — allowed those existing major versions to be used directly, and even preferred them over compatible versions (#26238) under two conditions:

1. The module with the incompatible version must not contain an explicit go.mod file.

2. The version must be annotated in the user’s go.mod and go.sum files with the suffix +incompatible, indicating that the selected version is not compatible with the original API for that path.

Unfortunately, those exceptions introduce a number of problems:

• #34189: If a user adopts modules (by adding an explicit go.mod file) and then accidentally tags a pre-module commit with an inappropriate major version, the erroneous pre-module tag will always be semantically higher than the highest valid module-enabled release tag (1.99[…].99[…]).

• #27009: If a user has already tagged a major version above 1 and adopts modules (by adding an explicit go.mod file), then they must either adopt the major-subdirectory layout for their project (disrupting developers on the project), or break upgrades for non-module users (who will be expecting the unversioned import path).

• #31543: Since the damage of introducing breaking changes is already done, users often expect that their v2-or-higher repository can be migrated to modules without changing its import path, but for various reasons that does not work in the general case.

• #33795, #32695: Since +metadata tags in general are not semantically meaningful, the fact that +incompatible is semantically meaningful requires numerous special cases that are difficult to test and maintain, and creates confusion about when incompatible versions are or are not allowed.

• #29731: The possibility of using a +incompatible version led us to support major-version wildcard queries, which are only useful for legacy repos and interfere with more useful branch queries.

• #34254: The constraints on +incompatible versions derive from the module path, but that introduces even more complexity (and makes things more difficult to debug) with replace directives, which involve two module paths that may or may not impose the same constraints on the corresponding versions.

---

In contrast, for another interesting case of legacy tagging — semantic versions with metadata (#31713) — we came up with what I believe is a simpler solution: instead of accepting the non-canonical version tags as-is, we instead rewrite them to canonical pseudo-versions with an appropriate major version.

In light of the problems we have encountered with incompatible major versions, I believe that we should have applied a similar strategy for incompatible versions: perhaps using them for “latest” version resolution, but rewriting them to canonical pseudo-versions.

Unfortunately, the decision was made, and cannot be unmade in light of our subsequent experience without breaking compatibility.

---

…or can it?

Observation

Since a +incompatible version cannot have an explicit go.mod file, it cannot impose any transitive requirements on module selection. Therefore, a +incompatible version selected as the minimal version of a module cannot impact the version selected for any other module.

This implies that if we ignore the +incompatible versions in the module graph entirely, we will not accidentally drop requirements that pertain to other modules.

This leads to the following proposal (see the comment below; updates will be linked from here).

CC @jayconrod @thepudds @hyangah @katiehockman @heschik