go: cmd/go: preserve basic GOPATH mode indefinitely
In Feb 2021 (https://go.dev/blog/go116-module-changes), we wrote:
It’s still possible to build packages in GOPATH mode by setting the
GO111MODULEenvironment variable tooff. … We plan to drop support for GOPATH mode in Go 1.17. In other words, Go 1.17 will ignoreGO111MODULE. If you have projects that do not build in module-aware mode, now is the time to migrate. If there is a problem preventing you from migrating, please consider filing an issue or an experience report.
The time has come to decide what to do about GOPATH mode. There are at least two important problems with maintaining GOPATH mode as it exists today:
-
The old, GOPATH-mode
go getis being left behind as far as security improvements like the module proxy, checksum databases, and so on. All the focus is on modules. -
The old, GOPATH-mode source layout does not provide a way to identify the language version used by the source code. There is a bit of divergence here in that a go.mod with no go line assumes Go 1.16, while in GOPATH mode we assume the latest version of Go. This means if someone has downloadable packages they develop in GOPATH mode, they can use language features introduced after Go 1.16, like generics, but when clients download that code in module mode as a “legacy” module, the code is interpreted as Go 1.16 and does not compile. The divergence will become greater over time as other language features or changes are made, such as #60078.
On the other hand, GOPATH mode remains the only way to work on dependency-free legacy packages that existing module code may still depend on (so-called “+incompatible” modules), and it remains the only way to build historical source code that predates modules, especially code that depends on the multiple layers of vendoring that Go 1.5 introduced (and modules removed).
Given these tensions, I propose that we do the following:
- Commit to preserving the ability to build GOPATH-layout source trees when GO111MODULE=off, indefinitely.
- Disable go get completely in GOPATH mode, since there is increasingly little code that it can download successfully and provides a worse and worse experience.
- Assume a language setting of Go 1.21 in GOPATH mode, instead of continuing to assume “the latest Go version”. This will ensure that if we change for loops in Go 1.22 (#60078) or make other changes in the future, the old, legacy code we are aiming to keep compiling does keep compiling as written.
We would make these changes for Go 1.22. The GOPATH mode enabler would continue to be GO111MODULE=off rather than introducing a cleaner name like GOMODULE=off, because there’s little point to upsetting whatever scripts people with legacy trees have already written.
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 24
- Comments: 19 (17 by maintainers)
An anecdotal data point: My project at work still depends on GOPATH mode to build. Dependencies are managed with a 2015 era tool that automates populating the GOPATH with the desired versions of dependencies, but provides little help deciding what the versions should be. We would have upgraded to Go modules long ago had it been easy, but due to the size of the project, the accumulated complexity of its dependency tree, and other reasons it hasn’t happened yet. Not for a lack of desire, by the way. The project policy for the toolchain version is to stay one major version behind, so it’s currently built with Go 1.19 and would upgrade to 1.20 shortly after 1.21 is released.
Fortunately we have been able to remove some of the barriers to adopting modules over the last several months and I believe we will be upgraded sometime in the next several months. But there may be other projects similar to ours that are not publicly visible, that still need GOPATH, are still actively worked on, and regularly upgrade the toolchain, but are farther away from using modules. Continuing to support GOPATH mode helps projects like that.
Speaking as Debian packager, I really appreciate that upstream can keep supporting GOPATH for
go build. I’m sorry to say that Debian still relies on it. We don’t usego getso it’s fine to drop GOPATH for it.Being able to run old (self-contained) code that predates modules is also worthwhile. That requires having GOPATH build support. Keeping build working is easy and it has real benefits.
The main quirks remaining to support GOPATH mode have to do with how we resolve dependencies in
vendordirectories, although even that has some overlap with how we support the vendored dependencies of the standard library.Beyond that, as Russ mentions, the
go getimplementation for GOPATH mode is complex and fairly subtle.Speaking as one of the primary maintainers of
cmd/go, I feel that the cost/benefit tradeoff supports Russ’s proposal. Thego getsupport is subtle and not worth the cost to maintain, but maintaininggo buildsupport for the time being is fairly inexpensive and may be valuable for the long tail of migrations (especially for enterprise users).Linking some related proposals here for posterity’s sake:
Based on the discussion above, this proposal seems like a likely accept. — rsc for the proposal review group
@willfaught It would not be a single module. The project has dozens of repositories that will each become a module. About a dozen of those contain packages depended on a few levels deep by the rest of them. We’ve already invested hundreds of man hours to get things into a state that will make the transition easier (while being careful not to break things or interrupt regular development). At this point the source code is in a good state for a migration. I’ve already written a tool that can create an equivalent
go.modfrom the file format our existing tool uses and we have a map of our internal dependencies.The biggest hurdle for us now is that the tool we’ve been using doesn’t enforce repeatable builds. It allows depending on a branch head. We only use that for internal dependencies, but that implies a certain workflow that developers are used to and that our CI systems have been adapted to. Next steps are working out how to adapt our CI systems and developer workflows to the fact modules don’t allow depending on branch heads. Trying things out in a small corner of the project where problems will have minimal impact, then educating people. Prior to the addition of workspaces (go.work) in Go 1.18 the workflow problem looked much more daunting, so I’m glad we have that tool available.
The issue has not been management being focused on shiny new features. The project was part way through a migration three years ago, but that was aborted after a key staff member left. I joined the project two years ago and was told almost immediately by management that adopting Go modules was a goal. It’s been on the roadmap for a while. But it’s a big and old code base with skeletons in many closets that we’ve been carefully working through.
It’s really not a matter of 8 hour focus days as much as managing the smooth evolution of an entrenched social-technical system into a better state.
What examples are there of GOPATH packages we want to keep working with the latest toolchains?
The steps to convert GOPATH packages to modules are:
Unless I missed something, that’s not much effort. This is basically equivalent to forking a package that you know will be deleted soon. In a sense, GOPATH packages would be “deleted.” Anyone who truly cares can fork their own copy, and turn it into a module for others to use too.
How much simpler would the toolchain be without GOPATH support?