go: cmd/go: go get should not add a dependency to go.mod

It’s not about the documentation. It’s a complete change to what go get has meant for the last 6+ years. Which has meant “download and compile some code so I have it locally”. That’s still a useful command to have in the universe of go modules.

Changing what an established command does, in such an important way, is a really bad idea.

I think it’s pretty clear, if I go get golang.org/x/tools/cmd/goimports … what I mean is “download and compile goimports, and put it somewhere for general use”. The fact that this command also stealthfully adds golang.org/x/tools/ to my current directory’s dependencies is surprising and extremely unwelcome.

Like I said in the bug report… we have a command for dealing with modules. Why would we not just add this functionality there? go mod add github.com/... is perfectly clear, and wouldn’t clash with current usage.

I was also surprised by this behavior. The suggestion that go get will have a different meaning in a “modules world” doesn’t solve the use case of go get golang.org/x/tools/cmd/goimports or installing any other tool. Do I need to change directories every time I need to install a new tool? What if a go.mod is created anywhere I am because in the modules world any place is a valid place for a Go project?

Regardless of these examples, what bothers me the most is that we’re using the same command that has existed for several years to do a different thing. Are we all supposed to accept that every tutorial, every doc, everything that has ever been said about go get is not valid anymore?

The global place for installing any library or tool still exists, so why do we need to do something special to get back the behavior that already exists?

I think it’s pretty clear, if I go get golang.org/x/tools/cmd/goimports … what I mean is “download and compile goimports, and put it somewhere for general use”.

In a GOPATH world I would agree. But we’re talking a modules world here. From my perspective I’m fully anticipating any go command to do something with my go.mod.

Put another way (and this is where https://github.com/golang/go/issues/24250 comes in), I would expect, in a modules world, to have to do something “special” in order to performa a “global” install (by definition outside the context of my go.mod)

Have to chime in that that I agree with @natefinch and @vdemario here. Using go get for installing tools has been the standard practice since go get was created. Changing that semantic is extremely problematic from a usability perspective, especially since the Go tooling currently lacks a command equivalent to “download this Go library, compile it, and install it somewhere I can use, but do not add it to my current project”.

We’re adding entirely new behavior via the go mod command. It seems to make the most sense to have explicitly adding a dependency to be a subset of that command, rather than hijacking an existing command and completely obliterating one of the two most common ways that command is currently used.

@natefinch, here’s my mental model for why this makes sense. I hope it will help you.

There must be some answer to the question of “which versions of which libraries are available to a build?” The answer used to be “the specific copies in GOPATH”. That is, the entire GOPATH tree was in some sense equivalent to a traditional dependency manager lock file. (Tools like godeps made this more explicit, providing conversion back and forth between the exploded GOPATH and a single file recording versions.) How did we add new libraries or modify which versions were being used? go get.

In the new module world, GOPATH is gone, and the answer to which versions of which libraries are available is the go.mod file. In essence, each go.mod is now a description of a “virtual GOPATH” containing the specific modules and versions listed in the go.mod and their dependencies. We used to say that GOPATH defined the workspace; now go.mod defines the same sense of workspace, although we no longer use that term.

Just as “go get” once pulled code into, or updated code already in, the GOPATH workspace, now it does the same for code in the go.mod-defined workspace. It used to operate by writing file trees in GOPATH. Now it effects the same changes by writing lines in go.mod.

I hope that viewed this way it’s clear that “go get” is doing what it has always done: set up which specific packages are going to be used by a future build.

I agree with this. I’ve already been surprised three separate times now by running go get and having tools unexpectedly added to my go.mod, or having a new go.mod file dropped into a directory where I didn’t intend.

Before modules, go get was used for two pretty distinct use cases: (1) downloading tools and (2) downloading libraries. It seems much, much less surprising and user-friendly for post-modules go get to specialize on use case (1) and leave (2) for other tools. In my experience with modules so far, I’ve handled (2) by simply adding new imports and running go test/go build.

There must be some answer to the question of “which versions of which libraries are available to a build?” The answer used to be “the specific copies in GOPATH”. That is, the entire GOPATH tree was in some sense equivalent to a traditional dependency manager lock file. (Tools like godeps made this more explicit, providing conversion back and forth between the exploded GOPATH and a single file recording versions.) How did we add new libraries or modify which versions were being used? go get.

No, not really.

I mean, some people did, but as I explained in https://github.com/golang/go/issues/27643#issuecomment-420767420 any serious team-based Go workflow before modules didn’t involve using go get for managing application dependencies. People used vendoring, or godep, or some other system.

In that world, the GOPATH is a handy global workspace for downloading and experimenting with libraries, scratch code, and (most importantly) installing Go tools globally.

In the module world, there is no equivalent way of saying “please install this tool globally, unrelated to the working directory I happen to be in”, and attempting to use the old method has the frustrating side effect of editing the current module.

You can’t tell me that’s not going to get checked in by accident all the time. Especially in large projects with a large list of dependencies.

You shouldn’t need to run tidy, because the go tool should never implicitly add a dependency that my code doesn’t actually use. There should be an explicit command for that with a name that accurately describes the action taking place.

The problem is that “get” is not the right verb for “please add this dependency to my project even though I’m not actually referencing it in my code”. If we were building the go tool from scratch with module support, I don’t think “get” is the verb that would be chosen for that command. And since the verb already has baggage from earlier use, I don’t know why it should be used now. That seems like it counts against an already inaccurate verb.

And I don’t see why a new verb is a problem at all. Yes, there’s lots of documentation out there using go get … if we don’t muck with go get, that documentation doesn’t have to be incorrect. You could still go get a library, then reference it in your code, and when you build or test, the tool adds it to the go.mod, because you’re actually using the code. Basically the same workflow as it is now.

I agree that go getin GOPATH mode performs a grab-bag of tasks and I also agree that it is unfortunate that go get conflates downloading packages with installing binaries.

Considering that, why does the module-aware go get also gets the new responsibilities of adding, removing, upgrading, and downgrading of dependencies?

It seems counter-intuitive that a command that already does too much is getting new responsibilities and side effects (such as updating go.mod).

@bcmills

[…] existing command lines don’t only use go get to install tools — they also use it to download package dependencies.

This is kind of a hobbyist workflow, I suppose, but most serious Go users are not using go get to get package dependencies; they are using vendoring, dep, or some other solution.

OTOH I suspect that many or most Go users make use of the “fetch/update a Go tool” functionality of go get.

Just a quick note - as far as I’ve seen, it’s always been clear that all of cmd/go would be adapted to work with modules. This is why something like go dep ensure would have never been a thing. To me, go mod add or go mod update fall in the same category.

I agree that the semantics of go get and go install may get a bit confusing during this gopath-to-modules transition period. Still, I don’t think it’s confusing enough to warrant stuffing all of these operations as sub-sub-commands under go mod. That’s not a great long-term solution either, in my opinion.

Those of you who followed this thread might be interested in #30515, to add a “global install” command of some sort.

Here are three quick current examples of instructions using go get -u:

go-redis (https://github.com/go-redis/redis)

Install:

go get -u github.com/go-redis/redis

grpc (https://grpc.io/docs/quickstart/go.html)

Use the following command to install gRPC.

$ go get -u google.golang.org/grpc

aws-sdk (https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/setting-up.html)

Install the AWS SDK for Go To install the SDK and its dependencies, run the following Go command.

go get -u github.com/aws/aws-sdk-go/…

Or another simple data point is that 3 of the first 5 packages listed under “Popular Packages” on https://godoc.org included go get -u instructions based on quick peek. That might not be representative, though.

Also, those might not be perfect examples (for example, aws-sdk also includes commands, I think), or maybe turns out to be uncommon after a more detailed search (given what I did was just very brief).

In any event, it still might be smart to change the current approach for what go get means, or to add go add or go update. Just wanted to provide some quick examples.

(minor edit, including added simple data point from “Popular Packages”)

it seems even weirder for go get -u not to upgrade dependencies, or for go get -u to affect the module where go get would not

I feel like this is a disconnect between people who use a single gopath for everything (i.e. google) and people who exclusively use vendored dependencies.

Every serious public project for a long time has used vendored dependencies. go get -u would never update any of these dependencies, because they all use vendoring.

Also, apparently trying to install a tool outside a module doesn’t even work:

/tmp
$ set -gx GO111MODULE on # fish shell syntax
/tmp
$ go get golang.org/x/tools/cmd/goimports
go: cannot find main module; see 'go help modules'

I have never in 6 years of writing Go, run go test example.com/foo/bar. If I want to twiddle with example.com/foo/bar before I use it, I go get it, switch to that directory, and run local commands there, because it’s so much easier (like being able to look at files etc).

Literally the only time I specify an absolute package path is using go get. Every other time, I cd to the directory and use relative package names… because it just saves so much typing if I have to run more than one command.

In fact, this is one thing I worry about when changing go get. it was super useful for… getting code. I would propose that we leave go get as it is for expressly that purpose - getting Go code onto your machine so you can look at and fiddle with it as needed (as well as a super easy way to install binaries).

Here’s my workflow:

1. google for that package name that I can’t quite remember
2. go to the github page, check out the readme, maybe peek at the code to make sure it looks sane
3. click on the godoc.org badge to get a better view of the API
4. run go get on my machine to get the code locally.
5. open that (now local) repo in my editor to look at the code / run tests etc.
6. go back to my project, add the import, add a little code so goimports won’t take away the import 😉
7. rebuild my project.

Right now there’s usually a 6.5 where I run dep ensure, but I’m happy to have go build do that for me.

The thing about “having modules built into the go tools” is that I expect that only packages the code actually depends on will be added to the go.mod. Adding something my code doesn’t explicitly use should only happen through an explicit command that I run that says “yes, I know I’m not actually using this, but add it to my go.mod anyway”.

Regarding possibly changing the behavior of go get or introducing new commands, one consideration raised is the degree to which some or most historical install instructions can continue to work (as discussed above for this issue, and as you can see that is also a very important consideration for #24250 to make sure go get some/command always works in a module-world, although the best full solution for #24250 might need to wait until modules are on by default – see for example https://github.com/golang/go/issues/24250#issuecomment-377553022 and https://github.com/golang/go/issues/24250#issuecomment-411860657 comments by Russ, though of course further discussion about that aspect probably deserves to live in #24250).

Another consideration for the “should go get update go.mod?” issue here that I wanted to briefly highlight is what was termed the build “isolation rule”, which was described in this blog post back in Feb:

https://research.swtch.com/vgo-cmd

If you are thinking about how to best modify the go command behavior (e.g., as discussed above by altering what go get does, or perhaps introduce a new go add and go update, or ___), it is worth reading that blog post. (It is not too long, and it is the first 2/3 or so that is most relevant to the discussion here).

Some of the key snippets (though still worth reading the post itself):

the ‘isolation rule’:

The result of a build command should depend only on the source files that are its logical inputs, never on hidden state left behind by previous build commands.)

That is, what a command does in isolation—on a clean system loaded with only the relevant input source files—is what it should do all the time, no matter what else has happened on the system recently.

and:

Of course, all the [new] vgo get variants record the effect of their additions and upgrades in the go.mod file. In a sense, we’ve made these commands follow the isolation rule by introducing go.mod as an explicit, visible input replaces a previously implicit, hidden input: the state of the entire GOPATH.

and:

every command should have only one meaning, no matter what other commands have preceded it.

and:

Plain [pre-module] go get, without -u, violates the command isolation rule and must be fixed. [The pre-module go get has three meanings]:

• If GOPATH is empty, go get rsc.io/quote downloads and builds the latest version of rsc.io/quote and its dependencies (for example, rsc.io/sampler).
• If there is already a rsc.io/quote in GOPATH, from a go get last year, then the new go get builds the old version.
• If rsc.io/sampler is already in GOPATH but rsc.io/quote is not, then go get downloads the latest rsc.io/quote and builds it against the old copy of rsc.io/sampler.

Overall, [pre-module] go get depends on the state of GOPATH, which breaks the command isolation rule. We need to fix that.

There are different behaviors that could be chosen while still satisfying the build isolation rule when it comes to go get (up to even “eliminate go get entirely and instead introduce new commands X and Y” could satisfy the build isolation rule), but at least wanted to highlight that blog post for people as a design consideration given that post describes some of the philosophy behind having go build by default download code as needed and some of the corresponding current changes in behavior for go get in a modules world.

A note on local binaries, https://github.com/twitchtv/retool does exactly what you want, @myitcv … the only drawback being that you need to run them by running retool do <binary>. So like retool do dep ensure

To be fair, that question was answered in 2012 😃

A testament to Go 1 compatibility! 🙂

I feel like this is a disconnect between people who use a single gopath for everything (i.e. google) and people who exclusively use vendored dependencies.

Most Google-internal users pretty much aren’t using GOPATH (or the go tool) at all. This isn’t a Google-vs.-outside issue so much as a hobbyist-vs.-enterprise one.

Every serious public project for a long time has used vendored dependencies. go get -u would never update any of these dependencies, because they all use vendoring.

Yes, we clearly need to work on the migration path for vendoring. That’s one of my top priorities for 1.12.

go get foo/bar/baz explicitly tells the go tool “ignore the directory that I’m in. Instead, find this package, make sure it and its dependencies are downloaded locally, build it, and install it.”

But go get -u foo/bar/baz upgrades arbitrarily many dependencies: it may affect the current directory, depending on what else is involved.

As weird as it is for go get of a tool to update the current module but install the tool globally, it seems even weirder for go get -u not to upgrade dependencies depending on the mode, or for go get -u to affect the module where go get would not.

I agree that the semantics of go get and go install may get a bit confusing during this gopath-to-modules transition period. Still, I don’t think it’s confusing enough to warrant stuffing all of these operations as sub-sub-commands under go mod. That’s not a great long-term solution either, in my opinion.

Ok, but that still leaves us without a convenient equivalent to “ignore my current directory, pull down this repository, compile it, and then install it for global use”, which is a semantic that go get is very commonly used for today. The go mod version has no single command equivalent to this use case, the only options are using go get followed by go mod tidy (which can have ancillary side effects, since tidy doesn’t respect build tags), or changing directories (also not currently possible, since lack of a go.mod fail is a failure, but even if that’s fixed, that’s 3 commands, cd -> go get -> cd back), or by manually doing a git clone followed by go install. None of those are very clean.

Perhaps a solution would be to add a flag to go install (-u appears to be unused) to tell it to fetch and update from remote before compiling and installing (and that flag would require that a package be named and it have a resolvable import path). That would bake the usage into another command with better-aligned semantics, would retain close to the same functional flow as go get does, and wouldn’t change the default behavior of go install either (unless that ends up being the desired pathway).

one more thing I just encountered that is related.

running go build foo/bar/baz in a module directory … adds foo/bar/baz to the go.mod. That seems really weird. Now, I’ll grant you, go build foo/bar/baz in a module context… sorta doesn’t make any sense… in which case maybe we just error out? I don’t really think there’s any action that makes sense in a module context where foo/bar/baz is not part of the current module.

I don’t know what your thoughts on this are, and whether it’s worth creating a separate issue for the “local” install target/run solution?

Yes, please file a separate issue.

(Let’s try to stick to one decision per issue so we don’t lose track when issues are closed. As I see it, the decision for this issue is “should go get update go.mod?”)

As weird as it is for go get of a tool to update the current module but install the tool globally,

I actually think the absence of a “default module-local install location” falls into the same category of “missing from Go 1.11” as https://github.com/golang/go/issues/24250. Let me try to explain by covering my “workflow”.

The tools I use on a day-to-day basis fall into two categories:

1. Tools I want to be controlled by a project’s go.mod
2. Tools that I need globally

Dealing with these in reverse order.

Category 2: I think it’s clear that with Go 1.11, there is a “gap” here and that this is covered by https://github.com/golang/go/issues/24250. Per the detail in that discussion, there are open questions on how to handle multiple versions, where the installed binaries should be put etc, but it all falls under that issue.

Category 1: by far the largest category of tools for me, made up largely of code generators that I use with go generate and the like. I absolutely want these to be version controlled. And I don’t want to be using (via my PATH) a “global” install of such a tool, even if the version just happens to match at that point in time. But both go get and go install currently (i.e. Go 1.11) have a target of $GOPATH/bin (ignoring multi-element `GOPATH values for now).

Hence the workflow I have effectively adopted is to create a module-local install target:

# create a new module
cd $(mktemp -d)
mkdir hello
cd hello
go mod init example.com/hello

# set GOBIN as a module-local install target
export GOBIN=$PWD/.bin

# update my PATH accordingly (I actually use https://github.com/cxreg/smartcd for this)
export PATH=$GOBIN:$PATH

# add a tool dependency (by definition, category 1 tool) following "best practice" laid out in 
# https://github.com/golang/go/issues/25922#issuecomment-412992431
cat <<EOD > tools.go
// +build tools

package tools

import (
        _ "golang.org/x/tools/cmd/stringer"
)
EOD

# install the tool
go install golang.org/x/tools/cmd/stringer

# verify we are using the module-local binary
which stringer

# which gives something like:
# /tmp/tmp.Hh0BNOF6k2/hello/.bin/stringer

As mentioned, one of the disconnects (in this thread) is that a go get or go install in a module context modifies the “local” go.mod but installs “globally”. This is, as @bcmills put it, “weird”. But is to my mind a gap in Go 1.11, just as there not being a “global” tool install is a gap (i.e. https://github.com/golang/go/issues/24250). Remember, module support is experimental and almost by definition incomplete.

go run is a potential alternative to the “local” install here (and a very attractive one to my mind), but it is a non-starter unless we can find a way to address https://github.com/golang/go/issues/25416.

There could be some convention that a .bin/ directory, alongside a go.mod, is the target for go get and go install (of main packages) for “local” installs? But this wouldn’t obviate the need for everyone to update their PATH and indeed .gitignore the .bin directory for every module they work on.

And I’m sure there are many other potential solutions.

But I thought it worth pointing out that I see this as a gap, as opposed to a final decision on workflow.

@bcmills I don’t know what your thoughts on this are, and whether it’s worth creating a separate issue for the “local” install target/run solution?

To be fair, that question was answered in 2012 😃

And go get -u is fine for binaries, just not really useful for libraries. but separating out which readmes are for binaries vs. libraries is pretty hard, I’d think.

This isn’t a Google-vs.-outside issue so much as a hobbyist-vs.-enterprise one.

Sorry! I almost deleted the google bit there. definitely was not trying to make this a google vs. the outside world.

@natefinch

We have a command for dealing with modules, and it’s go mod, that’s the command that should be used to add dependencies.

You are mistaken. The go mod command is for module-specific operations, not all operations “dealing with modules”. As its documentation says, “support for modules is built into all the go commands, not just go mod. For example, day-to-day adding, removing, upgrading, and downgrading of dependencies should be done using go get.”

go get in GOPATH mode already performs a grab-bag of tasks: it “downloads the packages named by the import paths, along with their dependencies. It then installs the named packages, like go install.” go get has the same behavior in module mode, except that it downloads entire modules and it does not yet work outside of a module (that’s #24250).

In my opinion, it is unfortunate that go get conflates downloading packages with installing binaries, but we’re essentially stuck with it: we need to maintain compatibility with existing, documented command lines, and existing command lines don’t only use go get to install tools — they also use it to download package dependencies.

That issue clarifies this one even more. Prior to Go 1.11, the semantics for go get were identical to go install, except that it pulled the code from a remote repository before running go install on them. The module functionality in Go 1.11 has completely abandoned the go install piece (actually, it’s still there, but it’s mostly an anachronism at this point) because go get has been hijacked for adding a dependency to a module.

And that’s a bit of a problem, since go get was validly used just as commonly for the go install functionality as it was for the abstraction of git clone. Abandoning the install functionality is extremely jarring and leads to some very unexpected side-effects, like the necessity of running go mod tidy after installing a tool that’s not also a dependency, and as @vdemario just pointed out, the complete inability to install tools unless you’re currently in a module (which you then need to tidy).

@vdemario please see, as linked above, https://github.com/golang/go/issues/24250