go: cmd/go: go install cmd@version errors out when module with main package has replace directive
What version of Go are you using (go version
)?
go version go version go1.16 linux/amd64
Does this issue reproduce with the latest release?
yes
What did you do?
cd /tmp; go install bitbucket.org/jatone/genieql@latest
What did you expect to see?
go install finishing successfully
What did you see instead?
go install bitbucket.org/jatone/genieql@latest: bitbucket.org/jatone/genieql@v0.0.0-20210306180040-be9c7b630c18
The go.mod file for the module providing named packages contains one or
more replace directives. It must not contain directives that would cause
it to be interpreted differently than if it were the main module.
given that genieql is the main module this error appears to be erroneous.
other attempts I made all of which i would have expected to work properly (and use to):
cd /tmp; go install bitbucket.org/jatone/genieql/...@latest
go install bitbucket.org/jatone/genieql/...@latest: bitbucket.org/jatone/genieql@v0.0.0-20210306180040-be9c7b630c18
The go.mod file for the module providing named packages contains one or
more replace directives. It must not contain directives that would cause
it to be interpreted differently than if it were the main module.
cd /tmp; go install bitbucket.org/jatone/genieql/cmd/...@latest
go install bitbucket.org/jatone/genieql/cmd/...@latest: bitbucket.org/jatone/genieql@v0.0.0-20210306180040-be9c7b630c18
The go.mod file for the module providing named packages contains one or
more replace directives. It must not contain directives that would cause
it to be interpreted differently than if it were the main module.
cd /tmp; go install bitbucket.org/jatone/genieql/cmd/genieql@latest
go install bitbucket.org/jatone/genieql/cmd/genieql@latest: bitbucket.org/jatone/genieql@v0.0.0-20210306180040-be9c7b630c18
The go.mod file for the module providing named packages contains one or
more replace directives. It must not contain directives that would cause
it to be interpreted differently than if it were the main module.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 27
- Comments: 45 (20 by maintainers)
Links to this issue
Commits related to this issue
- Fix Go-based subctl install docs Because we have `replace` statemetns in `submariner-operator/go.mod` to pin the K8s version, when run from outside of the submariner-operator repo `go install` declin... — committed to dfarrell07/submariner-website by dfarrell07 3 years ago
- Fix Go-based subctl install docs Because we have `replace` statements in `submariner-operator/go.mod` to pin the K8s version, when run from outside of the submariner-operator repo `go install` declin... — committed to dfarrell07/submariner-website by dfarrell07 3 years ago
- Merge the api module back into the main module Go dependencies are handled by package, not by module, and per-package splits are handled correctly. Having a separate package makes it easier to track ... — committed to skitt/submariner-operator by skitt 3 years ago
- Merge the api module back into the main module Go dependencies are handled by package, not by module, and per-package splits are handled correctly. Having a separate package makes it easier to track ... — committed to skitt/submariner by skitt 3 years ago
- Merge the api module back into the main module Go dependencies are handled by package, not by module, and per-package splits are handled correctly. Having a separate package makes it easier to track ... — committed to skitt/submariner-operator by skitt 3 years ago
- Drop obsolete security replaces Our dependency tree pulls in the same or newer versions of /x/crypto and /x/text, remove the replace directives. This helps with go install (see <https://github.com/g... — committed to skitt/submariner by skitt 3 years ago
- Unpin Kubernetes dependencies This doesn't seem to break anything, and helps with go install (see <https://github.com/golang/go/issues/44840>). Signed-off-by: Stephen Kitt <skitt@redhat.com> — committed to skitt/submariner by skitt 3 years ago
- Drop obsolete security replaces Our dependency tree pulls in the same or newer versions of /x/crypto and /x/text, remove the replace directives. This helps with go install (see <https://github.com/g... — committed to skitt/submariner by skitt 3 years ago
- Fix Go-based subctl install docs Because we have `replace` statements in `submariner-operator/go.mod` to pin the K8s version, when run from outside of the submariner-operator repo `go install` declin... — committed to submariner-io/submariner-website by dfarrell07 3 years ago
- Merge the api module back into the main module Go dependencies are handled by package, not by module, and per-package splits are handled correctly. Having a separate package makes it easier to track ... — committed to skitt/submariner by skitt 3 years ago
- Merge the api module back into the main module Go dependencies are handled by package, not by module, and per-package splits are handled correctly. Having a separate package makes it easier to track ... — committed to skitt/submariner by skitt 3 years ago
- Merge the api module back into the main module Go dependencies are handled by package, not by module, and per-package splits are handled correctly. Having a separate package makes it easier to track ... — committed to skitt/submariner by skitt 3 years ago
- Merge the api module back into the main module Go dependencies are handled by package, not by module, and per-package splits are handled correctly. Having a separate package makes it easier to track ... — committed to skitt/submariner by skitt 3 years ago
- Unpin Kubernetes dependencies This doesn't seem to break anything, and helps with go install (see <https://github.com/golang/go/issues/44840>). Signed-off-by: Stephen Kitt <skitt@redhat.com> — committed to skitt/submariner by skitt 3 years ago
- Unpin Kubernetes dependencies This doesn't seem to break anything, and helps with go install (see <https://github.com/golang/go/issues/44840>). Signed-off-by: Stephen Kitt <skitt@redhat.com> — committed to skitt/admiral by skitt 3 years ago
- Unpin Kubernetes dependencies This doesn't seem to break anything, and helps with go install (see <https://github.com/golang/go/issues/44840>). Signed-off-by: Stephen Kitt <skitt@redhat.com> — committed to skitt/admiral by skitt 3 years ago
- Unpin Kubernetes dependencies This doesn't seem to break anything, and helps with go install (see <https://github.com/golang/go/issues/44840>). Signed-off-by: Stephen Kitt <skitt@redhat.com> — committed to submariner-io/admiral by skitt 3 years ago
- Merge the api module back into the main module Go dependencies are handled by package, not by module, and per-package splits are handled correctly. Having a separate package makes it easier to track ... — committed to submariner-io/submariner-operator by skitt 3 years ago
- Merge the api module back into the main module Go dependencies are handled by package, not by module, and per-package splits are handled correctly. Having a separate package makes it easier to track ... — committed to skitt/submariner-operator by skitt 3 years ago
- Unpin Kubernetes dependencies This doesn't seem to break anything, and helps with go install (see <https://github.com/golang/go/issues/44840>). Signed-off-by: Stephen Kitt <skitt@redhat.com> — committed to skitt/shipyard by skitt 3 years ago
@jayconrod wrote:
As a go user, I reasonably expect
go install cmd@version
to just work and install the binary cmd using thego.mod
the authors of that package spent time getting exactly right.Instead I get an extremely cryptic message like this
As a user of said
cmd
, I have no idea how to fix that problem.From the instructions for
go install cmd@version
: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies need amendingWhich says the same thing as the error message, more or less.
So it is consistent, but not helpful in my opinion!
It is very hard having a large open source project without a few soft forks while you are waiting for upstreams to take your patches.
I first noticed this problem with github’s
gh
tool, and I think you’ll find that there are lot more high profile Go binaries out there with this problem.L28 contains a replace directive
Which is not valid for
go install
Closing as working as intended
I read the proposal in #40276 and it concludes with:
I am sad to learn that 4% is small for some, but it is still a big one when you are one of those inside 4%.
I think those soft forks are really important use case to support. We have this situation just now, we have a tool which depends on 3rd party library. That library have a data-loss bug users of our tool might encounter so we made a PR upstream with a fix and use
replace
to fix our tool. But now we cannot really release our tool but have to wait for upstream to merge it. In meantime, all our users might be loosing data. So now we have to wait for upstream to merge it.You can imagine similar scenario for a security issue, too. So sometimes a soft fork is useful to get an important fix out soon. We can build a binary and release it so I am not sure why users cannot do
go install ... @version
to get exactly the same binary, just compiled for them.Hi @rsc,
Thankyou for the above, but it seems to avoid addressing the key use case I and others in this issue seem to want:
The particular problem is with the statement
which ignores a somewhat common practice:
go install foo/bar/baz@version
It seems like you would class this as a misuse of replace directives based on what you said above?
The problem with that, is that the Go project has given developers an almost perfect tool - replace directives - for dealing with this, so it is unrealistic to think people would not use replace directives in this scenario. AFAIK there are no other options that don’t involve needing to change import paths of the forked software in every source file that consumes it, which is inelegant in the extreme!
It’s not like the users trying to use
go install
can go to the developers using the replace directives and ask them not to use them - they’ll rightly point out that extreme inelegancy in forking without using replace directives, and decline to do that.So you end up with unhappy users who have had the lovely convenience of
go install
dangled in front of them, and then snatched away.@jayconrod or other maintainers: given that this issue has a lot of traffic & there will be a lot of projects with replace directives:
Clearly people would like to be able to “go install” things with replace directives.
Ideally it would just respect the directives of the module containing the desired build target.
Given that both this issue & the ones linked to in the closing message are all closed, there’s no path forward to this right now (or is there?) - could we re-open this at least?
Regarding replace directives being “only for local development” reading back to the original vgo and modules proposals I find this about replace.
From vgo tour
From Minimal Version Selection
(emphasis mine)
also from Minimal Version Selection, section Who controls your build?
From the modules proposal:
(emphasis mine)
From the Q&A section of the modules proposal:
PS. I couldn’t find any mention in the proposal or in the vgo posts about replace being only appropriate for local builds but AFAICT the proposal did not introduce the distinction between local vs non-local builds, AFAICT that was initially introduced as a consequence of implementation details of
go get
.@rsc: My understanding of this issue is that we care about the use case where you use
go install
outside of a go module. I think what we really are asking for is thatgo install
should just be the same as doing git clone + go install inside the cloned repository. I think this is a well defined behavior. No surprises there. It would make the use case where you fix the program in your fork and wait for upstream to merge it possible.Installing inside a go module is tricky as you describe. And there is no need to support it there because inside a go module you can just modify go.mod yourself and add the replace directive there.
So, could we get
go install
to work outside of go modules with repositories which have replace directives?Unfortunately, I don’t believe that there is.
go install
just simply doesn’t work for some projects anymore. It’s incredibly frustrating.For anyone here from a Google search for this error message, try running this command to install a package from a repo (“like
npm install -g
”):(replace
github.com/apenwarr/git-subtac
with your desired package; that’s just an example of the one that led me here today)It worked as desired:
Note my
go version
isgo1.18.3 darwin/amd64
which is fairly outdated (latest is1.19.2
), so YMMV.thanks to Alec over at StackOverflow. more explanation here: https://stackoverflow.com/a/52764550/3793499
For the Go developers, I thought I’d document how I found this error message:
I don’t write Go, but I do occasionally run programs in it, and I have a reasonably up-to-date version on my system (
go version go1.18.3 darwin/amd64
). But the ecosystem is mostly unknown to me. So every time I need to install a package it’s a bit of a chore to remember the right$GOPATH
settings, check it’s actually installed, remember it’sgo version
notgo --version
, etc. And when something else goes wrong, I have to second guess all that configuration before I know if the error is my fault or not.Today I found this thread because I wanted to install a global binary (https://github.com/apenwarr/git-subtrac) that I could use on my system. That readme quite reasonably suggests:
in reality, this is what my session looked like (starting from
go version
, which took a few attempts to get right 😄):ok, seems a bit old but not too bad, and this repo is 2 years old, so let’s try the instructions from the readme:
add
@latest
? ok, I can try that… and I do like when an error message gives a suggestion 😃oh no 😦 I guess that’s why it was only a suggestion
*various
go help
mashing trying to find the answer to how to install a global binary into$GOPATH/bin
…*it’s not very helpful 😦
*google some words*
Finally find https://stackoverflow.com/a/52764550/3793499 with the answer
Why is this issue closed if there is no good solution?
In case there is a good solution, let me know. (I need it.)
Some more anecdotal evidence as to why this is an issue:
We are using the k8s client libs, but recently this change was introduced in the openAPI Spec models: https://github.com/kubernetes/kube-openapi/pull/402 https://github.com/kubernetes/kube-openapi/commit/38767cdfbb68f871eb458f7cde26177cb65c91e9
Now we can’t upgrade
k8s.io/api
ork8s.io/client-go
because of the model mismatches:So we are left with: 1.) Revert (and wait for transient patches) - https://github.com/kubernetes/kubernetes/issues/118340 - https://github.com/kubernetes-sigs/cli-utils/pull/625 2.) Run a fork (and make the patch ourselves) 😱 3.) Use a replace directive
Being a Lazy (Pragmatic…¯_(ツ)_/¯) Programmer, I chose option 3, as I think most Go Devs would. Now I can upgrade the module I want, without any of the side effects. It just works.
…But now we can’t
go install
the code because of the replace directive. Which is kind of annoying. I would expect go install to honor the replace directive, no different than if I clone the entire repo down and run go install from there.I would also argue that although the replace directive may have been intended for local development purposes, it is widely used across go projects to circumvent issues exactly like the one I posted here. Dependencies can get messy, and being able to “strategically” use a replace directive when necessary is a nice way to work around (hack) those issues.
Someone flagged recent discussion on this closed issue to me privately. As a general note, we don’t watch closed issues very carefully - they’re closed.
Replace directives remain for local development. If you want to post software that others can go install, it needs to build without the replace directives. If we respected the replace directives during go install foo@latest, then that would create commands that can be installed that way but not using ‘go get foo; go install foo’ in a module where you are trying to track the version of foo you are using. We are also discussing integrating tools more explicitly with a module in #48429, and again there it would be a serious problem if tools existed that worked with ‘go install foo@latest’ but not with a regular build graph from another module. (And it is a non-starter for one tool in your module to force the use of its own replace directives. What if you don’t want those replace directives? Or what if different tools have conflicting replace directives?)
If you are using dependencies that don’t function without replace directives, that may be a sign to rethink the use of those dependencies, or to upstream fixes that make the replace directives unnecessary.
For the specific case of go4.org/unsafe/assume-no-moving-gc, there are two better answers than replace:
These work within the ecosystem and keep it healthy. Replace directives fragment the ecosystem and make it brittle. My post The Principles of Versioning in Go has a worked example. It looks at forced package downgrades, but replaces are analogous andhave all the same problems.
We are aware of the problem of go4.org/unsafe/assume-no-moving-gc specifically. In May we worked with the upstream author on a change that makes it no longer break at each new Go release. So if you run ‘go get go4.org/unsafe/assume-no-moving-gc@latest’ in your own module one last time, you should stop seeing breakages from that package, and you can remove the replace.
@jayconrod this seems to cause some significant inconveniences and if I understand the following passage from #40227 correctly, it should be possible to ease the restrictions and restore a smoother user experience:
This whole thing is especially tricky since everything needs to be fixed at the root of the issue and even then it’s problematic.
If we look at https://github.com/openshift/hypershift/issues/803 for example and its replace directives: https://github.com/openshift/hypershift/blob/b5e8c84eefb207aaff9be8b9d4f2c606d9019f80/go.mod#L129
it’s pointing to https://github.com/kubevirt/containerized-data-importer-api/blob/main/go.mod which has its module name different from its repository. However, a contributor cannot easily fix this because, the change with a following
go mod tidy
results in something like:where
kubevirt.io/containerized-data-importer-api v1.47.0
would either need to be changed by hand which might fail in CI or a follow up PR that means extra effort.tl;dr: if I submitted a proposal to relax the behavior would that be something the team would generally consider?
The same thing. We can’t install 3rd party tools with go 18 any more.
This would be a lot less of a problem if the
go
commandIt would still be a problem, but it would be significantly less of one.
It seems it was closed in March with the idea to gather more information. Considering the amount of references to this bug in the mean time, it might be a good idea to reopen the discussion.
I wish I could install it too.
I would like to be able to install it with
go install
as I would like many people to test it before I soft fork it and submit a pull request to the developer.@mitar Similarly I have a lot of soft-forks which I need replace directives for. The users can’t install with “go install” because of the limitation on replace directives, but it would make way more sense if Go install just behaved as if I had run “go install” with cwd set to the root of the module containing the referenced package. Issue:
go install -v github.com/aperturerobotics/bifrost/cmd/bifrost@master
- fails w/ this error.@seankhliao I’m just not sure if it is a bug or a proposal, the discussion around go installs (new) behavior was long, varied, and confused. =) this particular problem was pointed out during that discussion and seemingly ignored/disregarded/left open to be revisited. so here we are.
Is there any proper workaround around this? Or now the installation of projects with a replace inside mod file is not feasible at all?
A more helpful error message I think:
I think this message is better because judging the comments in this thread:
go build
is usually the next step when one sees this error.FWIW I’m pro
go install@version
respecting in-repo replace directives whether its by just making that change directly or part of some rethink of supporting soft forks.We have soft forks of tools due to various reasons and now we are blocked from upgrading from go 1.16 b/c of this issue. Given that Go 1.16 isn’t being supported forever we’ll soon be stuck with outdated software or rewrite a lot of module imports to make hard forks…
The only work around I have figured out is to not use
go install git...
and manually git clone followed by manual go build/run/install. You could even write a replacement command that would work withgo run
, setup git clone in a temporary directory, run go build/install, remove the code. Basically hiding this problem from your users. It is lame, but should work and be close to what go actually does.Seems nobody has pointed this before, but if you follow the multi module repository documentation: https://github.com/golang/go/wiki/Modules#faqs--multi-module-repositories , you won’t be able to install your application with go install. It would be good to either document this in the faqs or relax the rules for this case to be handle as described in the faqs.
I hit this too.
Do I misinterpret that there’s just no recourse here?
@seanbethard you must git clone that project and run “go install .” within the directory.
As someone who is using a lot of replace directives, this is a quite annoying limitation for go install.
Example: https://github.com/aperturerobotics/bifrost#examples
Here I have to say - well, you could use go install, but we have replace directives, so instead clone it and build. It defeats the purpose of go install.
So what do you do if you encounter a replace? lol
I just installed go for the first time to use godo.
go get github.com/digitalocean/godo@v1.106.0
go install github.com/digitalocean/godo@v1.106.0
I don’t get it. (get it?) What’s
go.mod
’s deal anyway?@jayconrod
Here’s a case study: I have a project that relies on some Gtk4 bindings. Those bindings in turn rely on go4.org/unsafe/assume-no-moving-gc which uses build constraints that are manually added as each new Go version is released as a form of an assertion that Go never adds a moving GC. This causes problems, as it means that my app will not build with new versions of Go unless I wait for that module to be updated and then I also update my dependencies to use that new version. It also means that old versions of the app will never build with newer versions of Go.
I got tired of this and added a
replace go4.org/unsafe/assume-no-moving-gc => ./internal/empty
whereinternal/empty
was a module with nothing in it as a way of getting rid of the assertion. This worked great, until someone tried to install my app usinggo install
at which point it broke due to this issue. Due to the utter lack of viable workarounds, I was forced to remove thereplace
directive and go back to the manual update process every six months or so.I would like to add another use case for why applying
replace
is required:Summary:
It is required to be able to use BuildInfo (
debug.ReadBuildInfo()
) as version embedding mechanism which is imo the go1.18 way to go (and will be even cleaner whengo build package@version -o somePath
will work #52898)Long version:
I’ve recently switch version embedding from the previous pattern of
ldflags -X foo.version=1.2.3
to using the wonderful new 1.18 BuildInfo and usego install ...@tag
to create even the binary distributions of my projectsSo for my projects
go install project@version
is now the only way to create the right binariesI found a bug in golang.org/x/net that is fixed in a cl but not yet available so I patched it into a fork
https://github.com/fortio/proxy/blob/main/go.mod
but I can’t install it! so I can’t have it show the right version
is there at least a --please-do-it-anyway flag or way to build the binary with the BuildInfo and with a go.mod replacement that I am missing?
Ah I believe I have a good explanation about why this should be supported.
the replace directive is fundamentally about compilation, its meant to inform the build system how to patch dependencies. this is true both during development where you may need to patch them while debugging a problem which is when you compile your program. and during installation of a main module (previously the main package) onto a system.
so during compilation there is no distinction between the main package and the main module, however, during dependency management replace directives have always been ignored and any software that was relying on a dependency where there was a replace directive was broken (conceptually if not in practice). and this is where we should be erroring out (i.e. in go get in this new world) when we see a replace directive. not during compilation (go install).