go: cmd/go: get fails on gitlab subgroups due to go-import meta tags referring to nonexistent repos

@xavivars, this has already been fixed at head and backported to the 1.13 branch. It will be included in Go 1.13.3.

Just tested the latest build on release-branch.go1.13 and can confirm it works with a private GitLab subgroup.

@umputun, note that if the module is already in the build list, we don’t bother searching all possible paths for it. So one workaround, if you already know the module or repo path, is to simply run something like

go mod edit -require gitlab.com/umputuntests/sub/example@master

in order to prime the appropriate paths before running any remaining go get subcommands.

foo.example.com$ go1.13 mod init foo.example.com
go: creating new go.mod: module foo.example.com

foo.example.com$ go1.13 mod edit -require gitlab.com/umputuntests/sub/example@master

foo.example.com$ go1.13 list -m all
go: finding gitlab.com/umputuntests/sub/example master
go: finding github.com/davecgh/go-spew v1.1.0
go: finding github.com/pmezard/go-difflib v1.0.0
go: finding github.com/stretchr/objx v0.1.0
go: finding github.com/stretchr/testify v1.3.0
foo.example.com
github.com/davecgh/go-spew v1.1.0
github.com/pmezard/go-difflib v1.0.0
github.com/stretchr/objx v0.1.0
github.com/stretchr/testify v1.3.0
gitlab.com/umputuntests/sub/example v0.0.2

I’m still facing this issue with go version go1.13.3 darwin/amd64

@umputun and others: this should be fixed at head and on release-branch.go1.13. Could someone with access to a private GitLab subgroup please try a build from release-branch.go1.13 and confirm?

(The instructions are here, but instead of git checkout master you’ll want to run git checkout release-branch.go1.13.)

Is there a reason we can’t do the module search in parallel but still walk the results in reverse order of specificity, thus ignoring errors unless every path had an error?

Yeah, given the timeframe for fixes to the serving paths we should probably just do that.

(I’m not particularly happy about burying errors in general, but it seems like the least-unpleasant solution here.)

Is there a reason we can’t do the module search in parallel but still walk the results in reverse order of specificity, thus ignoring errors unless every path had an error?

The permissions argument seems like a pretty distant edge case. In general, Go assumes one module path means one source code repository. If you were to return different results based on user identity, you could do that at the git layer in the same way you might at the ?go-get= resolution layer. Either way, you’re seriously breaking the Go view of the world, and you’ll probably know you are, but have deliberately decided to take on any pain associated with differing code results.

The way things are now implemented, that potential for pain for offenders now becomes real pain for anybody who relied on the more obvious behavior before, including indirectly, as in the case of those simply using these version control systems.

@TheDiveO We had a similar problem, we solved it by using the replace directive. For example:

require (
	private.gitlab.instance/project/team/somename v1.0.0
)

replace (
	private.gitlab.instance/project/team/somename => private.gitlab.instance/project/team/somename.git v1.0.0
)

However please be aware that this solution does not work cleanly when trying to upgrade that said package to non-tagged versions, for example you would need to do something like:

go get private.gitlab.instance/project/team/somename.git@somecommithash 

Which then will fail with a message similar to:

go: finding private.gitlab.instance/project/team somecommithash
go: finding private.gitlab.instance/project/team/somename.git somecommithash
go: downloading private.gitlab.instance/project/team/somename.git vXYZ-somecommithash
go: extracting private.gitlab.instance/project/team/somename.git vXYZ-somecommithash
go get: private.gitlab.instance/project/team/somename.git@vXYZ-somecommithash: parsing go.mod:
	module declares its path as: private.gitlab.instance/project/team/somename
	        but was required as: private.gitlab.instance/project/team/somename.git

We literally have to copy vXYZ-somecommithash, manually edit go.mod and then overwrite that value, changing it to something like

require (
	private.gitlab.instance/project/team/somename v1.0.0
)

replace (
	private.gitlab.instance/project/team/somename => private.gitlab.instance/project/team/somename.git vXYZ-somecommithash
)

doesn’t work with 1.12 either

go version go1.12.12 linux/amd64, from golang:1.12 container:

go get -v git.internal.com/commons/pkg/syncs

Fetching https://git.internal.com/commons/pkg/syncs?go-get=1
Parsing meta tags from https://git.internal.com/commons/pkg/syncs?go-get=1 (status code 200)
get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at https://git.internal.com/commons/pkg/syncs?go-get=1
get "git.internal.com/commons/pkg/syncs": verifying non-authoritative meta tag
Fetching https://git.internal.com/commons/pkg?go-get=1
Parsing meta tags from https://git.internal.com/commons/pkg?go-get=1 (status code 200)
Fetching https://git.internal.com/commons/pkg?go-get=1
Parsing meta tags from https://git.internal.com/commons/pkg?go-get=1 (status code 200)
get "git.internal.com/commons/pkg": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at https://git.internal.com/commons/pkg?go-get=1
Fetching https://git.internal.com/commons?go-get=1
Parsing meta tags from https://git.internal.com/commons?go-get=1 (status code 200)
Fetching https://git.internal.com?go-get=1
Parsing meta tags from https://git.internal.com?go-get=1 (status code 200)
go get git.internal.com/commons/pkg/syncs: git ls-remote -q origin in /go/pkg/mod/cache/vcs/82b580717104e8b7bf1de06ef2b675a3ca5bf7d044f776fac4629633677f22da: exit status 128:
	> GitLab: The project you were looking for could not be found.
	fatal: Could not read from remote repository.

	Please make sure you have the correct access rights
	and the repository exists.	

However with ~/.netrc everyting seems to work (1.13.3):

go get -v git.internal.com/commons/pkg/syncs
get "git.internal.com/commons/pkg": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg?go-get=1
get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg/syncs", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg/syncs.git"} at //git.internal.com/commons/pkg/syncs?go-get=1
go: finding git.internal.com/commons/pkg/syncs v1.1.0
go: downloading git.internal.com/commons/pkg/syncs v1.1.0
go: extracting git.internal.com/commons/pkg/syncs v1.1.0
get "git.internal.com/commons/pkg/multierror": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg/multierror", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg/multierror.git"} at //git.internal.com/commons/pkg/multierror?go-get=1
git.internal.com/commons/pkg/multierror
git.internal.com/commons/pkg/syncs	

So, looks like .gitconfig insteadOf doesn’t play well with go mods, but .netrc is fine.

have you tested private group or subgroup on gitlab?

No, I didn’t but tried it now with similar setup - commons group public, pkg subgroup public and syncs repo private (internal in gitlab’s permissions). I see very similar results:

go version go1.13.3 darwin/amd64

go get -v git.internal.com/commons/pkg/syncs
get "git.internal.com/commons/pkg": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg?go-get=1
get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg/syncs?go-get=1
get "git.internal.com/commons/pkg/syncs": verifying non-authoritative meta tag
go get git.internal.com/commons/pkg/syncs: git ls-remote -q https://git.internal.com/commons/pkg.git in /Users/umputun/go-home/pkg/mod/cache/vcs/882b5ac6f69791979e3abd70af30e871e32fe91173f2e25079ef6a88464ec13e: exit status 128:
	> GitLab: The project you were looking for could not be found.
	fatal: Could not read from remote repository.

	Please make sure you have the correct access rights
	and the repository exists.

Not sure if important, but to access private repos from go I use insteadOf rewrite:

 [url "git@git.internal.com:"]
     insteadOf = https://git.internal.com/

I’ve just tried with Go 1.13.3 but I still failed in go get my subgroup repository.

$ go version
go version go1.13.3 linux/amd64

$ go get -v gitlab.com/foo/bar/example
get "gitlab.com/foo/bar": found meta tag get.metaImport{Prefix:"gitlab.com/foo/bar", VCS:"git", RepoRoot:"https://gitlab.com/foo/bar.git"} at //gitlab.com/foo/bar?go-get=1
get "gitlab.com/foo/bar/example": found meta tag get.metaImport{Prefix:"gitlab.com/foo/bar", VCS:"git", RepoRoot:"https://gitlab.com/foo/bar.git"} at //gitlab.com/foo/bar/example?go-get=1
get "gitlab.com/foo/bar/example": verifying non-authoritative meta tag
go get gitlab.com/foo/bar/example: git ls-remote -q https://gitlab.com/foo/bar.git in /home/koala/Workspace/gopath/pkg/mod/cache/vcs/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx: exit status 128:
	The project you were looking for could not be found.
	fatal: Could not read from remote repository.
	
	Please make sure you have the correct access rights
	and the repository exists.

P.S. My repository is a private repository. I don’t know if it is a matter or not.

@gopherbot, please backport to 1.13: this is a regression, and prevents previously-working modules from being fetched.

I’m running into issues with GitLab subgroups as well, but I’m getting a different error:

$ go get -u example.com/group/subgroup/my-go-project.git
get "example.com/group/subgroup": found meta tag get.metaImport{Prefix:"example.com/group/subgroup", VCS:"git", RepoRoot:"https://example.com/group/subgroup.git"} at //example.com/group/subgroup?go-get=1
go: finding example.com/group/subgroup/my-go-project.git latest
go get example.com/group/subgroup/my-go-project.git: git ls-remote -q https://example.com/group/subgroup.git in C:\Users\dev\go\pkg\mod\cache\vcs\5af58fdf9c9bd0fd298253dde7a0add3f4e5d6be34206b0b46f6805ee365d4ea: exit status 128:
        remote: The project you were looking for could not be found.
        fatal: repository 'https://example.com/group/subgroup.git/' not found

It almost seems like it’s assuming that the second part of the path must be the project. Going back to 1.12 fixes it.

Edit: Nevermind, re-ran it with -v and it’s the same issue mentioned above with GitLab returning the parent.

have you tested private group or subgroup on gitlab?

No, I didn’t but tried it now with similar setup - commons group public, pkg subgroup public and syncs repo private (internal in gitlab’s permissions). I see very similar results:

go version go1.13.3 darwin/amd64

go get -v git.internal.com/commons/pkg/syncs
get "git.internal.com/commons/pkg": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg?go-get=1
get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg/syncs?go-get=1
get "git.internal.com/commons/pkg/syncs": verifying non-authoritative meta tag
go get git.internal.com/commons/pkg/syncs: git ls-remote -q https://git.internal.com/commons/pkg.git in /Users/umputun/go-home/pkg/mod/cache/vcs/882b5ac6f69791979e3abd70af30e871e32fe91173f2e25079ef6a88464ec13e: exit status 128:
	> GitLab: The project you were looking for could not be found.
	fatal: Could not read from remote repository.

	Please make sure you have the correct access rights
	and the repository exists.

Not sure if important, but to access private repos from go I use insteadOf rewrite:

 [url "git@git.internal.com:"]
     insteadOf = https://git.internal.com/

it can not work as expected.

@yookoala, you probably need to configure a git credential helper, or add a .netrc file containing credentials. (The go command is not intended to prompt for interactive logins.)

See https://git-scm.com/docs/gitcredentials and #29953.

Same problems…

Thanks @bcmills and @umputun for all that context.

Our plan is to start returning a 404 in response to the first stage ?go-get=1 request if the project doesn’t exist or if the user is not sufficiently authenticated. We’re tracking the issue https://gitlab.com/gitlab-org/gitlab/issues/30612. We don’t have capacity to look into this in the current development cycle, but I’ll try to have this scheduled soon. Of course, anyone from the community would also be welcome to contribute a fix too 😄

@bcmills - I had a chance to test this issue with gitea. For 1.13 the situation is similar to gitlab’s, i.e. the same regression (worked with 1.12 and fails on 1.13). However, with gotip it worked fine. gitea seems to return go meta on any path, pretty much the same way as gitlab. However for those non-existing repos gitea, unlike gitlab, returns 404.

Probably adding 401 to the list of the errors on the parent discovery you consider as “missing module” may fix it.

I don’t think that would be appropriate. Per RFC 2616 §10.4.2 (emphasis mine):

If the request already included Authorization credentials, then the 401 response indicates that authorization has been refused for those credentials. If the 401 response contains the same challenge as the prior response, and the user agent has already attempted authentication at least once, then the user SHOULD be presented the entity that was given in the response, since that entity might include relevant diagnostic information.

I can certainly make the case that we should interpret a repo 404 as “not found”, but a server that explicitly tells the go command to check a particular repo really should not also tell the go command that it is not authorized to know whether that repo actually exists.

At the very least, if a more RFC-compliant response code is not feasible I’d like to hear the rationale from someone at GitLab.