go: cmd/go: get: panic: internal error: can't find reason for requirement on golang.org/x/sync@...

We’re seeing this happen for a lot of users of Caddy trying to build with plugins added. In our case, it fails on pprof.

We use our build tool xcaddy which is essentially a glorified shell script (written in Go) that just strings a bunch of go commands together to make a build.

See details below, basically makes a main.go, then go mod init caddy, then go get for each plugin in order, also including the main Caddy module to ensure plugins don’t inadvertently use a more recent version of Caddy (happens sometimes if the plugin’s main branch changes its go.mod to a beta release of Caddy), then go mod tidy and go build.

It doesn’t get to the go build in this case, because it panics during go get.

$ go version                                                                                                   
go version go1.20 linux/amd64

$ xcaddy build v2.6.3 --with github.com/dulli/caddy-wol@a8cbd416
2023/02/10 03:17:04 [INFO] Temporary folder: /tmp/buildenv_2023-02-10-0317.2187046618
2023/02/10 03:17:04 [INFO] Writing main module: /tmp/buildenv_2023-02-10-0317.2187046618/main.go
package main

import (
	caddycmd "github.com/caddyserver/caddy/v2/cmd"

	// plug in Caddy modules here
	_ "github.com/caddyserver/caddy/v2/modules/standard"
	_ "github.com/dulli/caddy-wol"
)

func main() {
	caddycmd.Main()
}
2023/02/10 03:17:04 [INFO] Initializing Go module
2023/02/10 03:17:04 [INFO] exec (timeout=10s): /usr/local/go/bin/go mod init caddy 
go: creating new go.mod: module caddy
go: to add module requirements and sums:
	go mod tidy
2023/02/10 03:17:04 [INFO] Pinning versions
2023/02/10 03:17:04 [INFO] exec (timeout=0s): /usr/local/go/bin/go get -d -v github.com/caddyserver/caddy/v2@v2.6.3 
go: added github.com/beorn7/perks v1.0.1
go: added github.com/caddyserver/caddy/v2 v2.6.3
go: added github.com/caddyserver/certmagic v0.17.2
go: added github.com/cespare/xxhash/v2 v2.1.2
go: added github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0
go: added github.com/golang/mock v1.6.0
go: added github.com/golang/protobuf v1.5.2
go: added github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38
go: added github.com/google/uuid v1.3.0
go: added github.com/klauspost/cpuid/v2 v2.2.3
go: added github.com/libdns/libdns v0.2.1
go: added github.com/matttproud/golang_protobuf_extensions v1.0.1
go: added github.com/mholt/acmez v1.0.4
go: added github.com/miekg/dns v1.1.50
go: added github.com/onsi/ginkgo/v2 v2.2.0
go: added github.com/prometheus/client_golang v1.14.0
go: added github.com/prometheus/client_model v0.3.0
go: added github.com/prometheus/common v0.37.0
go: added github.com/prometheus/procfs v0.8.0
go: added github.com/quic-go/qpack v0.4.0
go: added github.com/quic-go/qtls-go1-18 v0.2.0
go: added github.com/quic-go/qtls-go1-19 v0.2.0
go: added github.com/quic-go/qtls-go1-20 v0.1.0
go: added github.com/quic-go/quic-go v0.32.0
go: added go.uber.org/atomic v1.9.0
go: added go.uber.org/multierr v1.6.0
go: added go.uber.org/zap v1.24.0
go: added golang.org/x/crypto v0.5.0
go: added golang.org/x/exp v0.0.0-20221205204356-47842c84f3db
go: added golang.org/x/mod v0.6.0
go: added golang.org/x/net v0.5.0
go: added golang.org/x/sys v0.5.0
go: added golang.org/x/term v0.5.0
go: added golang.org/x/text v0.6.0
go: added golang.org/x/tools v0.2.0
go: added google.golang.org/protobuf v1.28.1
2023/02/10 03:17:05 [INFO] exec (timeout=0s): /usr/local/go/bin/go get -d -v github.com/dulli/caddy-wol@a8cbd416 github.com/caddyserver/caddy/v2@v2.6.3 
panic: internal error: can't find reason for requirement on github.com/google/pprof@v0.0.0-20210407192527-94a9f03dee38

goroutine 1 [running]:
cmd/go/internal/modget.(*resolver).updateBuildList.func1({{0xc000724ee8, 0x17}, {0xc0000cc6c0, 0x22}})
	/usr/local/go/src/cmd/go/internal/modget/get.go:1760 +0xd4
cmd/go/internal/modget.(*resolver).updateBuildList(0xc000142000, {0xb39158, 0xc0000ae000}, {0x0, 0x0, 0x0})
	/usr/local/go/src/cmd/go/internal/modget/get.go:1765 +0x54c
cmd/go/internal/modget.(*resolver).applyUpgrades(0xc000142000, {0xb39158, 0xc0000ae000}, {0x0?, 0x0, 0xc0000b5d70?})
	/usr/local/go/src/cmd/go/internal/modget/get.go:1312 +0x465
cmd/go/internal/modget.runGet({0xb39158, 0xc0000ae000}, 0xc0000d44b0?, {0xc0000c2160, 0x2, 0x2})
	/usr/local/go/src/cmd/go/internal/modget/get.go:351 +0x458
main.invoke(0xe4e240, {0xc0000c2130, 0x5, 0x5})
	/usr/local/go/src/cmd/go/main.go:225 +0x3d9
main.main()
	/usr/local/go/src/cmd/go/main.go:179 +0x7ce
2023/02/10 03:17:08 [FATAL] exit status 2

Edit: After a bit more investigation, it looks like this is because Caddy upgraded its quic-go dependency in https://github.com/caddyserver/caddy/commit/798c4a3ba48b55e85867780ed399dc084c15b322, which in turn pulled in pprof because of quic-go upgrading ginkgo to v2 in https://github.com/quic-go/quic-go/commit/870fbe7ab0b871d879767c83b9744d5d3ac54b97. That’s a test dependency, but it’s causing build issues 😭

Edit2: I tried forking the Caddy plugin in question, and bumping its dependency on Caddy to v2.6.3 (latest as of this writing) and go mod tidy, so that the pprof versions match, and that does fix the build. At least I have a workaround to tell users (i.e. ask plugin maintainers to bump their deps) but it’s unfortunate because it causes a horizon point for plugin compatibility.

Thank you for all your great work, @bcmills 😃

I’ve mailed CL 471595 with a proposed fix. Folks who are affected by this bug: please try out a build with that change (either now or after it is merged) and let me know if any problems remain.

That change may also fix some of the “infinite hang” problems, since it fixes a source of errors in the build list. (go get iterates until the build list stabilizes, and I observed some infinite loops in testing caused by pruned dependencies being reintroduced or repeatedly re-eliminated.)

@godcong, the best temporary workaround is probably to just manually update the require line for golang.org/x/sync to the version you want and then run go mod tidy to clean it up.

(This bug does not affect go mod tidy, only go get.)

I see exactly the behavior that you describe. Thank you for distilling it down to a simple reproducer!

@anacrolix, if you’re getting an infinite hang try running go get with -v and/or sending SIGQUIT, and please do file a separate issue.

@bcmills The CL fixes this issue for me too. Specifically in https://github.com/anacrolix/torrent/issues/421#issuecomment-1496917847 I saw

~/src/rms-torrent [2]% go1.19 get github.com/anacrolix/torrent@latest
panic: internal error: can't find reason for requirement on github.com/mattn/go-sqlite3@v1.14.16

goroutine 1 [running]:
cmd/go/internal/modget.(*resolver).updateBuildList.func1({{0x14000029640?, 0x140001b37a0?}, {0x140001a2fa8?, 0x14000112b20?}})
	/Users/anacrolix/src/go1.19/src/cmd/go/internal/modget/get.go:1760 +0xf8
cmd/go/internal/modget.(*resolver).updateBuildList(0x14000412000, {0x102b7ae90, 0x140001a2000}, {0x0, 0x0, 0x0})
	/Users/anacrolix/src/go1.19/src/cmd/go/internal/modget/get.go:1765 +0x450
cmd/go/internal/modget.(*resolver).resolveQueries(0x14000412000, {0x102b7ae90, 0x140001a2000}, {0x140001b4048, 0x1, 0x102b601e0?})
	/Users/anacrolix/src/go1.19/src/cmd/go/internal/modget/get.go:1243 +0x180
cmd/go/internal/modget.runGet({0x102b7ae90, 0x140001a2000}, 0x140001be420?, {0x1400019e1a0, 0x1, 0x1})
	/Users/anacrolix/src/go1.19/src/cmd/go/internal/modget/get.go:314 +0x38c
main.invoke(0x102e6e2c0, {0x1400019e190, 0x2, 0x2})
	/Users/anacrolix/src/go1.19/src/cmd/go/main.go:225 +0x2f8
main.main()
	/Users/anacrolix/src/go1.19/src/cmd/go/main.go:179 +0x7ac

And now I get

~/src/rms-torrent [2]% go get github.com/anacrolix/torrent@latest
go: upgraded github.com/anacrolix/torrent v1.49.0 => v1.49.1

I think I still get the infinite hang I mentioned at https://github.com/golang/go/issues/56494#issuecomment-1496919121.

@mholt, you could patch the Go toolchain to pretend that it’s 1.20, but in this sort of case — especially since https://github.com/quic-go/quic-go/issues/2727 was filed back in 2020 — it’s kind of “you break it, you get to keep all the pieces”. Reaching into release-specific internals of the standard library is definitely not something the Go project supports. 🤷‍♂️

Hi @clarkmcc I ended up upgrading the dependency that required the conflicting package. I recommend you start from there. Good luck.

Thank you. It worked for me!

I also ran into this issue, and can confirm that it still happens with go1.19.4 and go1.20rc2, using the following steps to reproduce:

% mkdir /tmp/standalone
% cd /tmp/standalone
% go mod init standalone
% go get github.com/evcc-io/evcc   

The error message is:

go: downloading github.com/evcc-io/evcc v0.0.0-20230106164214-dbf13052101f
panic: internal error: can't find reason for requirement on github.com/rogpeppe/go-internal@v1.6.1

goroutine 1 [running]:
cmd/go/internal/modget.(*resolver).updateBuildList.func1({{0xc00028fb40, 0x1f}, {0xc000122b70, 0x6}})
	/usr/local/go/src/cmd/go/internal/modget/get.go:1760 +0xd4
cmd/go/internal/modget.(*resolver).updateBuildList(0xc0001d0000, {0xb390f8, 0xc00002a138}, {0x0, 0x0, 0x0})
	/usr/local/go/src/cmd/go/internal/modget/get.go:1765 +0x54c
cmd/go/internal/modget.(*resolver).applyUpgrades(0xc0001d0000, {0xb390f8, 0xc00002a138}, {0x0?, 0x0, 0xc0000c1d70?})
	/usr/local/go/src/cmd/go/internal/modget/get.go:1312 +0x465
cmd/go/internal/modget.runGet({0xb390f8, 0xc00002a138}, 0xc00002c660?, {0xc0000241a0, 0x1, 0x1})
	/usr/local/go/src/cmd/go/internal/modget/get.go:351 +0x458
main.invoke(0xe4e240, {0xc000024190, 0x2, 0x2})
	/usr/local/go/src/cmd/go/main.go:225 +0x3d9
main.main()
	/usr/local/go/src/cmd/go/main.go:179 +0x7ce

As a workaround, I added all the modules in the panic message to some unused internal package in my main module and in order to break the cycle and that fixed it: https://github.com/ipfs/kubo/blob/ee11ef6d2ca0df494e4d709f8354109452185a47/internal/i-hate-what-I-am-doing-here.go

Hi @clarkmcc I ended up upgrading the dependency that required the conflicting package. I recommend you start from there. Good luck.

@godcong Removing all mention of golang.org/x/sync from go.mod then running my original go get command fixed it all for me.