go: cmd/go: occasional "failed to cache compiled Go files" when 'go list -compiled' runs in parallel

What version of Go are you using (`go version`)?

Go 1.11.4

Does this issue reproduce with the latest release?

~~It does not reproduce with Go 1.12beta1.~~ It reproduces at tip at a somewhat lower rate. See @dominikh’s comment, below.

What operating system and processor architecture are you using (`go env`)?

linux/amd64

What did you do?

@dominikh wrote a small reproducer: https://play.golang.org/p/ghOtMwBnGu7

It may take a few runs to fail. Clean out your build cache or use GOCACHE=/tmp/empty to make it more reliably fail.

What did you expect to see?

Calling go list in parallel should succeed.

What did you see instead?

go build <some package>: failed to cache compiled Go files

I think this happens only when -compiled is provided to go list (which is new in Go 1.11?)

The context in which this comes up is multiple tools running golang.org/x/tools/go/packages.Load at the same time. I believe the issue is in go list but possibly the bug (or at least the workaround) should be in golang.org/x/tools/go/packages.

I also believe this is much more likely to happen with an empty GOCACHE. We’ve mainly been seeing this our CI environment. github.com/google/wire has also been seeing this in Travis: https://github.com/google/wire/issues/66.

The effects of this issue tend to be pretty far away from the go list invocation. We’ve run into it via staticcheck (http://staticcheck.io/) as well as our own go/packages-using code.

~~I can’t get this to reproduce in 1.12beta1. It would be helpful if someone can find the CL that fixed it. I think it would also be valuable if the fix could be backported to Go 1.11 in the meantime.~~ (It does reproduce at tip.)

The ecosystem is rapidly converging on the new, go list-based way of writing tooling (typically via go/packages) so I expect this will arise often.

/cc @bcmills @dominikh @zombiezen @ianthehat

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 1
Comments: 18 (11 by maintainers)

Commits related to this issue

fix for GOCACHE failing in parallel https://github.com/golang/go/issues/29667 Co-authored-by: Maria Shaldibina <mshaldibina@pivotal.io> — committed to cloudfoundry/diego-release by deleted user 5 years ago
cmd/go/internal/cache: verify that timestamp is positive An apparent typo caused us to re-check size instead. Updates #29667 Change-Id: Icc1fb9cef1ba77b8490c8fbe6c343be06ff48d8d Reviewed-on: https:... — committed to golang/go by bcmills 5 years ago
cmd/go/internal/work: include detail in errors from cache operations Since the cache has been required since Go 1.12, also remove redundant checks for a nil cache. Updates #29127 Updates #29667 Cha... — committed to golang/go by bcmills 5 years ago

Most upvoted comments

We’re seeing this with go 1.11.5 and 1.12.5 (both the official Docker images).

After reading @cespare’s comment about putting a mutex on go list, we added this go wrapper to our build process, which works around the problem nicely, letting us perform parallel builds.

#!/bin/sh

set -e

for i in "$@"; do
	if [ "$i" = list ]; then
		set -o noclobber
		while ! echo "$$" >/tmp/go-list-lock; do
			sleep 0.25
		done
		trap 'rm -f /tmp/go-list-lock' EXIT
	fi
done

/usr/local/go/bin/go "$@"

jwm on Jun 11, 2019

This bug has been biting us a lot as we run more and more tools in CI environments which use go/packages. In case it helps other people, the workaround we recently started using is a shim binary that passes its arguments to the go tool. If it sees that it is being called as go list, it uses a global lock file to prevent simultaneous go list executions. In CI and script environments where the workaround is needed, the fake go is put in a directory at the front of the $PATH.

I also looked into writing a GOPACKAGESDRIVER tool which would apply the same lockfile-based mutual exclusion around go list but it looked like that would require copy-pasting a lot of code out of go/packages.

cespare on Feb 2, 2019