go: cmd/go: can't run go builds concurrently if they download modules
What version of Go are you using (go version)?
go1.11beta2
What operating system and processor architecture are you using (go env)?
amd64 docker container
Our current build workflow is to run glide install first and then start a number of go build jobs in parallel.
When I take a clean system and start the go build jobs in parallel they fail due to issues downloading modules, git shallow.lock files and race conditions creating and populating /go/src/mod directories.
If I first do something like go get ./foo ./bar ./baz to download the modules I can run the builds in parallel.
Is there an equivalent of glide install or dep ensure?
Is there a plan to make downloading of modules thread safe?
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 20 (4 by maintainers)
Commits related to this issue
- Simplify travis-ci build This change avoids the need to copy the source out of the GOPATH by applying GO111MODULE=on as an environment variable. Additionally, avoids issues with parallel module downl... — committed to wfscheper/hugo by wfscheper 6 years ago
- Simplify travis-ci build This change avoids the need to copy the source out of the GOPATH by applying GO111MODULE=on as an environment variable. Additionally, avoids issues with parallel module downl... — committed to wfscheper/hugo by wfscheper 6 years ago
- Simplify travis-ci build This change avoids the need to copy the source out of the GOPATH by applying GO111MODULE=on as an environment variable. Additionally, avoids issues with parallel module downl... — committed to wfscheper/hugo by wfscheper 6 years ago
- cmd/go: support background processes in TestScript This will be used to test fixes for bugs in concurrent 'go' command invocations, such as #26794. See the README changes for a description of the se... — committed to golang/go by bcmills 6 years ago
- internal/syscall/windows: add LockFileEx and UnlockFileEx for use in cmd/go Updates #26794 Change-Id: Ic1d3078176721f3d2e5d8188c234383037babbaf Reviewed-on: https://go-review.googlesource.com/c/1451... — committed to golang/go by bcmills 6 years ago
- cmd/go/internal/renameio: add package renameio.WriteFile writes files atomically by renaming temporary files. See the subsequent changes for usage examples. Updates #26794 Updates #22397 Change-Id... — committed to golang/go by bcmills 6 years ago
- cmd/go/internal/cache: write shared mutable files atomically Updates #26794 Change-Id: I2a50e3b756ff6a2bbaee4737ca7ed053b01c8d0e Reviewed-on: https://go-review.googlesource.com/c/146378 Reviewed-by:... — committed to golang/go by bcmills 6 years ago
- cmd/go/internal/lockedfile: add package and support library lockedfile.File passes through to os.File, with Open, Create, and OpenFile functions that mimic the corresponding os functions but acquire ... — committed to golang/go by bcmills 6 years ago
- cmd/go/internal/{clean,test}: lock testexpire.txt Also check to make sure we don't overwrite a newer timestamp with an older one. testexpire.txt may be written concurrently, and a partially-written ... — committed to golang/go by bcmills 6 years ago
- cmd/go/internal/modfetch: make Repo.Zip write to an io.Writer instead of a temporary file This will be used to eliminate a redundant copy in CL 145178. (It also decouples two design points that were... — committed to golang/go by bcmills 6 years ago
- cmd/go/internal/modfetch: lock files and directories We employ the following new locking mechanisms: • Zip files and list files within the module cache are written using atomic renames of temporar... — committed to golang/go by bcmills 6 years ago
- cmd/go/internal/{modcmd,modload}: lock edits to go.mod Use an arbitrary lockfile to serialize edits, and use atomic renames to actually write the go.mod file so that we never drop version requirement... — committed to golang/go by bcmills 6 years ago
- cmd/go/internal/modfetch/codehost: add lockfiles for repos The lockfile guards calls that may change the repo's filesystem contents. We don't know how robust VCS implementations are to running simul... — committed to golang/go by bcmills 6 years ago
- cmd/go/internal/modfetch: make directories read-only after renaming, not before The call to os.Rename was failing on the darwin builders, despite having passed in the TryBots. (I tested this change b... — committed to golang/go by bcmills 6 years ago
We have many services in a monorepo written in a number of different languages. The top level CI builder doesn’t know about what language they are written it. It just invokes a number of Makefiles most of which happen to be Go programs.
I did switch our build process to call
go mod downloadas a pre-build step and that is working fine for us. This works around the race condition.I haven’t gone back to test this but the issue of having parallel builds fail due to races for the on disk module cache may still remain.
We have a different “multiple builds at once” use-case at hashicorp/terraform: we build the same application source (and thus the same dependencies) for a number of different
GOOS/GOARCHpairs concurrently during our release process.We are currently doing this with
gox, which is a wrapper aroundgo buildthat deals with templating the executable output paths to produce a systematic result. A number of other projects at HashiCorp, and I believe also elsewhere, are using this tool in the same way.It sounds like it may be viable for
goxto simply run an additional single command to completion before it starts its parallel work to ensure that all of the dependencies are available. Would that bego list ./..., or similar?