go: x/tools/gopls: imports extremely slow with large module cache

What version of Go are you using (`go version`)?

$ go version
go version devel +44c9354c5a Fri Jun 21 05:21:30 2019 +0000 linux/amd64
$ go list -m golang.org/x/tools
golang.org/x/tools v0.0.0-20190620191750-1fa568393b23
$ go list -m golang.org/x/tools/gopls
golang.org/x/tools/gopls v0.0.0-20190620191750-1fa568393b23

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN="/home/myitcv/gostuff/src/github.com/myitcv/govim/cmd/govim/.bin"
GOCACHE="/home/myitcv/.cache/go-build"
GOENV="/home/myitcv/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/myitcv/gostuff"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/myitcv/gos"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/myitcv/gos/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/myitcv/gostuff/src/github.com/myitcv/govim/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build977371100=/tmp/go-build -gno-record-gcc-switches"

What did you do?

The issue I keep tripping across is making refactoring changes that, for example, rename variables. If I fail to rename all instances of the variable (perhaps I’ve copy-pasted some code from somewhere), there’s a chance I’ve left behind a selector expression with the old variable name… which is now treated by imports as if it were a qualified identifier for a package that has not yet been imported.

Like https://github.com/golang/go/issues/32726, imports then heads off into the module cache searching through all potential matches. With a 5GB module cache, this is not a quick operation.

I think imports should:

pre-load and cache its results from the module cache
have that cache be invalidated where the modification time of any non-module cache-based go.{mod,sum} reachable from the main module changes
when the cache is invalidated, re-warm the cache in the background

Yes, this will cause possibly stale results in the case that another main module causes changes in the module cache, but I think that’s a sacrifice worth making for the general speed of imports.

What did you expect to see?

Instant responses to imports-based format-on-save.

What did you see instead?

~10sec delays.

cc @stamblerre @ianthehat @heschik

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 14
Comments: 34 (20 by maintainers)

Most upvoted comments

Can you sync to master (go get -u golang.org/x/tools/gopls@master)? I just submitted a fix for this nil pointer.

stamblerre on Jul 1, 2019

Thanks. See some benefits here as a result of https://golang.org/cl/184921 and https://golang.org/cl/186301.

Pre-warming of this cache has not been implemented yet.

This will also help. However, for Vim users (via whatever plugin) this will also likely become something of a pain. Because Vim users typically start/stop Vim regularly, especially when used from the terminal. Hence, if gopls is being started/stopped each time (communicating over stdin/stdout) then every time Vim is started (and there are often multiple instances of Vim open) this pre-warming will happen.

So any pre-warming will, I suspect, need to go hand-in-hand with ensuring the socket-based communication with gopls is on a par with the pipe-based approach. I would guess it is, but cc @stamblerre and @ianthehat for thoughts?

myitcv on Sep 5, 2019

We have started reusing the imports resolver in gopls (https://golang.org/cl/184921) and added some caching of packages in the module cache (https://golang.org/cl/186301), which both work to speed up imports when there have been no go.mod file changes.

A new caching design is currently being implemented to avoid repeat work of scanning the module cache.

Pre-warming of this cache has not been implemented yet.

suzmue on Aug 20, 2019

Chiming in here to say I’ve been staying reasonably current on gopls and things seem to be working much better for me now. I can’t pinpoint the exact time things got better, but I haven’t had a case of getting stuck importing and needing to resolve by hand for at least a week.

bmhatfield on Aug 1, 2019