go: x/tools/go/packages: Load is 2-3 orders of magnitude slower than go/build.Import

As per @heschik in #29452

Converting a tool from go/build to go/packages incurs a huge slowdown. I tried to min-repro it here. Note that we have a full vendor/ dir, no un-vendored deps, and modules are NOT turned on yet.

What version of Go are you using (go version)?

1.12.1

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

linux, amd64

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/usr/local/google/home/thockin/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/usr/local/google/home/thockin/src/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/google-golang"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/google-golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build445939239=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Build and run the two programs below and time them.

What did you expect to see?

Roughly the same timing

What did you see instead?

0.008s vs 1.272s

go-build.go:

package main

import (
        "fmt"
        "go/build"
        "log"
        "os"

        "github.com/davecgh/go-spew/spew"
)

func main() {
        if err := visit("."); err != nil {
                fmt.Println(err)
                os.Exit(1)
        }
}

func visit(pkgName string) error {
        pkg, err := findPackage(pkgName)
        if err != nil {
                return err
        }
        spew := spew.ConfigState{DisableMethods: true, Indent: "  "}
        spew.Dump(pkg)
        return nil
}

func findPackage(pkgName string) (*build.Package, error) {
        log.Println("find", pkgName)
        pkg, err := build.Import(pkgName, getwd(), build.ImportComment)
        if err != nil {
                return nil, err
        }
        return pkg, nil
}

func getwd() string {
        pwd, err := os.Getwd()
        if err != nil {
                panic(fmt.Sprintf("can't get working directory: %v", err))
        }
        return pwd
}

go-packages.go:

package main

import (
	"fmt"
	"log"
	"os"

	"github.com/davecgh/go-spew/spew"

	"golang.org/x/tools/go/packages"
)

func main() {
	if err := visit("."); err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
}

func visit(pkgName string) error {
	pkgs, err := findPackage(pkgName)
	if err != nil {
		return err
	}
	spew := spew.ConfigState{DisableMethods: true, Indent: "  "}
	spew.Dump(pkgs[0])
	return nil
}

func findPackage(pkgName string) ([]*packages.Package, error) {
	log.Println("find", pkgName)
	pkgs, err := packages.Load(&packages.Config{Mode: packages.LoadFiles}, pkgName)
	if err != nil {
		return nil, err
	}
	return pkgs, nil
}

Testing:

$ ls
go-build.go  go-packages.go

$ go build ./go-build.go 

$ go build ./go-packages.go 

$ time ./go-build >/dev/null
2019/03/27 11:51:10 find .

real	0m0.008s
user	0m0.008s
sys	0m0.000s

$ time ./go-packages >/dev/null
2019/03/27 11:51:17 find .

real	0m1.272s
user	0m0.094s
sys	0m0.109s