go: cmd/go: `go env GOMOD` is too slow for use in interactive commands

We’re porting a lot of tools to support modules, and most of them need to know whether modules are enabled to turn on the newer, slower, potentially buggier code path. Running go env GOMOD takes about 10ms, which is not negligible for an interactive command like godef or guru. The algorithm for determining if modules are on is just simple enough to tempt people to roll their own. In fact, there’s already a second implementation in go/build: https://github.com/golang/go/blob/429bae715876c69853bb63db1733f580e293c916/src/go/build/build.go#L977

This is unfortunate, especially since the algorithm changed in http://golang.org/cl/148517. (Maybe not in a way that matters for go/build? Not sure.)

I think it’d be helpful to have a standalone internal package that implemented this functionality that could be used for go env, go/build, and copied into x/tools for other tools to use.

@bcmills @jayconrod @ianthehat

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 27 (18 by maintainers)

Commits related to this issue

internal/lazyregexp: add a lazy Regexp package This was implemented as part of go/doc, but it's going to be useful in other packages. In particular, many packages under cmd/go like web and vcs make s... — committed to golang/go by mvdan 5 years ago
cmd/go: add benchmark that execs 'go env GOARCH' 'go env' is used for many quick operations, such as in go/packages to query GOARCH and GOMOD. It often is a bottleneck; for example, go/packages doesn... — committed to golang/go by mvdan 6 years ago
cmd/go: avoid compiling most regexes at init These regexes are all related to commands like get and build, so they're unnecessary for simpler commands like env. In particular, we need env to be fast,... — committed to golang/go by mvdan 6 years ago
internal/lazytemplate: add a lazy template wrapper Similar to internal/lazyregexp, this will allow removing unnecessary work from init functions with trivial refactors, thanks to sync.Once. Copy the... — committed to golang/go by mvdan 5 years ago
cmd/go: delay parsing the testmain template The template is over a hundred lines and full of pipelines, and text/template isn't optimised to parse quickly, so it's no wonder that delaying the parsing... — committed to golang/go by mvdan 6 years ago
errors: record only single frame See Issue #29382 and Issue #30468. 3 frames are no longer needed as of https://go-review.googlesource.com/c/go/+/152537/ name old time/op new t... — committed to golang/go by mpvl 5 years ago
errors: improve performance of New See Issue #29382 and Issue #30468. Improvements in this CL: name old time/op new time/op delta New-8 352ns ± 2% 225ns ... — committed to golang/go by mpvl 5 years ago
cmd/go: further reduce init work The first biggest offender was crypto/des.init at ~1%. It's cryptographically broken and the init function is relatively expensive, which is unfortunate as both crypt... — committed to golang/go by mvdan 5 years ago

Most upvoted comments

I’ve found a few more ways to remove work from cmd/go, and now go env GOARCH went from ~5ms to ~2.3ms on the added benchmark. Similarly, time go env GOARCH has gone from ~15-20ms to ~10-12ms on my laptop, though those numbers jump around much more than the benchmark ones.

I think we should be able to fairly easily shave off some more from cmd/go’s init for 1.13. On perf report, regexp.MustCompile still owns 7%, and go/build.init is an astonishing 10%, so I think we could realistically shave off another 20% or even more.

I’ll stop with the CLs for now, to get some input on the benchmark and style of the CLs. Happy to work on more once these have been approved.

mvdan on Dec 30, 2018

time go env GOARCH on Bash seems to give numbers around 10ms, while the benchmark gives numbers around 4ms.

echo 'package main; func main(){}' > c.go
go build ./c.go
time ./c

time reports 5-8 milliseconds to run a no-op go program on my machine, so it’s probably overhead from loading/starting the process and whatever go runtime initialization is involved. time /usr/bin/true (an actual 17k compiled executable) takes 3-6 milliseconds on my machine so the time for the no-op go program seems reasonable to me.

mark-rushakoff on Dec 21, 2018

The algorithm has changed before, and will likely change again (whenever the default behavior switches over). Probably it’s better to make go env GOMOD fast than to duplicate it with version skew.

I’m a total novice with the Linux perf tool, but it seems to imply that we’re spending a significant fraction of our latency initializing maps, regexps, and templates that aren’t actually going to be used.

From

$ GOMAXPROCS=1 perf record --freq=max --call-graph fp -- go env GOMOD
$ perf report

I get the following:

-   76.18%     0.00%  go       go                  [.] runtime.main                                                                  ◆
   - runtime.main                                                                                                                    ▒
      - 71.40% main.init                                                                                                             ▒
         - 61.68% cmd/go/internal/bug.init                                                                                           ▒
            - cmd/go/internal/envcmd.init                                                                                            ▒
               - 48.18% cmd/go/internal/modload.init                                                                                 ▒
                  - 46.52% cmd/go/internal/modfetch.init                                                                             ▒
                     - 41.80% cmd/go/internal/get.init                                                                               ▒
                        - 23.85% cmd/go/internal/work.init                                                                           ▒
                           + 22.31% regexp.MustCompile                                                                               ▒
                           + 0.82% cmd/go/internal/work.init.0                                                                       ▒
                           + 0.72% debug/elf.init                                                                                    ▒
                        - 8.70% cmd/go/internal/get.init.1                                                                           ▒
                             regexp.MustCompile                                                                                      ▒
                           + regexp.compile                                                                                          ▒
                        - 7.08% cmd/go/internal/web.init                                                                             ▒
                           - 4.61% net/http.init                                                                                     ▒
                              - 1.91% internal/x/net/http/httpguts.init                                                              ▒
                                   internal/x/net/idna.init                                                                          ▒
                                 - internal/x/text/unicode/norm.init                                                                 ▒
                                      1.25% runtime.mapassign_fast32                                                                 ▒
                                      0.66% runtime.aeshash32                                                                        ▒
                              - 1.37% internal/x/net/http2/hpack.init                                                                ▒
                                   internal/x/net/http2/hpack.newStaticTable                                                         ▒
                                 + runtime.mapassign                                                                                 ▒
                                0.70% runtime.mapassign_fast64                                                                       ▒
                              + 0.63% mime/multipart.init                                                                            ▒
                           - 2.47% crypto/tls.init                                                                                   ▒
                              + 1.82% crypto/des.init                                                                                ▒
                              + 0.65% crypto/x509.init                                                                               ▒
                        + 0.81% regexp.MustCompile                                                                                   ▒
                        + 0.75% cmd/go/internal/get.init.0                                                                           ▒
                          0.61% crypto/tls.init                                                                                      ▒
                     + 2.73% cmd/go/internal/modfetch/codehost.init                                                                  ▒
                     + 1.15% archive/zip.init                                                                                        ▒
                     + 0.83% regexp.MustCompile                                                                                      ▒
                  + 0.84% cmd/go/internal/imports.init                                                                               ▒
                  + 0.83% regexp.MustCompile                                                                                         ▒
               - 12.96% cmd/go/internal/load.init                                                                                    ▒
                    text/template.(*Template).Parse                                                                                  ▒
                    text/template/parse.Parse                                                                                        ▒
                    text/template/parse.(*Tree).Parse                                                                                ▒
                  + text/template/parse.(*Tree).parse                                                                                ▒
         - 4.54% cmd/go/internal/base.init                                                                                           ▒
            - cmd/go/internal/cfg.init                                                                                               ▒
               - 3.83% go/build.init                                                                                                 ▒
                  - 1.96% go/doc.init                                                                                                ▒
                     + 1.38% regexp.MustCompile                                                                                      ▒
                  + 0.78% go/build.defaultContext                                                                                    ▒
         + 1.64% cmd/go/internal/test.init                                                                                           ▒
         + 1.58% flag.init                                                                                                           ▒
         + 0.83% cmd/go/internal/list.init                                                                                           ▒
         + 0.83% cmd/go/internal/clean.init                                                                                          ▒
      + 3.87% main.main                                                                                                              ▒

bcmills on Dec 21, 2018