go: runtime: hangs on Windows due to McAfee's mfehcinj.dll apparently hijacking or suspending threads
What version of Go are you using (go version
)?
go version go1.15.4 windows/amd64
Does this issue reproduce with the latest release?
I can reproduce it very regularly at least on my system.
What operating system and processor architecture are you using (go env
)?
go env
Output
set GO111MODULE= set GOARCH=amd64 set GOBIN= set GOCACHE=C:\Dev\Temp\go-build set GOENV=C:\Users\Alex\AppData\Roaming\go\env set GOEXE=.exe set GOFLAGS= set GOHOSTARCH=amd64 set GOHOSTOS=windows set GOINSECURE= set GOMODCACHE=C:\Dev\workdir\pkg\mod set GONOPROXY= set GONOSUMDB= set GOOS=windows set GOPATH=C:\Dev\workdir set GOPRIVATE= set GOPROXY=https://proxy.golang.org,direct set GOROOT=C:\Dev\go set GOSUMDB=sum.golang.org set GOTMPDIR=C:\Dev\Temp set GOTOOLDIR=C:\Dev\go\pkg\tool\windows_amd64 set GCCGO=gccgo set AR=ar set CC=gcc set CXX=g++ set CGO_ENABLED=1 set CGO_CFLAGS=-g -O2 set CGO_CPPFLAGS= set CGO_CXXFLAGS=-g -O2 set CGO_FFLAGS=-g -O2 set CGO_LDFLAGS=-g -O2 set PKG_CONFIG=pkg-config set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Dev\Temp\go-build995151114=/tmp/go-build -gno-record-gcc-switches
What did you do?
I experienced a lot of freezes for several days now, when using the command:
go generate ./..
in one of my projects.
go generate
does trigger four different mockgen commands, thats all.
These mockgen commands in turn trigger some go list
and go build
commands, which freeze very often.
I don’t think it has anything to do with go generate or mockgen. It just seems to trigger the issue more often as when I just do go build
. But also go build
as a standalone command sometimes triggered this issue.
What did you expect to see?
Successful execution of all go commands that were called from go generate --> mockgen.
What did you see instead?
The main issue here is, that go build
and also go list
that is executed by mockgen freezes very often.
In windows, by using a process monitor tool, I can see, that go generate
called mockgen
that in turn called go build
.
It happens very often, that the go build -o prog.bin.exe prog.go
process is does not return and seems to be stuck.
I also found out, when setting GOMAXPROCS=1, go generate ./...
works perfectly without a single one of this issues appearing.
Furthermore when go build
was stuck I attached with dlv and got the following list of goroutines:
Goroutine 1 - User: c:/go/src/runtime/sema.go:56 sync.runtime_Semacquire (0x756e89)
Goroutine 2 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
* Goroutine 3 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
Goroutine 4 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
Goroutine 5 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
Goroutine 15 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
Goroutine 18 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
Goroutine 19 - User: c:/go/src/runtime/symtab.go:940 runtime.step (0x744a7e) (thread 30604)
Goroutine 20 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
Goroutine 1267 - User: c:/go/src/cmd/go/internal/work/exec.go:169 cmd/go/internal/work.(*Builder).Do.func3 (0xb6f83f)
Goroutine 1268 - User: c:/go/src/cmd/go/internal/work/exec.go:169 cmd/go/internal/work.(*Builder).Do.func3 (0xb6f83f)
Goroutine 1269 - User: c:/go/src/cmd/go/internal/work/exec.go:169 cmd/go/internal/work.(*Builder).Do.func3 (0xb6f83f)
Goroutine 1270 - User: :0 ??? (0x7ffbb80dc144) (thread 21588)
The following goroutines are waiting at the provided code lines:
1 waits at: c:/go/src/cmd/go/internal/work/exec.go:188
1267 waits at: c:/go/src/cmd/go/internal/work/exec.go:169
1268 waits at: c:/go/src/cmd/go/internal/work/exec.go:169
1269 waits at: c:/go/src/cmd/go/internal/work/exec.go:169
I think the other goroutines are not that interesting, they look like this:
the other routines look something like this:
0 0x000000000072be3a in runtime.gopark
at c:/go/src/runtime/proc.go:307
1 0x0000000000715929 in runtime.goparkunlock
at c:/go/src/runtime/proc.go:312
2 0x0000000000715929 in runtime.bgsweep
at c:/go/src/runtime/mgcsweep.go:182
3 0x000000000075b9a1 in runtime.goexit
at c:/go/src/runtime/asm_amd64.s:1374
or this:
0 0x000000000072be3a in runtime.gopark
at c:/go/src/runtime/proc.go:307
1 0x000000000070e174 in runtime.gcBgMarkWorker
at c:/go/src/runtime/mgc.go:1891
2 0x000000000075b9a1 in runtime.goexit
at c:/go/src/runtime/asm_amd64.s:1374
As mentioned, this issue does not appear, when using GOMAXPROCS=1. Otherwise it appears nearly always.
Any idea?
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 25 (12 by maintainers)
I disagree.
I think Go should have asyncpreemptoff=1 by default on Windows.
None of our Windows users complained about preemption being a problem, but now @ybbus and I are suffering.
So far, we only know about developers are affected. But maybe this also affects users who run our programs.
And, while I agree that the problem is most likely with buggy antivirus software, that is the world we have to live in.
Alex
You’re probably better off setting asyncpreemptoff, which switches the runtime to cooperative scheduling of goroutines on available cores (GOMAXPROCS) – it was the default until 1.14.
Since that fixes it, maybe McAfee (or your IT dept) has an update planned.
Ah, one other experiment you could try (via #36494). @ybbus, what happens if you set
GODEBUG=asyncpreemptoff=1
in the environment, still withGOMAXPROCS=8
? (Does that change the frequency of the hangs?)(And by the way, @ybbus, thanks for the detailed investigation of what’s running on those threads!)
Seems this is a bug in mcafee av
Just another thought: It seems that mcafee acquires one or two threads in the running go process. Maybe the following could be an explanation:
MAXPROCS=1 mcafee does not even have a chance to start a thread and do strange stuff --> everything works
MAXPROCS=8 enough threads (time) to start up the mcafee hook. since mcafee wants to check the binary before execution, it blocks evrything else but needs some other threads, that it does not get because of the MAXPROCS=8, so it waits forever and also blocks all others forever go runtime also waits for a thread, since it does not really know that there is some “stranger” also requesting threads and so go also blocks because go runtime is not aware of threads beeing stolen.
That is my guess and would fit to my following observation:
The build again runs smoothly, when setting GOMAXPROCS to a high value like 256. At least 90% of the time, unfortunately it also gets stuck time by time
Maybe this is enough on the one hand for go itself and on the other hand also for mcafee to do whatever it does…
64 and 128 for example still does not seem to be enough --> again always hanging
256 only works when directly invoking go build. When trying this with
go generate ./..
it again hangs after sometime. At least later in time. When setting to 1024 , then also the go generate runs without problems (most of the time)Any thoughts on that? I can just guess here since I have no real clue.
Hi Dave, I will try that out. In the meantime I also checked a bit more using Sysinternals Process explorer.
It seems that even though I disabled most of the McAfee stuff, it still tries to be part of the problem I think. I did not even know that is is going to directly inject itself in other processes, but that is what you can see on the screenshot.
The stack of all threads of the hanging go processes look the same and seem to “WaitForSingleObject”. The only stack that looks different is the one that also seems to have something injected from McAfee.
I know too little of the low level windows os stuff to have some opinion about this, but for me it actually looks like some issue with mcafee.
I just don’t get, why there are tons of programs running on my machine that don’t have a similar problems with McAffe. Seems to be something go specific? Also it is quite strange that GOMAXPROCS seems to make a huge difference here.