go: x/build: "unexpected stale targets" on darwin builders

Two flakes in the darwin-amd64-10_11 builders over the past week or so, both with the same symptom: unexpected stale targets failures for internal/cpu, and a build ID mismatch for runtime/internal/sys.

CC @jayconrod @ianlancetaylor @aclements

darwin-amd64-10_11 at 0ca4f6be35d2abf54128b57c3aa1a1fcbd5adfe3

:: Running /var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/make.bash with args ["/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/make.bash"] and env ["TERM_PROGRAM=Apple_Terminal" "TERM=xterm-256color" "SHELL=/bin/bash" "TMPDIR=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/" "Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.55fT0Y9EVR/Render" "TERM_PROGRAM_VERSION=361.1" "TERM_SESSION_ID=D33F650A-CE77-4BF0-B95E-A37406DCC122" "USER=gopher" "SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.43nQd98RPC/Listeners" "__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0" "PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin" "PWD=/Users/gopher" "LANG=en_US.UTF-8" "XPC_FLAGS=0x0" "XPC_SERVICE_NAME=0" "SHLVL=2" "HOME=/Users/gopher" "LOGNAME=gopher" "SECURITYSESSIONID=186a8" "GO_BUILDER_ENV=macstadium_vm" "GO_STAGE0_NET_DELAY=300ms" "GO_STAGE0_DL_DELAY=0s" "WORKDIR=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir" "GOROOT_BOOTSTRAP=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go1.4" "GO_BUILDER_NAME=darwin-amd64-10_11" "GOROOT_BOOTSTRAP=/Users/gopher/go1.4" "GOBIN=" "TMPDIR=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/tmp" "GOCACHE=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/gocache"] in dir /var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src

Building Go cmd/dist using /Users/gopher/go1.4.
Building Go toolchain1 using /Users/gopher/go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
HASH[build runtime/internal/sys]
HASH[build runtime/internal/sys]: "devel 0ca4f6be35d2abf54128b57c3aa1a1fcbd5adfe3"
HASH[build runtime/internal/sys]: "compile\n"
HASH[build runtime/internal/sys]: "goos darwin goarch amd64\n"
HASH[build runtime/internal/sys]: "import \"runtime/internal/sys\"\n"
HASH[build runtime/internal/sys]: "omitdebug false standard true local false prefix \"\"\n"
HASH[build runtime/internal/sys]: "modinfo \"\"\n"
HASH[build runtime/internal/sys]: "compile PRWJHAzAczrrtWSoyeY4 [] []\n"
HASH[build runtime/internal/sys]: "=\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/arch.go: d9b0b7e72538d421b2607acaba60ca49f20ef584b3d1d191c6729e35fbb8101d
HASH[build runtime/internal/sys]: "file arch.go 2bC35yU41CGyYHrKumDK\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/arch_amd64.go: d301f8642ec7fcc35d0d39a4b07e0187971afea7c6d0dfd5ddcef1d031670160
HASH[build runtime/internal/sys]: "file arch_amd64.go 0wH4ZC7H_MNdDTmksH4B\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/intrinsics.go: 8b469a461e1d983706e0b3635715ce70691adc5db7c4e067b88cc59f40cd66f4
HASH[build runtime/internal/sys]: "file intrinsics.go i0aaRh4dmDcG4LNjVxXO\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/stubs.go: 23b3e5c631b086fe7a2dec4bf044600e034bf6a8eeb25e0a19efc4ce6311423d
HASH[build runtime/internal/sys]: "file stubs.go I7PlxjGwhv56LexL8ERg\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/sys.go: 55e021891200a7e6a5c371c8a1ab71b6c15aeb16ea6c1b192185d17df8c8b18f
HASH[build runtime/internal/sys]: "file sys.go VeAhiRIAp-alw3HIoatx\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/zgoarch_amd64.go: 6f2fa4ebe8999a3b7cfe3ead6e24b8356ed842292f23cb1b4f995c0b5b45126b
HASH[build runtime/internal/sys]: "file zgoarch_amd64.go by-k6-iZmjt8_j6tbiS4\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/zgoos_darwin.go: 464151a4ab63f8c7aab345f457789cb12c9b68051fc73b8a675717d7886214dc
HASH[build runtime/internal/sys]: "file zgoos_darwin.go RkFRpKtj-Meqs0X0V3ic\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/zversion.go: 623491756e2adacebf4c53078883e4284349c6159d0404aba9975887d7660d06
HASH[build runtime/internal/sys]: "file zversion.go YjSRdW4q2s6_TFMHiIPk\n"
HASH[build runtime/internal/sys]: cd175367cc0e62c53ed8e20197f8e44b2270f61d3eb7b4f3c1991a5abb06ae16
runtime/internal/sys true
go tool dist: unexpected stale targets reported by /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/pkg/tool/darwin_amd64/go_bootstrap list -gcflags="" -ldflags="" for [cmd/asm cmd/cgo cmd/compile cmd/link runtime/internal/sys]:
	STALE cmd/asm: stale dependency: internal/cpu
	STALE cmd/cgo: stale dependency: internal/cpu
	STALE cmd/compile: stale dependency: internal/cpu
	STALE cmd/link: stale dependency: internal/cpu
	STALE runtime/internal/sys: build ID mismatch

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 1
Comments: 33 (24 by maintainers)

Commits related to this issue

cmd/go: skip the cmd/cgo staleness check on darwin builders Some of the darwin-amd64 builders are providing a stale environment. Let's un-break them while we investigate. Updates #37573 Updates #335... — committed to golang/go by bcmills 4 years ago
cmd/go,cmd/link: do not check for staleness in most tests Instead, check that stale packages in the standard library are not rebuilt when already present in the build cache, and are not installed imp... — committed to golang/go by bcmills 3 years ago
cmd/dist: consistently set PWD when executing a command in a different directory For #33598 Change-Id: If0de906ffa2fcc83bb2a90f9e80a5b29d7667398 Reviewed-on: https://go-review.googlesource.com/c/go/... — committed to golang/go by bcmills 3 years ago
all: consistently set PWD to match Dir for subprocesses Also consistently deduplicate Env entries, mainly to reduce confusion when reading logs. This change is probably larger than what is strictly ... — committed to golang/build by bcmills 3 years ago
dashboard: update host-ios-arm64-corellium-ios to match its previous env In CL 353549 I refactored environment-handling in the buildlet such that an explicit per-host GOROOT_BOOTSTRAP environment con... — committed to golang/build by bcmills 3 years ago
cmd/go: insert goroot to the hash of build cache when the packages include C files This reverts CL 351851, which itself reverted CL 348991. The problem with the original CL, as far as I can tell, wa... — committed to golang/go by bcmills 3 years ago
cmd/dist: run a checkNotStale on the builders before running tests This should be a no-op, but if the sharded builders are for some reason distributing stale snapshots — or testing them with mismatch... — committed to golang/go by bcmills 4 years ago
[release-branch.go1.17] cmd/dist: consistently set PWD when executing a command in a different directory Fixes #52995 Updates #33598 Change-Id: If0de906ffa2fcc83bb2a90f9e80a5b29d7667398 Reviewed-on:... — committed to golang/go by bcmills 3 years ago
buildlet,cmd/coordinator: clean up and document buildlet path invariants For golang/go#53957. Updates golang/go#33598. Change-Id: Ie2c47dc0dfd82614b70c983f11462b70bd0703d2 Reviewed-on: https://go-re... — committed to golang/build by bcmills 3 years ago

Most upvoted comments

Delightful! CL 353549 does seem to have done the trick. We could submit CL 223755 to prevent future regressions, but if I understand correctly it might seriously degrade the performance of certain builders (notably plan9). It was very useful for debugging but I’m indifferent as to whether we actually merge it.

bcmills on Oct 8, 2021

I mailed out a CL to fix this, but I think it will require some deployment work to determine whether it fixes the problem.

I think the change in CL 353549 most relevant to the issue in https://github.com/golang/go/issues/33598#issuecomment-931637710 is in x/build/cmd/buildlet that runs on builders. Builders are generally configured with a “stage0” that’s responsible for updating the buildlet binary whenever a new one becomes uploaded, so we should see results after we do so.

dmitshur on Oct 6, 2021

This builder issue is blocking the fix for #48319, which is a real bug that I would like to be able to fix for Go 1.18.

bcmills on Oct 1, 2021

Looking at the latest test failure from CL 223755, I see:

Installed Go for darwin/amd64 in /private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go

That message comes from here, and implies that GOROOT is /private/…: https://github.com/golang/go/blob/b1bedc0774d8a3a7ff8778e933ee92e8638e9493/src/cmd/dist/build.go#L1707-L1708

But when I gomote ssh into an instance, I see:

# `gomote push` and the builders use:
# - workdir: /var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14
# - GOROOT: /var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go
# - GOPATH: /var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/gopath
# - env: GO_BUILDER_NAME=darwin-amd64-10_14 GOROOT_BOOTSTRAP=/Users/gopher/goboot

Note that these lack the /private prefix. So I think something somewhere is forgetting to set PWD and/or GOROOT, and as a result the implicit GOROOT is ending up as whatever the OS reports instead. (To my recollection, on macOS /var is a symlink to /private/var.)

bcmills on Sep 30, 2021