go: cmd/dist: remove precompiled .a files from binary distributions [freeze exception]

I think it’s pretty important that a Go binary installation work on a system with no C compiler installed. And at least in the past on macOS Go programs would only work if using the cgo version of the net package. So I think we at least need to provide the .a files for the standard library packages that use cgo, which I think is currently net and os/user.

@ianlancetaylor

I think it’s pretty important that a Go binary installation work on a system with no C compiler installed.

I would agree with you a priori, but as I understand the current state of the world, the introduction of the build cache broke this case many releases back, with no complaints (or at least not enough that we’ve tried to fix it). The .a files that ship do not actually have the right cache keys embedded inside them to be used by essentially any out-of-the-box install of Go. Instead, they get recomputed (in the cache only, not in the install location) the first time you build something.

Assuming I am right about that (I have not done the experiment myself), then I think it is OK to just drop the .a files entirely and not worry about the “cgo without C compiler” case anymore.

Another reason to do this is that internet speeds vary wildly. To some people, downloading an extra 50MiB is a split second, but to many others it’s staring at their screen for an extra minute. I don’t have an ETA for FTTH reaching my building in the UK, for example 😃

CPU speeds and compile times also vary, but I think the lower bound is much less worrying - even with a five-year-old thin laptop, one should be able to build the standard library in 10-20s.

I don’t like that we as a team are trying to land major features of the release at the absolute last possible moment and I would prefer not to have to make this decision. With that said, we know that the go command doesn’t get much testing until the RC anyway, so in this specific case it probably doesn’t make much practical difference.

Approved. If there are a lot of follow-ups we should take another look.

My suggestion here would be to define that go install x only ever installs binaries, never .a files. That would change this from being about cmd/dist and the Go distribution to being about cmd/go. The distributions would shrink as a side effect, and the meaning of go install would become clearer.

@jayconrod I think you are correct that with our current implementation we could call the macOS libraries directly without having to use cgo. The same is true on AIX and Solaris, for that matter.

1. Change the net package to not require a C compiler on macOS.

I wonder how viable this is?

I know very little about the guts of the net package. Reading https://pkg.go.dev/net#hdr-Name_Resolution, it looks like there are Go and cgo implementation for name resolution. When cgo is available, they’re both compiled in and selected dynamically (GODEBUG=netdns=go or GODEBUG=netdns=cgo can pick one at run-time). On macOS, the cgo version seems to be preferred.

But does the cgo version actually need to be written with cgo? On macOS at least, we link dynamically with /usr/lib/libSystem.B.dylib even with CGO_ENABLED=0, so we don’t need the external linker. CL 227037 leads me to believe it’s possible to call C code from Go with assembly trampolines. (This is mostly a thought experiment; I don’t want to re-implement net or awaken Cthulhu).

Based on the discussion above, this proposal seems like a likely accept. — rsc for the proposal review group

I think it’s pretty important that a Go binary installation work on a system with no C compiler installed. And at least in the past on macOS Go programs would only work if using the cgo version of the net package.

Does that imply that on macOS it already isn’t possible to build a working Go program that depends on net with CGO_ENABLED=0?

If so, that would imply that without a working C compiler it also isn’t possible to build programs that depend on both net and some third-party package whose source files vary based on the cgo build constraint: the cgo constraint is met if CGO_ENABLED=1 even if the C compiler is present. So that would force users to choose between two options:

1. Build with CGO_ENABLED=1 and get build errors due to attempting to compile third-party //go:build cgo files without a working C compiler.
2. Build with CGO_ENABLED=0, and get the appropriate source files for third-party packages but a non-working net package.

I think we could resolve that in one of three ways:

1. Change the net package to not require a C compiler on macOS.
2. <strike>Change cmd/go to build the net package on macOS using a C compiler even if CGO_ENABLED=0.</strike>
3. Declare that a C compiler is required in order to build Go programs that depend on net on macOS.

This proposal has been added to the active column of the proposals project and will now be reviewed at the weekly proposal review meetings. — rsc for the proposal review group

I think we can plausibly check in the relevant cgo-generated pieces for the few stdlib packages that need cgo. Then we wouldn’t need a special case for “install runtime/cgo.a but no other .a files”.

CGO_ENABLED=0 is effectively the same as cross-compiled, and same headaches: netdns’s go is simply inadequate, at least for use cases where DNS queries are routed based on domain name to different resolvers by the operating system. /etc/resolv.conf does not adequately describe the OS DNS routing behavior; there’s no good way for netdns to ever do this correctly.

I would agree with you a priori, but as I understand the current state of the world, the introduction of the build cache broke this case many releases back, with no complaints (or at least not enough that we’ve tried to fix it). The .a files that ship do not actually have the right cache keys embedded inside them to be used by essentially any out-of-the-box install of Go. Instead, they get recomputed (in the cache only, not in the install location) the first time you build something.

@rsc I was under this impression too until looking at #47215 last week. The C compiler version only became part of the cache key in #40042. So this is true for 1.17rc1, but not for 1.16 or lower. If neither CC nor CGO_CFLAGS are explicitly set, the go command will happily use the precompiled runtime/cgo.a, even if no C compiler is installed. We’re looking at a narrow rollback of this in CL 335409.

(@bcmills pointed out cases where the cache keys don’t match on macOS, but they do match on Linux, and that’s the platform I’m most worried about, since a lot of folks are building in Docker without installing a C compiler).