go: proposal: cmd/link: by default, do not write out DWARF

I am not the least bit happy that I’m expected to debug something different than what I run, and that a recompilation is expected. I don’t run the debugger much because it’s not easy to run the debugger much, and I’m trying to fix that (“not too many people swimming across the river, why do we need a bridge?”). I do think it is onerous; and I view it as one of the things that Java got relatively right.

There’s the additional problem that the -N -l flag combo (treating it as a single flag) is not tested nearly as well as the default; if we honestly support it, we’ll need to run all.bash in that mode too, and anyone with widely-used libraries will want to also test using that mode, just in case. Doubling the testing load is not going to speed up developer workflow. And of course, we must document the flags, and people learning to use the compiler will need to learn about them if they plan to create binaries that delve or gdb can debug.

The main point of adding all this debugging information to Go binaries was these two things; to remove speed bumps from Go debugging, and to make progress on the do-we/don’t-we support problem for these flags.

Debugging information is also not just for debuggers. Years ago a friend of mine showed me what he called the “killer app for Python”, which was just really good backtraces that did a nice job of reporting the values of all the local variables in the stack trace. Their app would wrap all that up in a neat little bundle, ask the customer if they were willing to send it in as a bug report (with certain assurances about information confidentiality that probably wouldn’t fly now), and his next interaction with said customer was not to badger them for more information, but instead “here’s the dot-release that fixes your problem, download it at your convenience”. “And the customer thinks I’m a wizard!” We could/should do this for Go, at least as an option.

There are some alternatives, for instance, with reproducible builds we can generate debugging information on demand – but when we mention that to nearby colleagues working on other languages and platforms, their reaction is “yeah, right”. Their best practice is to build with debugging, make a copy (of either the debugging info alone, or the entire binary) and strip for shipping.

Is binary size causing some other problem that can be discussed publicly? Are we filling up disks? (not mine – with 10 copies of the Go repo in various states of build, plus all the go-gotten binaries for tools, plus testing and benchmarking, about 2% of my “disk”, counting both bin and pkg) . Is there some quota that we’re exceeding? Are builds taking too long, and if so, by how much, and have we explored other ways of saving that time that don’t reduce functionality? (the linker is due for a rewrite, the register allocator is definitely a time hog, Josh has done some interesting work on moving code from the front end to SSA that seems to save a little time, and has the potential to expose more of the compiler to multithreading).

Almost all my production server code has the prof built in.

As mentioned in the very first post of this issue, profiling is not affected when DWARF information is not present:

Even without DWARF at all, stack traces cased by panic would be unchanged and would contain symbols and line numbers. Pprof, objdump, and many other tools would still work.

Would building out two separate binaries one with debugging information and one without (Using go build or maybe a different command go release or some other command ) would maybe help with people that don’t like the idea of having there final binaries without DWARF or having to set a environment variable right off the bat? Don’t know if this is in scope / on-topic for this proposal, just wanted to throw out a thought.

“Debugging information” does not mean only “DWARF”. As I said in my post, we can improve self-debugging support independently of improving debugger support, which means DWARF.

@dr2chase can always get the DWARF he wants all the time by just setting the flag once in his profile. I am not proposing to turn DWARF off, I am proposing to turn it off by default as few programmers need the support it provides every day. You are not expected to run without DWARF, you are expected just to set the flag once, for yourself, if you want DWARF on.

Binary size matters. We copy binaries around, we push binaries over networks, we put them in containers, we store them in the cloud, we fill our bin directories with many programs; in those operations, the accumulations are significant.

As to your performance problems, my experiments with building the go command (cmd/go) show that generating DWARF for a significant program is about 40% of the build CPU time. That is a major chunk.

% rm go
rm: go: No such file or directory
bismarck=% time go build

real	0m1.205s
user	0m1.235s
sys	0m0.238s
% rm go
% time go build -ldflags=-w

real	0m0.730s
user	0m0.703s
sys	0m0.223s
% hoc -e 730/1235
0.5910931174089069
% 

This is not as radical as it sounds.

This is pretty much what every other compiler does, I wouldn’t call it radical.

I propose we change the Go build environment to suppress DWARF by default, saving lots of CPU time and disk space. Instead, a global shell environment variable, say GODWARF=1, could be set to cause it to be written out

My preference would be for a go build flag. For example -g.

Yes please. As I understand it delve suggest compiling with -N -l already so adding another flag to the “I want to build for the debugger” step shouldn’t be a large burden.

Now that DWARF is compressed on Mac (and I assume also on Windows), it seems like we’re back to a 20% or so space overhead, and the main issue is now link latency.

Even on Linux I see DWARF aggregation approximately doubling the time spent in the linker (and it’s not like the non-DWARF parts of the linker are terribly fast). So this is not a Mac-specific problem. Of course, compression might be what’s taking all the time.

Putting this on hold pending better understanding of where all the CPU time is going and that there aren’t significant optimizations remaining that might make it faster. If 2X really is the cost of DWARF, then we should resume this conversation about whether it makes sense to pay that cost speculatively all the time when the probability of needing it is near zero.

The DWARF tables are present only for the debuggers.

Profilers read DWARF as well, for what it is worth. It’s nice to be able to collect a detailed profile without having to recompile. On the other hand (as with debuggers) very few people run profilers.

How about a tweak – continue to emit DWARF, but when linking “user” programs, don’t include the DWARF from the standard packages (people who use debuggers are rare, but people who debug the Go runtime are even fewer).

@davecheney, I agree with you that “recompile for debugging with Delve/gdb/lldb/etc” is not too onerous and well-established. I just want to point out for the record that the recent work is supposed to make it the case that you don’t need -N -l (and the consequent performance loss) just to run those debuggers. While I don’t mind training people that if you’re going to use Delve you might need a different build, it would be good if we could get away from “if you’re going to use Delve you have to give up significant performance”, and I believe Go 1.11 is a decent step toward that.