go: cmd/link: bad debug_frame section on macOS with buildmode=pie [Debugging]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (`go version`)?

go version go1.10 darwin/amd64

(also reproduces with go1.11 built from tip)

What operating system and processor architecture are you using (`go env`)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/vagrant/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/vagrant/go"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/rm/ywx69srx1kq_dtjz2ynznn000000gp/T/go-build680864595=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Build any program with go build -gcflags='-N -l' -buildmode pie

What did you expect to see?

debug_info has an entry for main.main that looks like this

 <1><40cda>: Abbrev Number: 2 (DW_TAG_subprogram)
    <40cdb>   DW_AT_name        : (indirect string, offset: 0x29120): main.main
    <40cdf>   DW_AT_low_pc      : 0x100090530
    <40ce7>   DW_AT_high_pc     : 0x100090636
    <40cef>   DW_AT_frame_base  : 1 byte block: 9c      (DW_OP_call_frame_cfa)
    <40cf1>   DW_AT_decl_file   : 0x1
    <40cf5>   DW_AT_external    : 1

but nothing in debug_frame covers anywhere close to the range 0x100090530…0x100090636, for example the first FDE is:

00000014 000000000000002c 00000000 FDE cie=00000000 pc=0000000000002600..0000000000002737
  DW_CFA_def_cfa_offset_sf: 8
  DW_CFA_advance_loc: 23 to 0000000000002617
  DW_CFA_def_cfa_offset_sf: 48
  DW_CFA_advance_loc1: 92 to 0000000000002673
  DW_CFA_def_cfa_offset_sf: 8
  DW_CFA_advance_loc: 1 to 0000000000002674
  DW_CFA_def_cfa_offset_sf: 48
  DW_CFA_advance_loc1: 124 to 00000000000026f0
  DW_CFA_def_cfa_offset_sf: 8
  DW_CFA_advance_loc: 1 to 00000000000026f1
  DW_CFA_def_cfa_offset_sf: 48
  DW_CFA_advance_loc: 60 to 000000000000272d
  DW_CFA_def_cfa_offset_sf: 8
  DW_CFA_advance_loc: 9 to 0000000000002736
  DW_CFA_nop

It seems to me that this is wrong but I could be missing something.

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 20 (14 by maintainers)

Commits related to this issue

proc: add workaround for debug_frame bug on macOS This adds a workaround for the bug described at: https://github.com/golang/go/issues/25841 Because dsymutil running on PIE does not adjust the addr... — committed to aarzilli/delve by aarzilli 3 years ago
proc: add workaround for debug_frame bug on macOS This adds a workaround for the bug described at: https://github.com/golang/go/issues/25841 Because dsymutil running on PIE does not adjust the addr... — committed to aarzilli/delve by aarzilli 3 years ago
proc: add workaround for debug_frame bug on macOS (#2374) This adds a workaround for the bug described at: https://github.com/golang/go/issues/25841 Because dsymutil running on PIE does not adj... — committed to go-delve/delve by aarzilli 3 years ago
proc: extend macOS workaround to amd64 Go change 064f34f (which exists in Go 1.19.2 and following) removed the pagezero_size option from linker calls (because it is deprecated). This expanded the pro... — committed to aarzilli/delve by aarzilli 2 years ago
proc: extend macOS workaround to amd64 Go change 064f34f (which exists in Go 1.19.2 and following) removed the pagezero_size option from linker calls (because it is deprecated). This expanded the pro... — committed to aarzilli/delve by aarzilli 2 years ago
proc: extend macOS workaround to amd64 Go change 064f34f (which exists in Go 1.19.2 and following) removed the pagezero_size option from linker calls (because it is deprecated). This expanded the pro... — committed to aarzilli/delve by aarzilli 2 years ago
proc: extend macOS workaround to amd64 Go change 064f34f (which exists in Go 1.19.2 and following) removed the pagezero_size option from linker calls (because it is deprecated). This expanded the pro... — committed to aarzilli/delve by aarzilli 2 years ago
proc: extend macOS workaround to amd64 (#3204) Go change 064f34f (which exists in Go 1.19.2 and following) removed the pagezero_size option from linker calls (because it is deprecated). This expand... — committed to go-delve/delve by aarzilli 2 years ago
[dsymutil] fix accidental 32bit truncation in patchFrameInfoForObject patchFrameInfoForObject accidentally truncates FDE addresses to the least significant 32bits, which causes the Go bug: https://gi... — committed to llvm/llvm-project by aarzilli 2 years ago

Most upvoted comments

Nop @zonghaishang i saw some people made workarounds that compiled the plugin into the main binary. They made a script that, just for debugging, automatically rearanged the project to compile into a single binary and after compilation restore the initial state. Its a major hack and afaik does not work generically. Yet as i understand the issue is dysmutil and only apple can fix dysmutil.

However i thought the workaround made things better, yet had no time to test it since we ditched plugins due to this issue for now all together.

Just because i assume @jeremyfaller has also other things on his desk ill leave a ping for him here if he heared back 😃 BTW is the bug report to apple public and can be viewed or commented ?

I solved the plug-in debugging problem with container technology.

build plugin in linux docker: https://github.com/zonghaishang/wasm-sdk/blob/go-plugin/go-plugin/Makefile#L27
build dlv debug program: https://github.com/zonghaishang/wasm-sdk/blob/go-plugin/go-plugin/build/sidecar/image/Dockerfile
start debug server: https://github.com/zonghaishang/wasm-sdk/blob/go-plugin/go-plugin/etc/script/run.sh#L18

The screenshot correctly identifies the plug-in information and can also be executed inside the plug-in in a single step(step into 😃 .

zonghaishang on Nov 14, 2021

Apologies, for the latency. I finally got a chance to take a look at this. As @heschik suspected, it’s dsymutil. [un]Surprisingly, dsymutil can’t even verify its own output while generating the debug information.

I’ve filed a bug with apple (7568255), and will follow up here with any responses.

I have little experience debugging dsymutil myself, but I know a few people internally (@google) with some experience. I’ll reach out to them next week, and see if they have any insights as well. I’ll update this bug as we hear from Apple, or I gain insight from the local knowledgable people.

jeremyfaller on Feb 7, 2020

Thanks for fixing this @aarzilli !

cherrymui on Dec 20, 2022

Any news on this one? Just hit it again and its highly frustrating and confusing when you can’t trust your debugger anymore. Since the delve workaround has been merged I assume I shouldn’t even be seeing this with current go and vscode?

andig on Apr 2, 2021

Well, I did follow up internally, and hit lots of roadblocks. No one had seen anything, and I just put it on hold mentally until I head from Apple. I never heard, and completely forgot about it. I’ll admit it fell off my radar after that though. I’ll ping the bug again with Apple, and see if I can pursue some extra channels we have.

jeremyfaller on Mar 8, 2021

@jeremyfaller did apple ever do anything about this? The problem has just become more pressing because on darwin/arm64 all externally linked executables are PIE (612b1194475a23760ec502b48a93fea7237f3ae6) so on that architecture anything that uses cgo will encounter this problem, see: https://github.com/go-delve/delve/issues/2346.

aarzilli on Mar 6, 2021

I’ll take a look in the next couple of days. Whatever happens though, this won’t make 1.14, so don’t expect a mainline fix for quite some time. 😃

jeremyfaller on Jan 2, 2020

@heschik : it seems like this issue is preventing debugging of go plugins on Mac OS X. My project is using go plugins and most of our engineers have Macs, so it’s inconvenient to not be able to debug the code.

As I have summarized in a comment in the related delve issue (https://github.com/go-delve/delve/issues/1628), @aarzilli has stated that this particular issue is also a blocker for delve supporting go plugins on Mac. I know very little about how delve, DWARF, etc. work, so I’m not sure if this is the only issue still blocking that from working. However, based on @aarzilli 's statements, I would say the earlier claim that this issue is unimportant is no longer true.

erikdw on Dec 30, 2019