go: cmd/go: dlopen of c-shared binary on MacOS fails with "segment filesize exceeds vmsize"

What version of Go are you using (go version)?

$ go version
go version go1.17.6 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/svanders/Library/Caches/go-build"
GOENV="/Users/svanders/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/svanders/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/svanders/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/svanders/sdk/go1.17.6"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/svanders/sdk/go1.17.6/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.17.6"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/svanders/nethsm/pkcs11/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/t8/5_d5bl793710gx_c4fh3_k6r0000gn/T/go-build3576067778=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Build any binary with -buildmode=c-shared and try to load as it PKCS#11 module with OpenSSH like ssh -I ./c-shared.so ...

What did you expect to see?

For arbitrary c-shared binaries it should fail with

dlsym(C_GetFunctionList) failed: dlsym(0x853ca80, C_GetFunctionList): symbol not found

and in case of a proper PKCS11 module it should not fail at all.

What did you see instead?

$ ssh -I ./pkcs11.so ...
dlopen ./pkcs11.so failed: dlopen(./pkcs11.so, 0x0002): in '.../pkcs11.so' segment filesize exceeds vmsize

However, loading the same library with pkcs11-tool (installed with brew) works. So it might be a bug in OpenSSH, but since it can load other pkcs11 modules, I guess that is unlikely.

Environment

MacOS 11.6.2

$ clang --version 
Apple clang version 13.0.0 (clang-1300.0.29.30)
Target: x86_64-apple-darwin20.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin