go: internal/poll: CopyFileRange returns EPERM on CircleCI Docker Host running 4.10.0-40-generic

What version of Go are you using (go version)?

$ go version
go version go1.15 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

Docker Host running Ubuntu 18.04 on Linux Kernel 4.10.0-40-generic

This is the Go env of the container trying to build:

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/circleci/.cache/go-build"
GOENV="/home/circleci/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/circleci/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/circleci/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/circleci/project/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build612039276=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I tried to build a Go binary (with Go 1.15) within a Docker container (from gimg/go:1.15) running on a CircleCI remote Docker engine (17.11.0-ce).

What did you expect to see?

Docker image should have been built correctly, and my binary compiled successfully with Go 1.15.

What did you see instead?

An error like this for every binary in my project:

/usr/local/go/pkg/tool/linux_amd64/link: cannot write /tmp/go-link-518048351/000003.o: write /tmp/go-link-518048351/000003.o: copy_file_range: operation not permitted
/usr/local/go/pkg/tool/linux_amd64/link: cannot write /tmp/go-link-518048351/000035.o: write /tmp/go-link-518048351/000035.o: copy_file_range: operation not permitted
The command '/bin/bash -exo pipefail -c go install -mod=vendor -v ./...' returned a non-zero code: 2

Details from my investigation:

I build my binaries on CircleCI in a Docker container. CircleCI allows the user to pin a specific Docker engine, but they default to a version (17.11.0-ce) that runs on a Docker Host powered by Linux Kernel 4.10.0-40-generic.

Despite the above kernel does support the directive copy_file_range, for some restrictions possibly setup by the vendor, the Go linker gets this error:

usr/local/go/pkg/tool/linux_amd64/link: cannot write /tmp/go-link-518048351/000003.o: write /tmp/go-link-518048351/000003.o: copy_file_range: operation not permitted

This seems to correspond to the EPERM error.

I’ve filed an issue report with the vendor asking why this is happening and I’m waiting for a confirmation: is this a scenario that we would like to take into account, and fallback as well to the other approach? I assume the EPERM can be returned also in the case of the user not having access to a file, therefore between the two branches of this switch I guess this scenario would be more similar to the second case where we might want a one-off fallback.

I spawn up a VirtualBox environment and tried to build the same code that breaks on CircleCI on three different kernels and all the builds were successful, proving that the issue I’ve encountered is probably related to some limitation/sandboxing artificially imposed by the vendor:

• 4.15.0-45-generic <- the officially LTS supported by Ubuntu 16.04
• 4.10.0-40-generic <- the one used by CircleCI
• 4.4.232-0404232-generic<- the latest prior to the introduction of copy_file_range

Unfortunately I can’t really reproduce the exact setup as CircleCI has its own internal AMIs/images that are not accessible to me, but I asked for the Kernel details and they told me the Docker Host that breaks my linker/builds runs on 4.10.0-40-generic.

The above has been posted as a comment on https://github.com/golang/go/issues/40731 but for easier traceability I’ve opened a dedicated issue.