go: race: not working with Alpine based image

@djui, there is no update. Nobody on the Go team is working on this, and nobody elsewhere in the community has posted anything here about it. It’s all yours if you want to work on it.

In case it helps anyone else, I rebased @neelance’s compiler-rt changes (https://github.com/golang/go/issues/14481#issuecomment-281972886) on top of compiler-rt@fe2c72c5 (as used by Go 1.13) and have been able to run -race using:

FROM golang:1.13-alpine3.10 as golang

# Make go test -race work on alpine by building (patched) sanitizer manually
# as it is not built by default
# Ref: https://github.com/golang/go/issues/14481#issuecomment-281972886
# SHA: https://github.com/golang/go/blob/go1.13/src/runtime/race/README
COPY 0001-hack-to-make-Go-s-race-flag-work-on-Alpine.patch /race.patch
RUN cd / \
    && apk add --no-cache --virtual .build-deps g++ git \
    && mkdir -p compiler-rt \
    && git clone https://llvm.org/git/compiler-rt.git \
    && cd compiler-rt \
    && git reset --hard fe2c72c59aa7f4afa45e3f65a5d16a374b6cce26 \
    && patch -p1 -i /race.patch \
    && cd lib/tsan/go \
    && ./buildgo.sh 2>/dev/null \
    && cp -v race_linux_amd64.syso /usr/local/go/src/runtime/race/ \
    && rm -rf /compiler-rt /race.patch \
    && apk del .build-deps

0001-hack-to-make-Go-s-race-flag-work-on-Alpine.patch.gz

Humm… how did we get the previous syso files for ppc?..

Something changed since last fall. It wasn’t that code itself, maybe someone turned ASLR on for ppc64le in some config somewhere?

Every once in a while I am thinking about writing a new tsan runtime fork in Go for Go. There is also a next gen algorithm that I think could reduce memory consumption few times and make it faster and remove restriction on number of goroutines and all the nice things. But I can never fully convince myself that it’s the right thing to do… or maybe just find time.

It would be nice. Pulling new tsan always seems to be a headache. But rewriting tsan in Go is probably a lot of work, many times the aforementioned headache. Probably not worth it. Unless you were locked in a room with nothing else to do for a month or so…

The fix is submitted to llvm. We now have 2 reasons to update race runtime: this and the chan synchronization change. @randall77

@djui why should the issue be closed if it’s still an issue, even if it doesn’t have anyone currently working on it? By that logic, most of the current issues should be closed, no?

@qdm12 this was merged in master, so it will be part of the upcoming Go 1.15 release due in July, unless it gets reverted.

Only important bug fixes are generally backported to stable Go releases. See https://github.com/golang/go/wiki/MinorReleases.

In particular, while this change is awesome and I’d also like to use it today, the change is probably too invasive and risky to backport to a release like 1.14.3. The first 1.15 beta should be out in just six weeks, so you could try the docker images that come out with that release, too.

Haha thanks a lot! I was actually adapting the patch right now. But you probably saved me minutes! 👍 🎈

EDIT: A Dockerfile example thanks to dnwe’s patch and Dockerfile, which works 🎉

ARG ALPINE_VERSION=3.11
ARG GO_VERSION=1.13

FROM golang:${GO_VERSION}-alpine${ALPINE_VERSION} AS race
WORKDIR /tmp/race
RUN apk --update -q --progress --no-cache add git g++
RUN git clone --single-branch https://github.com/llvm-mirror/compiler-rt . && \
    git reset --hard 69445f095c22aac2388f939bedebf224a6efcdaf
RUN wget -q https://github.com/golang/go/files/4114545/0001-upstream-master-69445f095-hack-to-make-Go-s-race-flag-work-on-Alpine.patch.gz -O patch.gz && \
   gunzip patch.gz && \
   patch -p1 -i patch
WORKDIR /tmp/race/lib/tsan/go
RUN sed -e 's,-Wno-unknown-warning-option,-Wno-error=deprecated,' -i buildgo.sh
RUN ./buildgo.sh

FROM golang:${GO_VERSION}-alpine${ALPINE_VERSION}
COPY --from=race /tmp/race/lib/tsan/go/race_linux_amd64.syso /usr/local/go/src/runtime/race/race_linux_amd64.syso
# ... more instructions
RUN go test -race ./...

@dvyukov has made an enormous number of valuable contributions to Go, but he is not a member of the Go team. And although I haven’t spoken to him recently my understanding is that these days he is focused largely on syzkaller (https://github.com/google/syzkaller).

That said, sure, we can take a look at a change to the compiler-rt project. But we can’t approve it. We certainly can and will do the packaging changes on the Go side when the compiler-rt project is updated.

I’m not trying to duck out of the appropriate work here. I’m saying that you shouldn’t sit back and wait for the Go team to fix this. That is fairly unlikely to happen.

@qdm12 yeah so just to clarify for everyone watching, the Dockerfile I pasted above is compiling race from fe2c72c5 which is what the Go 1.13 release is using (as per runtime/race/README). The Dockerfile you’ve put there is patching race on current master (69445f09) which is good for exercising the patch that could potentially be sent upstream, but isn’t necessarily what I’d recommend people run their CI with.

If you use Docker, I have an example with a multi stage build here where the fix mentioned above is built and then overwrites the existing file in the final image.

I have managed to build a race_linux_amd64.syso that works on Alpine. Here are the necessary changes: https://github.com/neelance/compiler-rt/commit/32aa655a999d33a4034ec4aea00ed3df1f9e77df I haven’t verified that it still works on other Linux platforms.

To build on Alpine:

1. Clone https://github.com/neelance/compiler-rt/
2. Go into /compiler-rt/lib/tsan/go/
3. ./buildgo.sh (The test step crashes, but the library works anyways. My guess is that the test does not use a proper Go environment.)
4. Copy race_linux_amd64.syso to $GOPATH/src/runtime/race/

I’m not sure how to continue from here. Any suggestions on getting that upstream?

Yes, we are using it successfully with the (rolling) 1.15-alpine tag.

llvm has moved to gihub: https://github.com/llvm/llvm-project

With a small tweak to buildgo.sh to remove the (no longer valid) -Wno-unknown-warning-option from the compile flags and to add a -Wno-error=deprecated to permit deprecated warnings not to fail the build, this Dockerfile is sufficient to build a Go 1.13 on alpine 3.11 docker image (which the recent question was about):

FROM golang:1.13-alpine3.11 as golang

# Make go test -race work on alpine by building (patched) sanitizer manually
# as it is not built by default
# Ref: https://github.com/golang/go/issues/14481#issuecomment-281972886
# SHA: https://github.com/golang/go/blob/go1.13/src/runtime/race/README
ADD https://github.com/golang/go/files/3615484/0001-hack-to-make-Go-s-race-flag-work-on-Alpine.patch.gz /race.patch
RUN cd / \
    && apk add --no-cache --virtual .build-deps g++ git \
    && mkdir -p compiler-rt \
    && git clone https://git.llvm.org/git/compiler-rt.git \
    && cd compiler-rt \
    && git reset --hard fe2c72c59aa7f4afa45e3f65a5d16a374b6cce26 \
    && gzip -dc /race.patch | patch -p1 \
    && cd lib/tsan/go \
    && sed -e 's,-Wno-unknown-warning-option,-Wno-error=deprecated,' -i ./buildgo.sh \
    && ./buildgo.sh \
    && cp -v race_linux_amd64.syso /usr/local/go/src/runtime/race/ \
    && rm -rf /compiler-rt /race.patch \
    && apk del .build-deps

I can confirm it works on golang:1.15-alpine, although not on golang:1.15-alpine3.12 for example.

Doesn’t work for me.

But I added

RUN apk add --update --no-cache build-base

to make it work. Seems similar to this gist: https://gist.github.com/miry/fece267c7faba904c360

If you also need git you can use instead:

RUN apk add --update --no-cache alpine-sdk

Here’s a version of the patch rebased against current upstream master (llvm-mirror/compiler-rt@69445f095) and with one additional !SANITIZER_GO added in sanitizer_common/sanitizer_linux_libcdep.cpp to preprocess out some new code there that relied on GetEnviron

0001-upstream-master-69445f095-hack-to-make-Go-s-race-flag-work-on-Alpine.patch.gz

@qdm12 OK, so whoever wrote that patch needs to update it. And maybe try to submit it to the compiler-rt project so that people don’t have to patch it themselves.

This issue is filed against the Go project but I don’t understand what the Go team can do to fix it. It seems to involve code that we do not control.

So maybe we could open a new ticket, link this ticket, with e.g. race: Make compiling and running race detector work with Musl. wdyt?

I can change the title of this issue but I don’t think that will speed up the fix for it. Unfortunately, I don’t have the time or the knowledge to work on this and it seems that nobody else from all the referenced issues has any plans on fixing this either. However, it seems that people find this issue and it’s easy for them to link to it so may I suggest to leave this like they are until someone figures how to fix this? Thank you.

@dlsniper My reasoning is focusing on the category:“Is it a bug or a feature” rather than:“If feature, is a requested feature implemented by now.” To me, the issue was opened in spirit of a bug, imho, and it became during the conversation that no promises or constraints are violated, thus more of a feature (getting down to: “Should any C stdlib be supported by Go?” or “Is Musl a supported ‘arch’?”). I may misclassify this and indeed any of these should be supported, therefore the question for feedback.

So maybe we could open a new ticket, link this ticket, with e.g. race: Make compiling and running race detector work with Musl. wdyt?

@mvdan Understandably, same here.

Hi @bradfitz I think it’s ok to close this issue. It’s not fair to leave it open as an issue if it’s clear by now this does not work by design without further work.

As you said, if anyone wants to pick it up, great!

For 👎 'ers: Would you agree?

hi, all,

just wondering. does anyone have a complete working example of a golang program working with -race ideally not that much different from the official Alpine-based build.

out of interest, are the flags not tested? i would imagine that this sort of issue would be picked up at compile time, unless the Alpine-based is configured to to include it… no idea, just guessing.

anywho, working sample would be much appreciated.

I can’t think of any simple fix. A complex fix would be to remove all dependencies on libc from race runtime (there is an open issue for that). alpinelinux wiki suggests that there are some ways to run glibc-based programs on alpine: http://wiki.alpinelinux.org/wiki/Running_glibc_programs Don’t know whether it will help for race detector or not.