grpc: C#: Running in Alpine Linux docker container fails with "Grpc.Core.Internal.UnmanagedLibrary Attempting to load native library "/app/libgrpc_csharp_ext.x64.so"

This issue still exists on .NET 5 alpine container.

Image: mcr.microsoft.com/dotnet/aspnet:5.0-alpine

crit: Microsoft.AspNetCore.Hosting.Diagnostics[6]

Application startup exception

System.IO.IOException: Error loading native library "/app/runtimes/linux/native/libgrpc_csharp_ext.x64.so". Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /app/runtimes/linux/native/libgrpc_csharp_ext.x64.so)

at Grpc.Core.Internal.UnmanagedLibrary..ctor(String[] libraryPathAlternatives)

at Grpc.Core.Internal.NativeExtension.LoadUnmanagedLibrary()

at Grpc.Core.Internal.NativeExtension.LoadNativeMethods()

at Grpc.Core.Internal.NativeExtension..ctor()

at Grpc.Core.Internal.NativeExtension.Get()

at Grpc.Core.Internal.NativeMethods.Get()

at Grpc.Core.GrpcEnvironment.GrpcNativeInit()

at Grpc.Core.GrpcEnvironment..ctor()

at Grpc.Core.GrpcEnvironment.AddRef()

at Grpc.Core.Channel..ctor(String target, ChannelCredentials credentials, IEnumerable`1 options)

Our best current workaround (in collaboration with @Belovolov)

FROM alpine:3.13 AS grpc-build
WORKDIR /opt
RUN apk add --update alpine-sdk autoconf libtool linux-headers cmake git && \ 
    \
    git clone -b v1.36.4 https://github.com/grpc/grpc --depth 1 --shallow-submodules && \
    cd grpc && git submodule update --init --depth 1 && \
    \
    mkdir -p cmake/build && cd cmake/build && \
    \
    cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo \
      -DgRPC_BACKWARDS_COMPATIBILITY_MODE=ON \
      -DgRPC_BUILD_TESTS=OFF \
      ../.. && \
    \
    make grpc_csharp_ext -j4 && \
    \
    mkdir -p /out && cp libgrpc_csharp_ext.* /out

Then in another stage:

COPY --from=grpc-build /out/libgrpc_csharp_ext.so /app/libgrpc_csharp_ext.x64.so

No libc6-compat or glibc necessary.

Downgrading lib6-compat to 1.19 fixed this issue for me. Not sure what is impact though.

RUN echo 'http://dl-cdn.alpinelinux.org/alpine/v3.8/main' >> /etc/apk/repositories && 
    apk update --no-cache &&
    apk add --no-cache bash libc6-compat=1.1.19-r11

BUMP!

This issue has been open for more than a year now. The only fix opens up critical security vulnerabilities. Is there a chance to get a proper fix any time soon?

There is a little issue with the workaround

RUN echo 'http://dl-cdn.alpinelinux.org/alpine/v3.8/main' >> /etc/apk/repositories && 
    apk update --no-cache &&
    apk add --no-cache bash libc6-compat=1.1.19-r11

Basically, it makes the container exposed to CVE-2019-14697 with CVSS V3 9.8 CRITICAL, and CVSS V2 7.5 HIGH scores.

@jtattermusch Will the Google Cloud dotnet libraries be updated to use grpc-dotnet?

Yes, in fact you should be able to use the client libraries with grpc-dotnet even today (at the cost at needing a bit of glue code), by using https://www.nuget.org/packages/Google.Api.Gax.Grpc.GrpcNetClient/

A much better integration of client libraries and grpc-dotnet is planned and should land over the next few months (and on supported platforms, grpc-dotnet will actually be the default option).

I happened to bump into this issue, and since there was no way around it, had to solve it. The solution is a slight overkill, but it worked. What I did was basically build the entire grpc library right on alpine. Its a long process but quite simple:

• clone the repo+tag you want use - for me it was https://github.com/grpc/grpc/releases/tag/v1.33.2
• build using cmake as described here https://github.com/grpc/grpc/blob/master/BUILDING.md. use cmake and -DBUILD_SHARED_LIBS=OFF (default)
• build artifacts will contain libgrpc_csharp_ext.x64.so
• replace the above file with one found on your alpine system (just copy)
• you should be good to go!

note: The pre-requisites should take care of it, but you may also need to install glibc for alpine. docker syntax:

# install glibc for grpc to work on alpine
ENV GLIBC_REPO=https://github.com/sgerrand/alpine-pkg-glibc
ENV GLIBC_VERSION=2.32-r0

RUN set -ex && \
    apk --update add libstdc++ curl ca-certificates && \
    for pkg in glibc-${GLIBC_VERSION} glibc-bin-${GLIBC_VERSION}; \
        do curl -sSL ${GLIBC_REPO}/releases/download/${GLIBC_VERSION}/${pkg}.apk -o /tmp/${pkg}.apk; done && \
    apk add --allow-untrusted /tmp/*.apk && \
    rm -v /tmp/*.apk && \
    /usr/glibc-compat/sbin/ldconfig /lib /usr/glibc-compat/lib

This continues to be an issue in Alpine; any chance of a long term fix for this?

Is there any chance this issue got resolved/fixed?

As workaround, I’m leveraging this https://github.com/grpc/grpc/issues/21446#issuecomment-698396812, and will soon test this other approach: https://github.com/grpc/grpc/issues/21446#issuecomment-807990690. But it would be ideal to get this fixed upstream. Thanks!

Here’s another workaround: try creating a link from /lib64 to /lib before installing libc6-compat, Alpine 3.8/musl 1.1.19 did this while Alpine 3.9/musl 1.1.20 onward have separate /lib and /lib64 with ld-linux-x86-64.so.2 in /lib64 and the rest in /lib.

This needs a real fix as mucking around with system directory structures would probably cause some other weird behaviors further down the line.

Got it! Big thank-you @elibendavid! I dug around our development/deployment setup, and found that since we build/deploy a self-contained app, the updated libgrpc_csharp_ext.so.14.0.0 had to be renamed and overwrite the file that gets automatically pulled down in the Docker publish folder.

Hopefully the issue gets an official fix in a future update as it’s a bit clumsy, but having this solution means Alpine 3.10+ should work fine with Grpc now!

I’ve been using mcr.microsoft.com/dotnet/aspnet:6.0.1-bullseye-slim as suggested by @rmclemente but now found an even slimmer alternative: distroless CBL-Mariner.

The docker tag is not exposed on the official dockerhub page but is listed under all available tags.

FROM mcr.microsoft.com/dotnet/aspnet:6.0.1-cbl-mariner1.0-distroless-amd64 as runtime

Thanks @jskeet, gotcha. JFYI: I was tagging you since you asked earlier on that thread about a minimal example to reproduce the issue, so was just sharing to the maintainers of this repo too in case that could help. Hope we’ll get an answer/ETA from them. Thanks!

@obawin This is highly dependant on your installation of .net runtime as well as your app installation. In my case it was - copy it to the exact location reported by the original error.

Looks like your originals are here (sym-links + bin):

/cmake/build/libgrpc_csharp_ext.so.14.0.0
./cmake/build/libgrpc_csharp_ext.so.14
./cmake/build/libgrpc_csharp_ext.so

Assuming you built on x64, you will want to rename the above file ( go for the main one not sym links ) to libgrpc_csharp_ext.x64.so - this is your source

Copy it to the location reported by the original error