confluent-kafka-dotnet: librdkafka fails to load on 2.1-aspnetcore-runtime-alpine

Description

using container 2.1-aspnetcore-runtime works, 2.1-aspnetcore-runtime-alpine fails

Unhandled Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. —> System.DllNotFoundException: Failed to load the librdkafka native library. submission-term-dd1.1.vclyzsukdl49@tcwsddu1 at Confluent.Kafka.Impl.LibRdKafka.Initialize(String userSpecifiedPath) submission-term-dd1.1.vclyzsukdl49@tcwsddu1 at Confluent.Kafka.Producer…ctor(IEnumerable1 config, Boolean manualPoll, Boolean disableDeliveryReports) submission-term-dd1.1.vclyzsukdl49@tcwsddu1 at Confluent.Kafka.Producer2…ctor(IEnumerable1 config, ISerializer1 keySerializer, ISerializer`1 valueSerializer, Boolean manualPoll, Boolean disableDeliveryReports)

How to reproduce

Create a ASP.Net Core 2.1 project with Confluent.Kafka 0.11.5. Include the standard producer. Build the project and deploy in Docker using the official Microsoft Alpine image.

Using the official (stretch) container works fine.

Checklist

Please provide the following information:

  • [x ] Confluent.Kafka nuget version: 0.11.5
  • Apache Kafka version:
  • Client configuration:
  • [x ] Operating system: Alpine Linux container
  • Provide logs (with “debug” : “…” as necessary in configuration)
  • Provide broker log excerpts
  • Critical issue

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 3
  • Comments: 36 (12 by maintainers)

Most upvoted comments

The official Alpine build is a good start but what I need is support in the NuGet package as @mhowlett mentioned. We’re developing on Macs and PCs, and deploying to Docker.

The Alpine image is significantly smaller than the default image (stretch) so it is a big win if we can use it.

thanks @darrenfurr . just wanted to note that we’re tracking this internally (off-the-shelf support for alpine that is). It will be done after 1.0-RC1. i’m not sure if it’ll make into 1.0 (likely not), but likely 1.0.1.

Does this work for others? Using alpine image still does not work for me.

definitely the microsoft images since this is what 99% of people will be using and the reason there’s so much demand for this.

This works for me with the 1.0.0-beta2 (admittidly on .Net Core 2.2):

FROM microsoft/dotnet:2.2-runtime-deps-alpine
# Add the kafka native lib
RUN apk update && \
    apk add librdkafka librdkafka-dev

Create your own (Customize) Alpine image and run on top of that image. In that image you may install librdkafka and other third-party packages.

FROM alpine:3.7 RUN apk add librdkafka \ librdkafka-dev \ –no-cache --repository https://pkgs.alpinelinux.org/package --allow-untrusted #Install .NET Core Runtime RUN apk add --no-cache \ ca-certificates \ \ # .NET Core dependencies krb5-libs \ libgcc \ libintl \ libssl1.0 \ libstdc++ \ tzdata \ userspace-rcu \ zlib \ && apk -X https://dl-cdn.alpinelinux.org/alpine/edge/main add --no-cache \ lttng-ust

#Configure Kestrel web server to bind to port 80 when present ENV ASPNETCORE_URLS=http://+:80 \ # Enable detection of running in a container DOTNET_RUNNING_IN_CONTAINER=true \ # Set the invariant mode since icu_libs isn’t included (see https://github.com/dotnet/announcements/issues/20) DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=true

Still on 0.11.5…

Hhhmm, that’s where I’m at now.

Dockerfile

FROM microsoft/dotnet:2.1-aspnetcore-runtime-alpine
RUN apk add librdkafka
WORKDIR /app
COPY --from=build-env /app/myApp/out .
ENTRYPOINT ["dotnet", "myApp.dll"]

Code:

Console.WriteLine("Using custom path for librdkafka.");
Confluent.Kafka.Library.Load("/usr/lib/librdkafka.so.1");

Getting an error, but not what I expected:

Using custom path for librdkafka.
Unhandled Exception: System.DllNotFoundException: Failed to load the librdkafka native library.
myApp.1.scz73zvncfsy@tcwsddu1 at Confluent.Kafka.Impl.LibRdKafka.Initialize(String userSpecifiedPath)

I expected it to work or to log

$"Failed to load librdkafka at location '{userSpecifiedPath}'. dlerror: '{PosixNative.LastError}'."

@edenhill @mhowlett is this a valid work-around until you add native support?