confluent-kafka-dotnet: Exception "Failed to load the librdkafka native library" when building consumer using 1.0.0-beta3

Description

Getting the following exception when building a consumer using 1.0.0-beta3:

Unhandled Exception: System.DllNotFoundException: Failed to load the librdkafka native library. at Confluent.Kafka.Impl.Librdkafka.Initialize(String userSpecifiedPath) at Confluent.Kafka.Consumer2..ctor(ConsumerBuilder2 builder) at Confluent.Kafka.ConsumerBuilder2.Build() at Confluent.Kafka.Examples.ConsumerExample.Program.Run_Consume(String brokerList, List1 topics, CancellationToken cancellationToken) in /src/confluent-kafka-dotnet/examples/Consumer/Program.cs:line 60 at Confluent.Kafka.Examples.ConsumerExample.Program.Main(String[] args) in /src/confluent-kafka-dotnet/examples/Consumer/Program.cs:line 192 Abort trap: 6

How to reproduce

Run the Consumer example console app using 1.0.0-beta3. (https://github.com/confluentinc/confluent-kafka-dotnet/tree/master/examples/Consumer)

If I run the Consumer example app with from the 1.0.0-beta2 release, it works without error.

Checklist

Please provide the following information:

Confluent.Kafka nuget version: 1.0.0-beta3
Apache Kafka version:
Client configuration:
Operating system: OSX 10.13.6 (17G65)
Provide logs (with “debug” : “…” as necessary in configuration)
Provide broker log excerpts
Critical issue

About this issue

Original URL
State: open
Created 5 years ago
Comments: 23 (5 by maintainers)

Most upvoted comments

Use this in dockerfile , it was installed and worked fine.

FROM mcr.microsoft.com/dotnet/core/runtime:3.0 AS runtime
RUN apt-get update -y
RUN apt-get install -y librdkafka-dev

If you have any errors i will share my full dockerfile

+10

ratri-git on Dec 2, 2019

The workaround for now is brew install zstd on osx

edenhill on Feb 13, 2019

@ritasker

Looks like you’re on an Arch Linux derivative. With libsasl installed, the library is actually libsasl2.so.3 while the prebuilt librdkafka.so is attempting to link to libsasl2.so.2.

I manually built librdkafka myself so it would link against the right libsasl, and then replaced ~/.nuget/packages/librdkafka.redist/1.0.0-rc7/runtimes/linux-x64/native/librdkafka.so with the version I manually built.

That should fix it.

vertexmachina on Mar 6, 2019

We also had to upgrade OpenSSL on osx by brew upgrade openssl to resolve the issue

brightermatter on Feb 15, 2019

It also happens on mcr.microsoft.com/dotnet/aspnet:6.0-alpine image.

System.DllNotFoundException: Failed to load the librdkafka native library.\n   at bool Confluent.Kafka.Impl.Librdkafka.TrySetDelegates(List<Type> nativeMethodCandidateTypes)\n   at void Confluent.Kafka.Impl.Librdkafka.LoadLinuxDelegates(string userSpecifiedPath)\n   at bool Confluent.Kafka.Impl.Librdkafka.Initialize(string userSpecifiedPath)

it can be fixed by apk add librdkafka-dev , but could it be solved in the kafka dotnet library?

foman on Jan 16, 2023