node-rdkafka: Error: librdkafka++.so.1: cannot open shared object file

Reposting of #394, as it was closed by the issue author without being fixed.

When trying to use run on Heroku (and I’m guessing other linuxed-powered server environments), the following error is triggered trying to require node-rdkafka@2.3.2:

Error: librdkafka++.so.1: cannot open shared object file: No such file or directory
    at Object.Module._extensions..node (module.js:681:18)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at bindings (/app/node_modules/bindings/bindings.js:81:44)
    at Object.<anonymous> (/app/node_modules/node-rdkafka/librdkafka.js:10:32)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)

The author of #394 suggested adding an Aptfile to your project, however, this is non-customary for npm packages. That said, I can’t exactly suggest a solution myself, being really unfamiliar with how Node.js binds to C/C++, the build systems work, etc.

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 12
Comments: 33 (12 by maintainers)

Most upvoted comments

I had the same Issue on Ubuntu 18.04. This worked for me:

sudo apt install librdkafka-dev
reinstall node-rdkafka

+13

Michael-Hoffmann on Jul 21, 2018

We temporrarly solved this issue by supplying the following env var: LD_LIBRARY_PATH=<application path>/node_modules/node-rdkafka/build/deps Yes, it ain’t pretty

bobrowadam on Jul 9, 2018

Sorry everyone. I’m back after a tough few weeks with little time to prioritize my open source contributions.

I upgraded the library to 2.3.3 which makes it so the configure step happens in the gypfile again. When I was seeing the issue you on this thread on a coworker’s machine what I did to the gypfile fixed it. Can you please try again on the new build?

webmakersteve on May 19, 2018

This problem showed up for me on Ubuntu 18.04 for the first time. It did not exist on Ubuntu 16.04 and earlier.

yfinkelstein on Aug 11, 2018

I encountered this when building our app on a node image and then transferring the built result to a different image. My testing indicates that @JaapRood’s suggestion that this is due to a relative/absolute path issue is correct. If I ensure that the build path on the first image exactly matches the path on the runtime image, the librdkafka dependency resolves correctly. If the paths differ at all, the dependency is not found.

ctrom on Jun 14, 2018

I’d like to see this fixed in the code for this project, but the workaround that we’re applying is to BUILD_LIBRDKAFKA=0 yarn install and provide the librdkafka dependency by doing an apk add librdkafka in the alpine image we are using. The BUILD_LIBRDKAFKA environment variable is used in the binding.gyp file to control whether the librdkafka dependency gets built as part of the npm/yarn install or should not be built and should be found in /usr.

Note that this only works with alpine:3.7 and higher (due to the version of librdkafka that is required) which means that you’ll need to be using at least the node:10-alpine image as the node:9-alpine image uses an older version of alpine.

ctrom on Jun 18, 2018

@bobrowadam 's fix worked for me!

pimvanderheijden on Jul 24, 2018

Could this be solved by removing all of the <(module_root_dir)>/ since the paths are relative by default?

File path names are relative to the directory in which the .gyp file lives.

https://gyp.gsrc.io/docs/UserDocumentation.md#add-a-source-file-that-builds-on-all-platforms

https://github.com/Blizzard/node-rdkafka/blob/master/binding.gyp#L79 would become build/deps/librdkafka.so, and so on?

alexander-alvarez on Jul 9, 2018

@wiesson When doing the BUILD_LIBRDKAFKA=0 yarn install, some of the build steps still need to run. You will need to install the build dependencies on the node:alpine image so the build can complete, or use an image that already has those dependencies in place.

ctrom on Jun 19, 2018

Just wanted to report back as I finally got around to setting up a reproducer. I have the same Jenkins build running npm install and creating docker images from 2.2.2 forward. No errors from the 2.2.x series but unfortunately this still affects me with the 2.3.3 version. As a side effect of the compile changes made to 2.3.3 I can no longer build successfully on my Mac.
In my Jenkinsfile I am using upstream Docker node:8.11 ( https://hub.docker.com/_/node/ ) and then building my image with Docker version 17.12.1 from CoreOS stable: https://coreos.com/releases/#1688.5.3 . I was not however able to reproduce locally with separate Docker containers on my Mac with Docker for Mac version: 18.05.0-ce

battlecow on May 23, 2018

@webmakersteve I run my app in docker and copy the entire node_modules directory over to the final image, I have had to go about deleting this lib directory and re-running npm install on image build.

battlecow on Apr 25, 2018

I had this issue when building and moving assets from one docker image to another… which we initially needed to do because we were using node-slim which doesn’t have all of the libraries to build librdkafka. Building it on the regular node image, and not copying it over to a different container from where it was built fixed the issue for me.

alexander-alvarez on Apr 19, 2018