node-rdkafka: node-rdkafka 2.3.0 Segmentation fault

Since 2.3.0, as soon as connect() method is called (from consumer or producer), a Segmentation fault occured.

Steps to reproduce using docker and e2e tests for brevity :

Dockerfile :

FROM node:9-stretch
RUN npm i node-rdkafka
WORKDIR /node_modules/node-rdkafka
RUN npm i
ENV KAFKA_HOST=localhost:29092

e2e tests run :

$ docker build -t rdkafka .
(...)

$ docker run --net host rdkafka make e2e
  Consumer
    commit
Makefile:63: recipe for target 'e2e' failed
make: *** [e2e] Segmentation fault (core dumped)

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 4
Comments: 17 (7 by maintainers)

Most upvoted comments

The problem here is that librdkafka built is not used inside node module, because node module links against system librdkafka, not the local librdkafka. And system’s librdkafka is usually linked against OpenSSL 1.1 but node is built against OpenSSL 1.0, and this causes the segmentation fault. See PR #388 for the fix.

Also in Debian, you should make sure that you have libssl1.0-dev installed, not libssl-dev (1.1) so that npm builds librdkafka correctly against OpenSSL 1.0.

+14

tvainika on Mar 30, 2018

On Fedora x86_64, with node 8.9.4 and suitable debug parts enabled (Ugh!):

Thread 10 "node" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe6108700 (LWP 3481)]
__strcmp_sse2_unaligned () at ../sysdeps/x86_64/multiarch/strcmp-sse2-unaligned.S:31
31		movdqu	(%rdi), %xmm1
(gdb) where
#0  __strcmp_sse2_unaligned () at ../sysdeps/x86_64/multiarch/strcmp-sse2-unaligned.S:31
#1  0x0000000000994b9a in lh_insert ()
#2  0x00000000009a0081 in OBJ_NAME_add ()
#3  0x00007fffe7938841 in ossl_init_ssl_base () at ssl/ssl_init.c:72
#4  ossl_init_ssl_base_ossl_ () at ssl/ssl_init.c:25
#5  0x00007ffff6ed4227 in __pthread_once_slow (once_control=0x7fffe7b77c3c <ssl_base>, init_routine=0x7fffe7938600 <ossl_init_ssl_base_ossl_>) at pthread_once.c:116
#6  0x00007ffff6ed42e5 in __GI___pthread_once (once_control=once_control@entry=0x7fffe7b77c3c <ssl_base>, init_routine=init_routine@entry=0x7fffe7938600 <ossl_init_ssl_base_ossl_>) at pthread_once.c:143
#7  0x00007fffe7d2d779 in CRYPTO_THREAD_run_once (once=once@entry=0x7fffe7b77c3c <ssl_base>, init=init@entry=0x7fffe7938600 <ossl_init_ssl_base_ossl_>) at crypto/threads_pthread.c:106
#8  0x00007fffe793897b in OPENSSL_init_ssl (opts=opts@entry=2097154, settings=settings@entry=0x0) at ssl/ssl_init.c:227
#9  0x00007ffff43fb13d in rd_kafka_transport_ssl_init () at rdkafka_transport.c:481
#10 0x00007ffff43d811a in rd_kafka_global_cnt_incr () at rdkafka.c:134
#11 rd_kafka_new (type=type@entry=RD_KAFKA_CONSUMER, app_conf=0x7fffd4000b90, errstr=errstr@entry=0x7fffe6107b70 "", errstr_size=errstr_size@entry=512) at rdkafka.c:1319
#12 0x00007ffff46c0960 in RdKafka::KafkaConsumer::create (conf=<optimized out>, errstr="") at KafkaConsumerImpl.cpp:63
#13 0x00007ffff48f036e in NodeKafka::KafkaConsumer::Connect() () from /home/andreas/project/collaborne-event-stream-library/node_modules/node-rdkafka/build/Release/node-librdkafka.node
#14 0x00007ffff48fd847 in NodeKafka::Workers::KafkaConsumerConnect::Execute() () from /home/andreas/project/collaborne-event-stream-library/node_modules/node-rdkafka/build/Release/node-librdkafka.node
#15 0x00007ffff48edcf2 in Nan::AsyncExecute(uv_work_s*) () from /home/andreas/project/collaborne-event-stream-library/node_modules/node-rdkafka/build/Release/node-librdkafka.node
#16 0x000000000143c001 in worker (arg=<optimized out>) at ../deps/uv/src/threadpool.c:83
#17 0x00007ffff6ecc50b in start_thread (arg=0x7fffe6108700) at pthread_create.c:465
#18 0x00007ffff6c0416f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

I’m using a non-SSL connection to a local kafka broker, but from what I understand this is generic SSL-support code initialization?

EDIT: On a whim I modified node_modules/node-rdkafka/configure in my project, added --disable-ssl, and executed npm rebuild. This fixes the immediate problem, and I can work again.

ankon on Mar 19, 2018

So, poking further through NodeJS: Essentially a third-party module cannot really link to OpenSSL without the danger of that conflicting in some way with whatever NodeJS comes bundled with.

How do I make the node process agree with the librdkafka SSL version?

Actually this is “reverse”, instead I needed to get librdkafka/node-rdkafka agree with the version used by NodeJS. On Fedora this required to remove the openssl-devel package, and instead installing the compat-openssl10-devel package:

$ sudo dnf install compat-openssl10-devel.x86_64 --allowerasing
$ rm -rf build && npm install # rebuild node-rdkafka
$ ldd build/Release/node-librdkafka.node 
	linux-vdso.so.1 (0x00007ffcc358e000)
	librdkafka.so.1 => /home/andreas/project/node-rdkafka/build/deps/librdkafka.so.1 (0x00007f72323d6000)
	librdkafka++.so.1 => /home/andreas/project/node-rdkafka/build/deps/librdkafka++.so.1 (0x00007f72321b9000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f7231e32000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f7231ae7000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f72318d0000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f72316b2000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f72312fc000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f72310e5000)
	libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f7230c87000)
	libssl.so.10 => /lib64/libssl.so.10 (0x00007f7230a1b000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f7230813000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f723060f000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f72328fc000)

With this setup my reproducible segfaults go away after running npm rebuild node-rdkafka in the affected projects.

Maybe this could be added as a requirement or such in the README file here?

ankon on Apr 3, 2018