fluent-bit: out_http: amd64.c:121: crash: Assertion `0' failed.

Reported on #429, we will track this problem here:

[2018/04/07 01:44:14] [ info] [out_http] HTTP STATUS=200
[2018/04/07 01:44:15] [ info] [out_http] HTTP STATUS=200
fluent-bit: /tmp/src/lib/monkey/deps/flb_libco/amd64.c:121: crash: Assertion `0' failed.
Fluent-Bit v0.13.0
Copyright (C) Treasure Data

[engine] caught signal
#0  0x7f6defdfa529      in  ???() at ???:0
#1  0x7f6defdf1e66      in  ???() at ???:0
#2  0x7f6defdf1f11      in  ???() at ???:0
#3  0x55994d80c389      in  ???() at ???:0
#4  0xffffffffffffffff  in  ???() at ???:0

This problem is associated to the bad use of co-routines implementation, likely there is an unexpected and explicit return from a co-routine that trigger this problem.

cc: @StevenACoffman @jgsqware @onorua

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 78 (71 by maintainers)

Commits related to this issue

Most upvoted comments

Same for me! Thanks

+1
jgsqware on Apr 24, 2018

That’s running an older version of Fluent Bit

On Mon, Apr 23, 2018, 15:45 Steve Coffman notifications@github.com wrote:

I applied the change to another cluster, and within the first few seconds I got:

$ kubectl logs -n kangaroo fluent-bit-qrb8v -c fluent-bit --previous [2018/04/23 21:41:44] [ info] [engine] started [2018/04/23 21:41:44] [ info] [filter_kube] https=1 host=kubernetes.default.svc.cluster.local port=443 [2018/04/23 21:41:44] [ info] [filter_kube] local POD info OK [2018/04/23 21:41:44] [ info] [filter_kube] testing connectivity with API server… [2018/04/23 21:41:50] [ info] [filter_kube] API server connectivity OK [2018/04/23 21:41:50] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020 [2018/04/23 21:41:51] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:51] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:51] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:51] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:51] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:51] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:51] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:51] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:51] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:51] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:51] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:51] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:51] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:51] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:52] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:52] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:52] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:52] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:52] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:52] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:52] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:52] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:53] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:53] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:53] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:53] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:54] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:54] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:54] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:54] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:54] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:54] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:55] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:55] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:55] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:55] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:56] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:56] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:56] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:56] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:57] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:57] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:57] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:57] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:58] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:58] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:58] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:58] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:58] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:58] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:58] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:58] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:59] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:59] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:59] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:41:59] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:00] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:00] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:00] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:00] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:01] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:01] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:01] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:01] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:01] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:01] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:02] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:02] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:02] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:02] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:02] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:02] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:03] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:03] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:03] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:03] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:03] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:03] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:03] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:04] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:04] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:04] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:04] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:04] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:05] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:05] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:05] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:05] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:05] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:05] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:05] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:05] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:06] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:06] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:06] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:06] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:06] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:06] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:06] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:06] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:06] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:07] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:07] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:07] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:07] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:07] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:08] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:08] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:08] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:08] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:08] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:08] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:09] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:09] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:09] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:09] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:09] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:09] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:09] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:10] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:10] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:10] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:10] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:10] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:10] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:11] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:11] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:11] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:11] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:11] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:11] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:12] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:12] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:12] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:12] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:12] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:12] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:12] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:12] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:12] [ info] [out_http] HTTP STATUS=200 Fluent-Bit v0.13.0 Copyright © Treasure Data

retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read retry read [2018/04/23 21:42:13] [ info] [out_http] HTTP STATUS=200 [engine] caught signal [2018/04/23 21:42:13] [ info] [out_http] HTTP STATUS=200 [2018/04/23 21:42:13] [ info] [out_http] HTTP STATUS=200 retry rea

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fluent/fluent-bit/issues/557#issuecomment-383733415, or mute the thread https://github.com/notifications/unsubscribe-auth/AAWkNlMl8JiUIUW55-lYeuHZto8NsWxcks5trkuFgaJpZM4TLGmM .

+1
edsiper on Apr 23, 2018

@edsiper fluent/fluent-bit-0.13-dev:0.16 with HTTP_Server On is in CrashLoopBackOff for all applied containers:

[2018/04/10 20:57:33] [ info] [out_http] HTTP STATUS=200
[2018/04/10 20:57:33] [ info] [out_http] HTTP STATUS=200
[engine] caught signal
Fluent-Bit v0.13.0
Copyright (C) Treasure Data

#0  0x55a212d8e727      in  __mk_list_del() at lib/monkey/include/monkey/mk_core/mk_list.h:87
#1  0x55a212d8e75e      in  mk_list_del() at lib/monkey/include/monkey/mk_core/mk_list.h:93
#2  0x55a212d8eb64      in  mk_http_thread_purge() at lib/monkey/mk_server/mk_http_thread.c:197
#3  0x55a212d8e8e6      in  thread_cb_init_vars() at lib/monkey/mk_server/mk_http_thread.c:104
#4  0x55a212d995e6      in  co_init() at lib/monkey/deps/flb_libco/amd64.c:117
#5  0xffffffffffffffff  in  ???() at ???:0

Some error immediately, some run for a minute, but then they all crash with the same error.

+1
StevenACoffman on Apr 10, 2018
Read more comments on GitHub
← fluent-bit: out_cloudwatch: Segmentation fault
fluent-bit: Output ES fails to flush chunk with status 0 →