istio: Istio ingress segfaults on upgrade from 0.7 to 0.8-20180525-23-50

Regpatrol for build 0.8-20180525-23-50 failed with a segfault.

https://ibmcloud-perf.istio.io/regpatrol/release-0.8-20180525-23-50/ No_mixer and Full_istio are both failed performance runs with many 504s.

Issues observed

istio-telemetry logs have accesslogs that report a lot of traffic to istio-policy, but istio-policy has no record of it. You can check and correlate istio-policy-6db8dccc8b-8p4jt.log, istio-telemetry-67bbd74f6d-7lv68.log
segfaults: I suspect that envoy could not handle the fact CDS delivered mixer_check_server and mixer_report_server clusters were removed temporarily. Perhaps mixer client should be resilient.

[2018-05-26 12:22:38.797][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:22:40.829][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:22:43.616][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:22:46.181][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:22:48.945][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:22:51.276][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:22:51.276][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:22:51.276][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:22:51.276][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:22:51.276][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:22:51.633][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:22:54.008][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:22:56.872][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:22:59.001][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:00.809][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:02.973][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:05.732][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:06.284][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:06.285][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:06.285][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:06.285][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:06.285][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:07.744][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:09.649][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:12.169][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:14.185][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:16.536][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:18.861][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:21.057][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:21.293][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:21.293][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:21.293][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:21.293][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:21.293][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:23.649][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:25.453][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:27.813][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:29.845][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:32.813][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:33.391][119][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:437] removing cluster mixer_check_server
[2018-05-26 12:23:33.392][119][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:437] removing cluster mixer_report_server
[2018-05-26 12:23:34.121][119][info][upstream] external/envoy/source/server/lds_api.cc:61] lds: add/update listener 'http_0.0.0.0_80'

[2018-05-26 12:23:35.830][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] Caught Segmentation fault, suspect faulting address 0x0
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:90] Backtrace obj</usr/local/bin/envoy> thr<0> (use tools/stack_decode.py):
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #0 0x994558
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #1 0x995dd1
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #2 0x99655d
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #3 0x996dbd
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #4 0x9979ea
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #5 0x7430ad
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #6 0x748054
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #7 0x84a791
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #8 0x821425
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #9 0x8247d0
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #10 0x8263cd
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #11 0x8cb65b
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #12 0x8cb03d
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #13 0x8d044a
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #14 0x8cc780
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #15 0x8ce90a
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #16 0x860f1e
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #17 0x86109c
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #18 0x733af6
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #19 0x7320de
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #20 0x7328fd
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #21 0x72c287
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #22 0xa14431
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #23 0xa14b8e
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #24 0x70976d
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #25 0x567790
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #26 0x421068
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj</lib/x86_64-linux-gnu/libc.so.6>
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #27 0x2b1adddd582f
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj</usr/local/bin/envoy>
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #28 0x46a828
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:110] end backtrace thread 0
2018-05-26T12:23:35.840806Z     warn    Epoch 0 terminated with an error: signal: segmentation fault
2018-05-26T12:23:35.840843Z     warn    Aborted all epochs

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 22 (20 by maintainers)

Most upvoted comments

@mandarjog The istio-ingress component has not been present in the istio manifest file since 05/17 build. That is why the current one has been running since then. If we cleanup everything, there will not be any new ingress installed for new builds. Regpatrol does not clean the previous build because it would also delete the istio-system namespace and with that the pvc where we keep Prometheus historical data. That can be worked around, but the fact is we did not see a reason for that. Old builds would be replaced by newer ones and everything used to work fine. With all the recent changes on 0.8, probably regpatrol needs to change.

jnativio on May 28, 2018