istio: Istio ingress segfaults on upgrade from 0.7 to 0.8-20180525-23-50
Regpatrol for build 0.8-20180525-23-50 failed with a segfault.
https://ibmcloud-perf.istio.io/regpatrol/release-0.8-20180525-23-50/ No_mixer and Full_istio are both failed performance runs with many 504s.
Issues observed
-
istio-telemetry logs have accesslogs that report a lot of traffic to
istio-policy
, butistio-policy
has no record of it. You can check and correlateistio-policy-6db8dccc8b-8p4jt.log, istio-telemetry-67bbd74f6d-7lv68.log
-
segfaults: I suspect that envoy could not handle the fact CDS delivered mixer_check_server and mixer_report_server clusters were removed temporarily. Perhaps mixer client should be resilient.
[2018-05-26 12:22:38.797][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:22:40.829][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:22:43.616][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:22:46.181][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:22:48.945][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:22:51.276][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:22:51.276][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:22:51.276][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:22:51.276][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:22:51.276][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:22:51.633][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:22:54.008][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:22:56.872][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:22:59.001][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:00.809][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:02.973][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:05.732][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:06.284][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:06.285][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:06.285][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:06.285][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:06.285][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:07.744][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:09.649][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:12.169][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:14.185][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:16.536][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:18.861][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:21.057][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:21.293][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:21.293][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:21.293][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:21.293][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:21.293][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,
[2018-05-26 12:23:23.649][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:25.453][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:27.813][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:29.845][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:32.813][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error
[2018-05-26 12:23:33.391][119][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:437] removing cluster mixer_check_server
[2018-05-26 12:23:33.392][119][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:437] removing cluster mixer_report_server
[2018-05-26 12:23:34.121][119][info][upstream] external/envoy/source/server/lds_api.cc:61] lds: add/update listener 'http_0.0.0.0_80'
[2018-05-26 12:23:35.830][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] Caught Segmentation fault, suspect faulting address 0x0
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:90] Backtrace obj</usr/local/bin/envoy> thr<0> (use tools/stack_decode.py):
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #0 0x994558
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #1 0x995dd1
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #2 0x99655d
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #3 0x996dbd
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #4 0x9979ea
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #5 0x7430ad
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #6 0x748054
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #7 0x84a791
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #8 0x821425
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #9 0x8247d0
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #10 0x8263cd
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #11 0x8cb65b
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #12 0x8cb03d
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #13 0x8d044a
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #14 0x8cc780
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #15 0x8ce90a
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #16 0x860f1e
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #17 0x86109c
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #18 0x733af6
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #19 0x7320de
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #20 0x7328fd
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #21 0x72c287
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #22 0xa14431
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #23 0xa14b8e
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #24 0x70976d
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #25 0x567790
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #26 0x421068
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj</lib/x86_64-linux-gnu/libc.so.6>
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #27 0x2b1adddd582f
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj</usr/local/bin/envoy>
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #28 0x46a828
[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:110] end backtrace thread 0
2018-05-26T12:23:35.840806Z warn Epoch 0 terminated with an error: signal: segmentation fault
2018-05-26T12:23:35.840843Z warn Aborted all epochs
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 22 (20 by maintainers)
@mandarjog The istio-ingress component has not been present in the istio manifest file since 05/17 build. That is why the current one has been running since then. If we cleanup everything, there will not be any new ingress installed for new builds. Regpatrol does not clean the previous build because it would also delete the istio-system namespace and with that the pvc where we keep Prometheus historical data. That can be worked around, but the fact is we did not see a reason for that. Old builds would be replaced by newer ones and everything used to work fine. With all the recent changes on 0.8, probably regpatrol needs to change.