gloo: Caught Segmentation fault, suspect faulting Envoy::Router::ConfigImpl::route()

Hi

Under high load (around 18M+ rpm) envoy crashed with the following backtrace after 15m of running.

[2019-03-28 15:21:53.325][8][info][main] [external/envoy/source/server/drain_manager_impl.cc:63] shutting down parent after drain
[2019-03-28 15:43:04.786][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:81] Caught Segmentation fault, suspect faulting address 0x0
[2019-03-28 15:43:04.786][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:69] Backtrace (use tools/stack_decode.py to get line numbers):
[2019-03-28 15:43:04.786][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #0: [0x7fc5b6b6d4b0]
[2019-03-28 15:43:04.790][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #1: Envoy::Router::ConfigImpl::route() [0xb78bd5]
[2019-03-28 15:43:04.794][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #2: Envoy::Http::ConnectionManagerImpl::ActiveStream::refreshCachedRoute() [0xa38be2]
[2019-03-28 15:43:04.797][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #3: Envoy::Http::ConnectionManagerImpl::ActiveStreamFilterBase::route() [0xa3c9ef]
[2019-03-28 15:43:04.802][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #4: Envoy::Extensions::HttpFilters::Transformation::TransformationFilter::checkResponseActive() [0x47c2ca]
[2019-03-28 15:43:04.805][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #5: Envoy::Extensions::HttpFilters::Transformation::TransformationFilter::encodeHeaders() [0x47bc7c]
[2019-03-28 15:43:04.809][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #6: Envoy::Http::ConnectionManagerImpl::ActiveStream::encodeHeaders() [0xa3a61f]
[2019-03-28 15:43:04.812][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #7: Envoy::Http::Utility::sendLocalReply() [0xbd8523]
[2019-03-28 15:43:04.816][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #8: Envoy::Http::ConnectionManagerImpl::ActiveStream::sendLocalReply() [0xa376ad]
[2019-03-28 15:43:04.820][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #9: Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders() [0xa384b2]
[2019-03-28 15:43:04.823][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #10: Envoy::Http::Http1::ServerConnectionImpl::onMessageComplete() [0xa46f9b]
[2019-03-28 15:43:04.827][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #11: Envoy::Http::Http1::ConnectionImpl::onMessageCompleteBase() [0xa45ec9]
[2019-03-28 15:43:04.831][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #12: Envoy::Http::Http1::ConnectionImpl::$_7::__invoke() [0xa4966d]
[2019-03-28 15:43:04.834][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #13: http_parser_execute [0xbf21ee]
[2019-03-28 15:43:04.838][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #14: Envoy::Http::Http1::ConnectionImpl::dispatchSlice() [0xa4580b]
[2019-03-28 15:43:04.842][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #15: Envoy::Http::Http1::ConnectionImpl::dispatch() [0xa455ef]
[2019-03-28 15:43:04.845][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #16: Envoy::Http::ConnectionManagerImpl::onData() [0xa35659]
[2019-03-28 15:43:04.849][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #17: Envoy::Network::FilterManagerImpl::onRead() [0x86a8ca]
[2019-03-28 15:43:04.853][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #18: Envoy::Network::ConnectionImpl::onReadReady() [0x86747e]
[2019-03-28 15:43:04.856][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #19: Envoy::Network::ConnectionImpl::onFileEvent() [0x866f51]
[2019-03-28 15:43:04.860][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #20: Envoy::Event::FileEventImpl::assignEvents()::$_0::__invoke() [0x862195]
[2019-03-28 15:43:04.863][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #21: event_process_active_single_queue [0xbe43cd]
[2019-03-28 15:43:04.866][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #22: event_base_loop [0xbe2980]
[2019-03-28 15:43:04.870][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #23: Envoy::Event::DispatcherImpl::run() [0x861687]
[2019-03-28 15:43:04.873][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #24: Envoy::Server::WorkerImpl::threadRoutine() [0x85c0d2]
[2019-03-28 15:43:04.876][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #25: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::$_0::__invoke() [0xd56c55]
[2019-03-28 15:43:04.876][89][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #26: [0x7fc5b6b62cf8]