envoy: Crash in ConnectionImpl::onMessageBeginBase
Description: Envoy crashes with the supplied configuration file, and using tcpkali as a server.
I tested with Envoy trunk on macOS, using f95269d5f06da435d14162b14cde664edec95109.
Repro steps:
# Start tcpkali in listening mode, with a dummy reply:
$ tcpkali -T300 -l127.1:4444 --listen-mode=active -e1 'HTTP/1.1 200 OK\nContent-Length: 4\n\nnice\r\n' --dump-all
# Start Envoy with the included configuration, it will crash immediately:
$ envoy-static -c envoy-crash.json
[2018-05-09 16:16:50.817][49754746][info][main] source/server/server.cc:188] initializing epoch 0 (hot restart version=disabled)
[2018-05-09 16:16:50.819][49754746][info][main] source/server/server.cc:190] statically linked extensions:
[2018-05-09 16:16:50.819][49754746][info][main] source/server/server.cc:192] access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2018-05-09 16:16:50.819][49754746][info][main] source/server/server.cc:195] filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash
[2018-05-09 16:16:50.819][49754746][info][main] source/server/server.cc:198] filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2018-05-09 16:16:50.819][49754746][info][main] source/server/server.cc:201] filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2018-05-09 16:16:50.826][49754746][info][main] source/server/server.cc:203] stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.statsd
[2018-05-09 16:16:50.826][49754746][info][main] source/server/server.cc:205] tracers: envoy.dynamic.ot,envoy.lightstep,envoy.zipkin
[2018-05-09 16:16:50.826][49754746][info][main] source/server/server.cc:208] transport_sockets.downstream: raw_buffer,ssl
[2018-05-09 16:16:50.826][49754746][info][main] source/server/server.cc:211] transport_sockets.upstream: raw_buffer,ssl
[2018-05-09 16:16:50.881][49754746][info][config] source/server/configuration_impl.cc:52] loading 1 listener(s)
[2018-05-09 16:16:50.900][49754746][info][config] source/server/configuration_impl.cc:92] loading tracing configuration
[2018-05-09 16:16:50.900][49754746][info][config] source/server/configuration_impl.cc:114] loading stats sink configuration
[2018-05-09 16:16:50.900][49754746][info][main] source/server/server.cc:387] starting main dispatch loop
[2018-05-09 16:16:50.901][49754746][info][upstream] source/common/upstream/cluster_manager_impl.cc:131] cm init: all clusters initialized
[2018-05-09 16:16:50.901][49754746][info][main] source/server/server.cc:371] all clusters initialized. initializing init manager
[2018-05-09 16:16:50.901][49754746][info][config] source/server/listener_manager_impl.cc:608] all dependencies initialized. starting workers
[2018-05-09 16:16:50.902][49754746][critical][assert] source/common/http/http1/codec_impl.cc:376] assert failure: !current_header_map_
[2018-05-09 16:16:50.902][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] Caught Abort trap: 6, suspect faulting address 0x7fff74b1fb6e
[2018-05-09 16:16:50.951][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:87] Backtrace obj<envoy-static> thr<0>:
[2018-05-09 16:16:50.951][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj<envoy-static 0x000000010416a7d6 _ZN8backward7details6unwindINS_14StackTraceImplINS_10system_tag10darwin_tagEE8callbackEEEmT_m>
[2018-05-09 16:16:50.951][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #0 0x10416a7d6:
[2018-05-09 16:16:50.951][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj<envoy-static>
[2018-05-09 16:16:50.985][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #1 0x10416a345: backward::StackTraceImpl<backward::system_tag::darwin_tag>::load_here(unsigned long) + 101
[2018-05-09 16:16:50.985][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #2 0x10416a141: backward::StackTraceImpl<backward::system_tag::darwin_tag>::load_from(void*, unsigned long) + 49
[2018-05-09 16:16:50.985][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #3 0x10416869e: Envoy::BackwardsTrace::captureFrom(void*) + 46
[2018-05-09 16:16:50.985][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #4 0x10416855f: Envoy::SignalAction::sigHandler(int, __siginfo*, void*) + 143
[2018-05-09 16:16:51.087][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:110] end backtrace thread 0
Abort trap: 6
Config:
{
"listeners": [
{
"address": "tcp://127.0.0.1:1111",
"filters": [
{
"type": "read",
"name": "http_connection_manager",
"config": {
"codec_type": "http1",
"stat_prefix": "mycluster",
"route_config": {
"virtual_hosts": [
{
"name": "mycluster",
"domains": ["*"],
"routes": [
{
"prefix": "/",
"use_websocket": true,
"cluster": "mycluster"
}
]
}
]
},
"filters": [
{
"type": "decoder",
"name": "router",
"config": {}
}
]
}
}
]
}
],
"admin": {
"access_log_path": "/dev/null",
"address": "tcp://127.0.0.1:8001"
},
"cluster_manager": {
"clusters": [
{
"name": "mycluster",
"connect_timeout_ms": 1000,
"type": "strict_dns",
"lb_type": "round_robin",
"health_check": {
"type": "http",
"timeout_ms": 1000,
"interval_ms": 5000,
"unhealthy_threshold": 1,
"healthy_threshold": 3,
"path": "/health",
"service_name": "my-service-name"
},
"hosts": [ { "url": "tcp://127.0.0.1:4444" } ]
}
]
}
}
Call Stack:
[2018-05-09 16:16:50.901][49754746][info][config] source/server/listener_manager_impl.cc:608] all dependencies initialized. starting workers
[2018-05-09 16:16:50.902][49754746][critical][assert] source/common/http/http1/codec_impl.cc:376] assert failure: !current_header_map_
[2018-05-09 16:16:50.902][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] Caught Abort trap: 6, suspect faulting address 0x7fff74b1fb6e
[2018-05-09 16:16:50.951][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:87] Backtrace obj<envoy-static> thr<0>:
[2018-05-09 16:16:50.951][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj<envoy-static 0x000000010416a7d6 _ZN8backward7details6unwindINS_14StackTraceImplINS_10system_tag10darwin_tagEE8callbackEEEmT_m>
[2018-05-09 16:16:50.951][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #0 0x10416a7d6:
[2018-05-09 16:16:50.951][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj<envoy-static>
[2018-05-09 16:16:50.985][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #1 0x10416a345: backward::StackTraceImpl<backward::system_tag::darwin_tag>::load_here(unsigned long) + 101
[2018-05-09 16:16:50.985][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #2 0x10416a141: backward::StackTraceImpl<backward::system_tag::darwin_tag>::load_from(void*, unsigned long) + 49
[2018-05-09 16:16:50.985][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #3 0x10416869e: Envoy::BackwardsTrace::captureFrom(void*) + 46
[2018-05-09 16:16:50.985][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #4 0x10416855f: Envoy::SignalAction::sigHandler(int, __siginfo*, void*) + 143
[2018-05-09 16:16:51.087][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:110] end backtrace thread 0
Abort trap: 6
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 20 (13 by maintainers)
Commits related to this issue
- http1: fix crash when upstream sends extra CR/LF between responses Requires pulling upstream http-parser fix. Fixes https://github.com/envoyproxy/envoy/issues/3337 Signed-off-by: Matt Klein <mklein... — committed to envoyproxy/envoy by deleted user 6 years ago
- http1: fix crash when upstream sends extra CR/LF between responses (#3913) Requires pulling upstream http-parser fix. Fixes https://github.com/envoyproxy/envoy/issues/3337 Signed-off-by: Matt K... — committed to envoyproxy/envoy by deleted user 6 years ago
Yeah we probably just need a check for this and then convert it to an invalid protocol exception. cc @alyssawilk