envoy: Crash in ConnectionImpl::onMessageBeginBase

Description: Envoy crashes with the supplied configuration file, and using tcpkali as a server.

I tested with Envoy trunk on macOS, using f95269d5f06da435d14162b14cde664edec95109.

Repro steps:

# Start tcpkali in listening mode, with a dummy reply:
$ tcpkali -T300 -l127.1:4444 --listen-mode=active -e1 'HTTP/1.1 200 OK\nContent-Length: 4\n\nnice\r\n' --dump-all

# Start Envoy with the included configuration, it will crash immediately:
$ envoy-static -c envoy-crash.json 
[2018-05-09 16:16:50.817][49754746][info][main] source/server/server.cc:188] initializing epoch 0 (hot restart version=disabled)
[2018-05-09 16:16:50.819][49754746][info][main] source/server/server.cc:190] statically linked extensions:
[2018-05-09 16:16:50.819][49754746][info][main] source/server/server.cc:192]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2018-05-09 16:16:50.819][49754746][info][main] source/server/server.cc:195]   filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash
[2018-05-09 16:16:50.819][49754746][info][main] source/server/server.cc:198]   filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2018-05-09 16:16:50.819][49754746][info][main] source/server/server.cc:201]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2018-05-09 16:16:50.826][49754746][info][main] source/server/server.cc:203]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.statsd
[2018-05-09 16:16:50.826][49754746][info][main] source/server/server.cc:205]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.zipkin
[2018-05-09 16:16:50.826][49754746][info][main] source/server/server.cc:208]   transport_sockets.downstream: raw_buffer,ssl
[2018-05-09 16:16:50.826][49754746][info][main] source/server/server.cc:211]   transport_sockets.upstream: raw_buffer,ssl
[2018-05-09 16:16:50.881][49754746][info][config] source/server/configuration_impl.cc:52] loading 1 listener(s)
[2018-05-09 16:16:50.900][49754746][info][config] source/server/configuration_impl.cc:92] loading tracing configuration
[2018-05-09 16:16:50.900][49754746][info][config] source/server/configuration_impl.cc:114] loading stats sink configuration
[2018-05-09 16:16:50.900][49754746][info][main] source/server/server.cc:387] starting main dispatch loop
[2018-05-09 16:16:50.901][49754746][info][upstream] source/common/upstream/cluster_manager_impl.cc:131] cm init: all clusters initialized
[2018-05-09 16:16:50.901][49754746][info][main] source/server/server.cc:371] all clusters initialized. initializing init manager
[2018-05-09 16:16:50.901][49754746][info][config] source/server/listener_manager_impl.cc:608] all dependencies initialized. starting workers
[2018-05-09 16:16:50.902][49754746][critical][assert] source/common/http/http1/codec_impl.cc:376] assert failure: !current_header_map_
[2018-05-09 16:16:50.902][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] Caught Abort trap: 6, suspect faulting address 0x7fff74b1fb6e
[2018-05-09 16:16:50.951][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:87] Backtrace obj<envoy-static> thr<0>:
[2018-05-09 16:16:50.951][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj<envoy-static                        0x000000010416a7d6 _ZN8backward7details6unwindINS_14StackTraceImplINS_10system_tag10darwin_tagEE8callbackEEEmT_m>
[2018-05-09 16:16:50.951][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #0 0x10416a7d6: 
[2018-05-09 16:16:50.951][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj<envoy-static>
[2018-05-09 16:16:50.985][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #1 0x10416a345: backward::StackTraceImpl<backward::system_tag::darwin_tag>::load_here(unsigned long) + 101
[2018-05-09 16:16:50.985][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #2 0x10416a141: backward::StackTraceImpl<backward::system_tag::darwin_tag>::load_from(void*, unsigned long) + 49
[2018-05-09 16:16:50.985][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #3 0x10416869e: Envoy::BackwardsTrace::captureFrom(void*) + 46
[2018-05-09 16:16:50.985][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #4 0x10416855f: Envoy::SignalAction::sigHandler(int, __siginfo*, void*) + 143
[2018-05-09 16:16:51.087][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:110] end backtrace thread 0
Abort trap: 6

Config:

{
  "listeners": [
    {
      "address": "tcp://127.0.0.1:1111",
      "filters": [
        {
          "type": "read",
          "name": "http_connection_manager",
          "config": {
            "codec_type": "http1",
            "stat_prefix": "mycluster",
            "route_config": {
              "virtual_hosts": [
                {
                  "name": "mycluster",
                  "domains": ["*"],
                  "routes": [
                    {
                      "prefix": "/",
                      "use_websocket": true,
                      "cluster": "mycluster"
                    }
                  ]
                }
              ]
            },
            "filters": [
              {
                "type": "decoder",
                "name": "router",
                "config": {}
              }
            ]
          }
        }
      ]
    }
  ],
  "admin": {
    "access_log_path": "/dev/null",
    "address": "tcp://127.0.0.1:8001"
  },
  "cluster_manager": {
    "clusters": [
      {
        "name": "mycluster",
        "connect_timeout_ms": 1000,
        "type": "strict_dns",
        "lb_type": "round_robin",
        "health_check": {
          "type": "http",
          "timeout_ms": 1000,
          "interval_ms": 5000,
          "unhealthy_threshold": 1,
          "healthy_threshold": 3,
          "path": "/health",
          "service_name": "my-service-name"
        },
        "hosts": [ { "url": "tcp://127.0.0.1:4444" } ]
      }
    ]
  }
}

Call Stack:

[2018-05-09 16:16:50.901][49754746][info][config] source/server/listener_manager_impl.cc:608] all dependencies initialized. starting workers
[2018-05-09 16:16:50.902][49754746][critical][assert] source/common/http/http1/codec_impl.cc:376] assert failure: !current_header_map_
[2018-05-09 16:16:50.902][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] Caught Abort trap: 6, suspect faulting address 0x7fff74b1fb6e
[2018-05-09 16:16:50.951][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:87] Backtrace obj<envoy-static> thr<0>:
[2018-05-09 16:16:50.951][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj<envoy-static                        0x000000010416a7d6 _ZN8backward7details6unwindINS_14StackTraceImplINS_10system_tag10darwin_tagEE8callbackEEEmT_m>
[2018-05-09 16:16:50.951][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #0 0x10416a7d6: 
[2018-05-09 16:16:50.951][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj<envoy-static>
[2018-05-09 16:16:50.985][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #1 0x10416a345: backward::StackTraceImpl<backward::system_tag::darwin_tag>::load_here(unsigned long) + 101
[2018-05-09 16:16:50.985][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #2 0x10416a141: backward::StackTraceImpl<backward::system_tag::darwin_tag>::load_from(void*, unsigned long) + 49
[2018-05-09 16:16:50.985][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #3 0x10416869e: Envoy::BackwardsTrace::captureFrom(void*) + 46
[2018-05-09 16:16:50.985][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #4 0x10416855f: Envoy::SignalAction::sigHandler(int, __siginfo*, void*) + 143
[2018-05-09 16:16:51.087][49754746][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:110] end backtrace thread 0
Abort trap: 6

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 20 (13 by maintainers)

Commits related to this issue

Most upvoted comments

Yeah we probably just need a check for this and then convert it to an invalid protocol exception. cc @alyssawilk