etcd: etcd client v3 v3.5.0 sends invalid `:authority` header field value

Starting from etcd v3.5.0, etcd client v3 sends invalid :authority header field value which violates RFC.

Suppose we created a client like so:

        c, err := clientv3.New(clientv3.Config{
                Endpoints: []string{
                        "https://example.com:2379",
                },
                ...
        }

And started making a connection and tried to get some stuff from etcd server.

In the previous etcd client v3, it sends :authority header field value like this:

example.com:2379

But now with the v3.5.0, it sends like this:

#initially=[https://example.com:2379]

According to the RFC7540, :authority header field contains the authority portion of the target URI. RFC3986 dictates that authority is:

authority   = [ userinfo "@" ] host [ ":" port ]
host        = IP-literal / IPv4address / reg-name
IP-literal = "[" ( IPv6address / IPvFuture  ) "]"
IPvFuture  = "v" 1*HEXDIG "." 1*( unreserved / sub-delims / ":" )
reg-name    = *( unreserved / pct-encoded / sub-delims )
unreserved  = ALPHA / DIGIT / "-" / "." / "_" / "~"
sub-delims  = "!" / "$" / "&" / "'" / "(" / ")"
                  / "*" / "+" / "," / ";" / "="

#initially=[https://example.com:2379] is not a valid authority.

It seems to me that there might be a bug in the code that authority is not parsed or extracted correctly. The annoying fact of this issue is that it works if server does not validate the invalid field value. But if you have a proxy between client and server which checks some non-compliant octets, then proxy refuses a request.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 6
Comments: 22 (8 by maintainers)

Commits related to this issue

fix: downgrade etcd (etcd-io/etcd/issues/13192) — committed to blairdrummond/minio-argocd-manifests by deleted user 3 years ago

Most upvoted comments

I have confirmed that both etcd and grpc never supported passing correct authority header in multi endpoint scenario. Even before v3.5 (https://github.com/etcd-io/etcd/pull/12671) changes to grpc loadbalancing we always send same authority based on first endpoint to all other endpoints. This was always limitation of golang grpc implementation, reason is that authority is always based on ClientConn struct that handles loadbalancing between multiple endpoints.

Code on grpc side:

I have confirmed behavior on etcd v3.4.16 with simple test:

Run etcd cluster with http2debug enabled (recommended to run each etcd in separate console)

GODEBUG=http2debug=2 bin/etcd-v3.4.16/etcd --name infra1 --listen-client-urls http://127.0.0.1:2379 --advertise-client-urls http://127.0.0.1:2379 --listen-peer-urls http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --enable-pprof --logger=zap --log-outputs=stderr &

GODEBUG=http2debug=2 bin/etcd-v3.4.16/etcd --name infra2 --listen-client-urls http://127.0.0.1:22379 --advertise-client-urls http://127.0.0.1:22379 --listen-peer-urls http://127.0.0.1:22380 --initial-advertise-peer-urls http://127.0.0.1:22380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --enable-pprof --logger=zap --log-outputs=stderr &

GODEBUG=http2debug=2 bin/etcd-v3.4.16/etcd --name infra3 --listen-client-urls http://127.0.0.1:32379 --advertise-client-urls http://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --enable-pprof --logger=zap --log-outputs=stderr &

Run etcdctl with multiple endpoints

./bin/etcdctl --endpoints=http://127.0.0.1:2379,http://127.0.0.1:22379,http://127.0.0.1:32379 get a

Look for log

2021-09-28 15:31:46.783764 I | http2: decoded hpack field header field ":authority" = "127.0.0.1:2379"

No matter which member gets requests, authority is always equal to address of first member.

Based on this results I conclude that Etcd was never able to provide correct authority header in multiple endpoint scenario. I will implement fix as originally proposed by @ptabor. Please let me know if I missed something.

serathius on Sep 28, 2021

In our usecase, we specify multiple endpoints to etcd client configuration. At the same time, we have a client-facing proxy at the remote endpoint for some reason. So the load balancing is done at the client, and we use proxy at the remote.

This means that we need to provide proper authority to each of the endpoints. Solution proposed by @ptabor would not be enough to fix this. Problem with authority was caused when switching from internal loadbalancing implementation to official grpc one. Based on https://github.com/grpc/grpc-go/issues/4717 it doesn’t look that current grpc implementation is not able to support what we need. This prompts me to propose revert to internal implementation.

When trying to implement @ptabor I stumbled upon lot’s of problems with tests. Even small changes in logic could result in code going into different URL parser branch and break some tests. I also found discrepancies between integration and e2e tests on edge cases. For example “unix://localhost:m00” address cannot be used in e2e (not correct URL due to bad port), but is used by integration tests that don’t use same validation and depend on multiple edge cases that should not normally happen.

Overall conclusion is that we need a set of robust e2e tests that will verify each officially supported name syntax. List of name patterns should be taken from https://github.com/grpc/grpc/blob/master/doc/naming.md. This should ensure that any fix we implement will not break any other case and confirm that revert and future reimplementation will work properly with authority header.

serathius on Sep 21, 2021