etcd: etcd not compatible with grpc v1.30.0

I don’t need a work around, I want to use grpc v1.30.0 to keep up to date.

Is there a timeline for the 3.5 milestone and whatever fix ends up getting committed?

Any updates?

I am afraid if this requires us to rewrite our client third time just to make it compatible with gRPC 😕

I would recommend not using gRPC APIs explicitly marked as experimental in the documentation. This will avoid any potential future breakages. If you are missing functionality you need from our stable APIs, please reach out to us by filing an issue, and we will work with you to find a solution.

A solution would be for the grpc-go team to:

• Revert the breaking of relevant experimental elements, and provide a backwards-compatible solution, so this very important dependency can move forward, if nothing else then through aliases.
• Put all experimental APIs in the future under an experimental package, to make it clear that the APIs are experimental by Golang API standards.

The grpc-go project is v1.0, and people (understandably) use features from it without checking the Godoc if something is experimental or not, as the general expectation in the Golang community is that a SemVer tag of v1.0+ means the Golang API is stable. Additionally, there have been things in grpc-go that have been experimental for years that the community comes to rely on (for example, interceptors were marked as experimental for a number of years, even though github.com/grpc-ecosystem had code that relied on them). Notices in a GitHub issue such as https://github.com/grpc/grpc-go/issues/3180 aren’t sufficient by Golang standards to denote a Golang API breakage, especially in such a core, practically standard-library project as grpc-go.

Perhaps the etcd team can work with the gRPC team to make sure this can happen? This seems to be breaking a significant number of people.

golang current package management is worse than c++ in the 1990s. What a mess.

I am afraid if this requires us to rewrite our client third time just to make it compatible with gRPC 😕

I will be working on some design doc about our long term plan. I would like to decouple clientv3 from gRPC as much as possible.

As a workaround, you can pin it to 1.29.1:

require ( google.golang.org/grpc v1.29.1 )

But we should migrate out of the deprecated naming namespace. (and in theory grpc should not deprecate within the same major version).

On @maja42’s note: In our org everyone is moving to the newer version of protobuf and gRPC except the components which rely on etcd. It is really painful from a maintenance perspective to have snowflake services which must be treated differently from others and cannot use newer features or bug fixes.

Is there a timeline for the 3.5 release and is it safe to assume that this will be fixed then? I’m currently building a simple gRPC server which happens to also talk to and etcd server. I think I would have to downgrade protoc-gen-grpc-go + the pin on grpc 1.29.1 😦

go.mod replace google.golang.org/grpc v1.30.0 => google.golang.org/grpc v1.26.0

Almost every Golang project in the industry does not consider Godocs to be clear labeling, and considers the only versioning policy to be SemVer.

A GitHub issue giving three months notice to delete a Godoc-labeled-experimental package, along with a markdown document explaining a different versioning policy, would not be considered acceptable for any other org in the industry - moreover, no other org does this.

If you need to experiment, you’re more than able to with clear labeling (such as an experimental package) that doesn’t cause non-experimental packages to break SemVer, and cause so much downstream pain for users of such a key library.

The grpc-go team not abiding by this has caused lots of pain in the Golang community - I think this needs to be re-considered.

A solution would be for the grpc-go team to:

We have no plans to change our development practices. We require the ability to experiment with new APIs and change them over time. There is no feasible way to restrict these to a separate package or module. We very clearly label our experimental APIs and our versioning policy.

The interceptors you reference are actually a success story: they were recently marked stable upon a request from a user. If you have something you want to use, and it is marked experimental, and you cannot tolerate breakages from upgrades, then please reach out to us in our repo and we will work with you to find an alternative or stabilize the necessary API. Your use case could help us determine whether the current version of the API is acceptable or needs work.

The solution to this issue is for etcd to stop using grpc APIs that are explicitly marked experimental.

We (Kubernetes) would also like to see etcd updated to use the latest grpc. Ideally, use grpc@v1.31.0 when it’s released to include https://github.com/grpc/grpc-go/issues/2628 (needed for https://github.com/kubernetes/kubernetes/issues/93320).

@gyuho 👋 do you have suggestions on which of the above approaches to take here?

I think there are two issues here and you are arguing about the wrong one. While the development process of grpc-go might or might not be good for the community, that issue should be discussed on it’s own on their github page. Their current approach certainly caused confusion down the line and the current mess for people trying to use both grpc and etcd.

What should be discussed here, though, is a plan on how to get out of the current situation and if there is a timeline people can expect. I already considered using a different store than etcd due to this issue and will probably do so once I get in a situation where I need libraries that are inherently unusable due to grpc versioning. I wouldn’t be surprised if this issue already drove off others.

After a bit of investigation. It looks like upgrading (switch from grpc/naming to grpc/resolver) isn’t possible without breaking compatibility.

clientv3/naming has public function signatures that depend on grpc/naming. A migration, which needs to be complete eventually requires 2 steps:

1. Create a new package clientv3/resolver and depreciate clientv3/naming, this allows usage of grpc up to version 1.29.
2. Eventually, remove clientv3/naming, allowing usage of grpc v1.30 and up.

Alternatively, we can copy grpc/naming over to this project or its own namespace and rewrite imports. This still produces a breaking change but is much easier for users to adopt. Since grpc/naming should not be used anymore, switching to resolver is still the goal, but this allow keeping clientv3/naming and update grpc at the same time.

replace github.com/coreos/etcd => go.etcd.io/etcd/v3 v3.5.0-alpha.0 works for me

1. The module support is ready - we are waiting for a maintainer to perform first experimental release using: (tracked by https://github.com/etcd-io/etcd/issues/12498).

2. I agree getting to grpc 1.30 is very difficult due to gogo (to be more preceise due to cmux test that depend on protobuf-1.4)

3. We can have grpc-1.30 compatible etcd implementation in 3.5 while we stay on 1.29.1. The API change proposal is awaiting feedback in: https://github.com/etcd-io/etcd/pull/12614

+1 to what @maja42 said. Is there a plan and timeline for how to get out of the current situation? Lots of people naturally want to move to modern versions of grpc and anything that depends on later versions of grpc, and are currently blocked.

@dfawley

I would recommend not using gRPC APIs explicitly marked as experimental in the documentation. This will avoid any potential future breakages. If you are missing functionality you need from our stable APIs, please reach out to us by filing an issue, and we will work with you to find a solution.

Hmm… What gRPC-go did was to replace an experimental API with another experimental API (see https://pkg.go.dev/google.golang.org/grpc@v1.30.0/resolver) with no migration path.

@gyuho Can we simply remove the gRPC naming support in etcd latest? I see no points on supporting it if gRPC-go cannot figure out its naming store in and keep it experimental for 4 years.

The users of etcd/naming package also depends on gRPC/naming. So it is OK just to remove it as we upgrade gRPC.

I wonder if this should be postponed to 3.6.

The community is anxiously waiting for proper module support. Since etcd is a common dependency in a lot of foundational components (viper, go-kit, etc), I’d prioritize getting the module support released over fixing this issue.

My 2 cents.

@gyuho I have a PR submit to update grpc-go version to v1.32.0. Please help to review.

we can’t simply remove etcd naming package because it’s used by etcd grpcproxy package.

etcd 3.5 uses grpc 1.41:

https://github.com/etcd-io/etcd/blob/b10adb6abe456778af0b12fe4709aa7b54ef7c16/go.mod#L35

Please open a new issue with an exact description if there is another problem with:

• grpc version that is incompatible
• etcd version that has the problem

Please review the client/naming API change proposal: https://github.com/etcd-io/etcd/pull/12614 that preserves the functionality of naming API (resolver on endpoint storage in etcd), while it abstracts grpc implementation away.

This will allow to make etcd-3.5 implementation grpc-1.30+ compatible (in terms of code). But the real upgrate to etcd-1.30 will be still blocked on lack of interoperability between gogo-generated protobufs & golang/protobuf-1.4.x runtime that is being transitively polled.

@ptabor thanks a lot for your work on the module support! I know it’s ready, I’ve been playing with it and it works great, but libraries can’t upgrade to it until it’s released (see my comment in #12498).

So at this point I’d try to push for an immediate experimental release and a hard, close deadline for 3.5. Every single day without proper module support is a pain. 😞

I know things are usually not that easy in open source, but there hasn’t been a single word about 3.5 from the maintainers for quite some time now which is a little bit troubling.

There are 2 recent attempts to update the grpc:

https://github.com/etcd-io/etcd/pull/12398

And: https://github.com/etcd-io/etcd/pull/12597 (from this attempt it seems that grpc-1.30+ is polling transitively protobuf-1.4.x that is gogo/protobuf-1.3.1 generated protos incompatible). If the findings are correct:

• either we need to solve the gogo/protobuf & protobuf-1.4.x compatibility issue first
• or assume the dependency of protobuf-1.4.x is soft and require all consumers of etcd to override the protobuf-1.4.x import with 1.3.5 import.

1. In 3.5 we will release this on a new package names: go.etcd.io/etcd/client/v3 instead of go.etcd.io/etcd/clientv3 so either way it will be opt-in change for customers migrating to 3.5.

2. customer that depended on google.golang.org/grpc/naming.Update will need to change their code do use go.etcd.io/etcd/client/v3/naming.Update (as this is struct, not interface), so either way it will be a change requiring a code adaptation.

@gyuho I have a PR submit to update grpc-go version to v1.32.0. Please help to review.

we can’t simply remove etcd naming package because it’s used by etcd grpcproxy package.

I will take a look.

I wonder how more complicated is proper change of grpcproxy code to make it depend on grpc/resolver packages. Do you have any unfinished prototype to compare ? High level cloning of deprecated ‘naming.go’ sounds like caring over the debt, while migration is paying it.

git grep ‘(grpc|client)/naming’ shows that the usage is very ‘local’:

clientv3/naming/doc.go://               "google.golang.org/grpc/naming"
clientv3/naming/grpc.go:        "google.golang.org/grpc/naming"
clientv3/naming/grpc.go://      "google.golang.org/grpc/naming"

proxy/grpcproxy/cluster.go:     gnaming "google.golang.org/grpc/naming"
proxy/grpcproxy/cluster.go:  naming "google.golang.org/grpc/naming"
proxy/grpcproxy/cluster.go:     "go.etcd.io/etcd/v3/clientv3/naming"
proxy/grpcproxy/register.go:    gnaming "google.golang.org/grpc/naming"
proxy/grpcproxy/register.go:    "go.etcd.io/etcd/v3/clientv3/naming"

tests/integration/clientv3/grpc_test.go:        "google.golang.org/grpc/naming"
tests/integration/proxy/grpcproxy/register_test.go:     gnaming "google.golang.org/grpc/naming"

I have good news and bad news:

I did a little experimentation. Adding replace google.golang.org/grpc/naming => github.com/xiegeo/grpc-naming v1.29.1-alpha to go.mod file allows uploading grpc to v1.30.0 without any coding changes.

But another part of etcd is exposed as incompatible:

# go.etcd.io/etcd/v3/clientv3/balancer/picker
..\..\..\..\pkg\mod\go.etcd.io\etcd\v3@v3.3.0-rc.0.0.20200707003333-58bb8ae09f8e\clientv3\balancer\picker\err.go:25:9: cannot use &errPicker literal (type *errPicker) as type Picker in return argument:
        *errPicker does not implement Picker (wrong type for Pick method)
                have Pick(context.Context, balancer.PickInfo) (balancer.SubConn, func(balancer.DoneInfo), error)
                want Pick(balancer.PickInfo) (balancer.PickResult, error)
..\..\..\..\pkg\mod\go.etcd.io\etcd\v3@v3.3.0-rc.0.0.20200707003333-58bb8ae09f8e\clientv3\balancer\picker\roundrobin_balanced.go:33:9: cannot use &rrBalanced literal (type *rrBalanced) as type Picker in return argument:
        *rrBalanced does not implement Picker (wrong type for Pick method)
                have Pick(context.Context, balancer.PickInfo) (balancer.SubConn, func(balancer.DoneInfo), error)
                want Pick(balancer.PickInfo) (balancer.PickResult, error)

This is because of this change: https://github.com/grpc/grpc-go/commit/4eb418e5b2e10614c9638bc7748fb524636e5c4a

The grpc maintiners went through the trouble of creating a V2Picker, then broke semantic versioning intentionally.

I think the best option is to bite the bullet and upgrade with the incompatible changes, we can do this while adding go.mod which already changes the module path.