protobuf: reflect/protoregistry: conflicts with same filename

It seems plausible that someone might want to link packages outside their control that both contain a common.proto, though.

Sure, but that doesn’t seem to be problem in this issue though.

In my opinion, the protobuf developer site is lacking guidance in general in this area, and the protoc tool itself is not particularly friendly towards users who do not work in a mono-repo like Google does (where every .proto source file is relative to the root of the mono-repo). The open-source world is fundamentally a collection of disjoint repositories. There needs to be better guidance on what is the appropriate import path for a .proto file and how to reference a .proto file that crosses repo boundaries.

finally, I got the conflict panic! And l’d like to paste some doc from protobuf here: doc from panic

All protocol buffers declarations linked into a Go binary are inserted into a global registry.

Every protobuf declaration (e.g., enums, enum values, or messages) has an absolute name, which is the concatenation of the 

package name with the relative name of the declaration in the .proto source file (e.g., 

my.proto.package.MyMessage.NestedMessage). The protobuf language assumes that all declarations are universally unique.

as the doc states: The protobuf language assumes that all declarations are universally unique. note: all declarations not file names. Please consider fix it.

So we have the situation @dsnet describes. We have Go program that depends on two external libraries both of which have a spec.proto and we get this warning when the program runs

2020/08/11 11:20:11 WARNING: proto: file "spec.proto" is already registered
A future release will panic on registration conflicts. See:
https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict

I agree with @strowk, at the very least there should be some second level logic based on package name to disambiguate source file name collisions. The workaround of changing how the protobuf is compiled does not work when an external library is managing how the protobuf is compiled.

Along the current logical arc, the implication is that all protobufs in all repositories globally, need to have unique filenames - which is not reasonable. I really hope the threat of panic in future releases does not come to fruition.

I also had similar problem and what I don’t understand is how it does not take “package” into account. In my case I have two events.proto files in two different folders, they generate events.pb.go , also in two different folders AND both of them have different “package” and go_package. Why do I get proto: file "events.proto" is already registered? Is this a bug in protobuf or I don’t understand that documentation - https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict ? Particulary that

Every protobuf declaration (e.g., enums, enum values, or messages) has an absolute name, which is the concatenation of the package name with the relative name of the declaration in the .proto source file

sounds like I should not have this problem, yet warning is printed

I think we should drop the requirement that the paths of registered files be unique.

The paths of source files are only weakly part of the protobuf data model at best. Unlike descriptor names, source file names are not used in any of the serialization forms. It is also expected that renaming files or moving descriptors between files have no user-visible effects.

In an environment where all the generated code is built at the same time, enforcing a requirement that source .proto files have unique names is easy. However, the nature of the Go build system is such that .pb.go files are usually generated in many different places at different times by different people and committed to source control. In this environment, enforcing uniqueness of source file paths is difficult.

If we relax this requirement, we need to define what Files.FindFileByPath returns when multiple files match the requested path. I think the simplest answer is for it to return an error.

Has the protobuf team made any further decisions on this issue?

Much like @rcgoodfellow, I have a Go program that depends on two third-party libraries, each of which defines a telemetry.proto file at the top level of their project. With current google.golang.org/protobuf, the program panics unless I set the GOLANG_PROTOBUF_REGISTRATION_CONFLICT environment variable.

This can’t be that uncommon a situation. At this point, I’m not really clear how the protobuf team expect the file-by-path registry to work outside of a monorepo setting.

We’re coming up this issues’ first birthday 🍰 Any resolution in sight?

@dsnet @neild It seems that the latest release went ahead and added enforcement as the default behavior. This of course broke code and we will have to go back and mitigate via env flags or build flags.

There could be a million reasons Go team has decided to do this, so this isn’t a criticism and like everything else, people aren’t going to be happy with the decision. I’m hoping you could address this issue here to let us know that this is just the way it is and close the issue or that there is some other way you plan to fix this in the future (by adding enforcement, this seems like the issue really should just be closed).

I would just like to move this issue towards resolution so we are all on the same page.

For anyone reading, your mitigations are:

go build -ldflags “-X google.golang.org/protobuf/reflect/protoregistry.conflictPolicy=warn”

Env var: GOLANG_PROTOBUF_REGISTRATION_CONFLICT=warn ./main

This is really becoming an issue. I also don’t get why would it look/care for the proto filename. I’m using different package names for the protobuf and still have the issue complaining about proto file being registered twice.

My protos structure: service-a/entities.proto service-b/entities.proto

different option go_package = "something"; in both of them. If I include both packages I get: panic: proto: file "entities.proto" is already registered

I know I can solve this with absolute paths but it’s definitely not ideal.

I met this problem recently, any one solved this?

We worked around it by using the organization/company/project name in the filenames (🤮), i.e. orgname_common.proto.

For those who just want to get around and not willing to change the build settings, you can have a check at this small CLI tool: https://github.com/tuimeo/go-proto-filename-prefixer

It modify the generated pb.go files and add prefix to the embedded filename.

Along the current logical arc, the implication is that all protobufs in all repositories globally, need to have unique filenames - which is not reasonable.

This remains just as bizarre and unreasonable today as it was two years ago.

@neild

but why i have two protof iles like a/b.proto, c/b.proto , it goes panic? It’s obviously in a differante hierarchy

See @dsnet’s comment here: https://github.com/golang/protobuf/issues/1122#issuecomment-852577631

It’s a bug for a program to include two .proto files with the same name. This decision is made at the protobuf project level, and the maintainers of the Go implementation do not have the ability to change it. Sorry.

I believe the recommendation is that any shared .proto files be given a name that ensures they have a unique prefix; for example, google/protobuf/any.proto has a prefix indicating the organization (google) and project (protobuf) that owns the file.

Hi, since the issue was closed by https://github.com/protocolbuffers/protobuf-go/commit/febffdd88e85cb4402205142aaa1a4cc64d0d375, but then it was rollbacked at https://github.com/protocolbuffers/protobuf-go/commit/21e33cc91079beb975323466e237f2486ea29c10, shouldn’t it be reopened for visibility?

Modifying the rawDesc like the proposed tool does is highly inappropriate. This issue is far more complex than presented.

It would be nice to at least know how to generate files with protoc to avoid this, I can’t find any docs on this

Like I have no idea how the path ends up in the generated binary blob

Different package YYYY and option go_package = XXX in multiple .proto on my end… but same as above:

WARNING: proto: file "service.proto" is already registered

I don’t understand why the file name is a problem if content is scoped to a package…

Does any of you have a reasonable solution? Should I modify all files (but it’s not a proper solution), or cross fingers so their panic warnings will never be implemented?

Thank you,

Agree with previous comments. Not everyone uses a monorepo, and enforcing unique file path across different services is unnecessary and troublesome.