grpc: Grpc.Tools doesn't support same filename in different folders

What version of gRPC and what language are you using?

Grpc.Tools 1.17.1

What operating system (Linux, Windows, …) and version?

Windows 10

What runtime / compiler are you using (e.g. python version or version of gcc)

C# MSBuild

What did you do?

One project with two protos having the same filename but in different folders.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
    <Protobuf_StandardImportsPath>$(Protobuf_StandardImportsPath);..\protos</Protobuf_StandardImportsPath>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Grpc.Tools" Version="1.17.1" PrivateAssets="All" />
    <Protobuf Include="..\protos\foo.proto" Link="protos\foo.proto" />
    <Protobuf Include="..\protos\dir1\bar.proto" Link="protos\dir1\bar.proto" />
    <Protobuf Include="..\protos\dir2\bar.proto" Link="protos\dir2\bar.proto" />
  </ItemGroup>
</Project>

What did you expect to see?

The code should compile, generating 3 cs files.

What did you see instead?

MSB3105: The item "obj\Debug\netstandard2.0\Bar.cs" was specified more than once in the "Sources" parameter.  Duplicate items are not supported by the "Sources" parameter.

Anything else we should know about your project / environment?

We have an extensive library of protos that get reused by different projects that also share some protos. Therefore a common root path is necessary. The name config.proto for example is used in a couple of directories.

Until now, I used the following command to generate the files:

%protoc_path%\protoc.exe --proto_path=%protoc_path%\..;..\protos --csharp_out=%output_path% --csharp_opt=base_namespace --grpc_out=%output_path% --plugin=protoc-gen-grpc=%protoc-grpc_path%\grpc_csharp_plugin.exe ..\protos\foo.proto ..\protos\dir1\bar.proto ..\protos\dir2\bar.proto

The option base_namespace placed the generated files in sub-directories and avoided the problem. This problem will still remains for grpc service files, since there is no option like this for grpc_opt. But that problem can’t be circumvented so easy right now.

/cc @kkm000

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 10
Comments: 32 (30 by maintainers)

Most upvoted comments

What would you say about having a working solution without any code changes? 💃

<Protobuf Include="**\*.proto" OutputDir="$(Protobuf_OutputPath)%(RelativeDir)" />

$(Protobuf_OutputPath) - returns obj path %(RelativeDir) - returns relative proto file structure

I am using this in project with 344 proto files (envoy xds api), see example below. All my proto files are located in Protos folder.

  <ItemGroup>
    <Protobuf Include="Protos\**\*.proto" ProtoRoot="Protos" 
         OutputDir="$(Protobuf_OutputPath)%(RelativeDir)" />
  </ItemGroup>

It was a one liner for all of my problems. Can anybody confirm if that works in your setup? @jtattermusch

wicharypawel on Apr 8, 2020

As the issue was opened almost a year now, I don’t expect it will be fixed soon, I will go for a workdaround

<ItemGroup>
    <Protobuf Include="..\protos\dir1\bar.proto" OutputDir="$(Protobuf_OutputPath)dir1" GrpcServices="Server"/>
    <Protobuf Include="..\protos\dir2\bar.proto" OutputDir="$(Protobuf_OutputPath)dir2" GrpcServices="Server"/>
</ItemGroup>

zhangpengchen on Oct 24, 2019

@Falco20019, I have created PR with update to Grpc.Tools documentation, with proposed solution, mainly for those people who are looking for solution today. When your PR will be merged I would ask you to update documentation.

wicharypawel on Apr 9, 2020

@kkm000 Any news on that? The problem is growing more and more on us. We are using the workaround from @zhangpengchen and have a knowledge-base article on it. But it’s still coming up every couple weeks…

Falco20019 on Mar 2, 2020

@jtattermusch Sounds like a great idea to me and would be a lot less fragile. And it would also solve the long path problem. So I would be fine with investigating that route as default.

I‘m currently out of office for the rest of the week. If no one else is looking into it, I might be able to look into it on the weekend or next week.

%(Link) is AFAIK also including the absolute path in this case and would not solve the problem.

Falco20019 on Apr 29, 2020

I’ve looked at https://github.com/grpc/grpc/pull/22630 and it seems too fragile and complicated for what we want to do. We should investigate:

the MSBuild way of solving this (as discussed in my previous comment this https://github.com/grpc/grpc/issues/17672#issuecomment-621039153)
one potential approach would be to use similar trick as the one we used for disambiguating the .protodep files (which are also generated by Grpc.Tools). See here: https://github.com/grpc/grpc/blob/d86acba1daa56ff4867280cb5b4e238d58869ced/src/csharp/Grpc.Tools/DepFileUtil.cs#L166. The idea is that perhaps we could also hash the proto file’s full path (regardless what it originally was) and put the generated .cs file under OutputPath/hash_of_the_full_path/Helloworld.cs. That way the generated .cs file would have unique path and it would be easy for Grpc.Tools to compute what that path should be.

jtattermusch on Apr 29, 2020

No problem, I just go the clean way and finally bring base_namespace to the grpc_csharp_plugin. Just created the PR for it: #22627 😃

I will add another PR for adding support to Grpc.Tools right now.

Falco20019 on Apr 9, 2020

Perhaps this has been already discussed, but would this work?

if .proto file’s location within the project is foo/bar/some.proto, can we just set the output directory for that specific .proto file to obj/Debug/framework/foo/bar/ ? The C# compiler doesn’t really care where is the .cs file located and this would prevent the name collision. At the same time, the output directory is easy to determine as it’s just the path of the original .proto file within the project (no protoc mangling of namespaces involved).

jtattermusch on Mar 27, 2019

Maybe the right solution would be to implement base_namespace option for grpc_csharp_plugin too and make Grpc.Tools use that option by default (for both protoc and the plugin) - that would prevent the name clash.

jtattermusch on Mar 1, 2019